kafka的开发,基于java的方式,使用maven管理jar包依赖。
我主要是基于C/c++,采用了librdkafka 。
librdkafka 是Apache Kafka 客户端C语言的高性能实现, 能够提供可靠并且表现优秀的客户端,同时它也提供比较初级的C++界面,提供了Kafka生产者、消费者接口。
1. librdkafka的安装
首先在 https://github.com/edenhill/librdkafka 下载 librdkafka-master.zip,上传至linux;
yum install -y unzip zip
unzip librdkafka-master.zip
yum -y update gcc
yum -y install gcc+ gcc-c++
以上两行没有的话configure出错
mklove/modules/configure.base: line 1183: c++: command not found
cd librdkafka-master
chmod 777 configure lds-gen.py
./configure
make
make install
最终头文件和库文件会分别安装在
/usr/local/include/librdkafka
/usr/local/lib
echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig
查看结果如下:
[root@izbp11qlpa7oecwpwy4cn4z test]# ll /usr/local/lib
total 22972
lrwxrwxrwx 1 root root 12 Jul 17 15:13 librdkafka -> librdkafka.1
lrwxrwxrwx 1 root root 14 Jul 17 15:13 librdkafka++ -> librdkafka++.1
-rwxr-xr-x 1 root root 14274004 Jul 17 15:22 librdkafka.a
-rwxr-xr-x 1 root root 2493016 Jul 17 15:22 librdkafka++.a
lrwxrwxrwx 1 root root 15 Jul 17 15:22 librdkafka.so -> librdkafka.so.1
lrwxrwxrwx 1 root root 17 Jul 17 15:22 librdkafka++.so -> librdkafka++.so.1
-rwxr-xr-x 1 root root 5783120 Jul 17 15:22 librdkafka.so.1
-rwxr-xr-x 1 root root 964000 Jul 17 15:22 librdkafka++.so.1
drwxr-xr-x 2 root root 4096 Jul 17 15:22 pkgconfig
[root@izbp11qlpa7oecwpwy4cn4z test]# ll /usr/local/include/librdkafka
total 256
-rwxr-xr-x 1 root root 74175 Jul 17 15:22 rdkafkacpp.h
-rwxr-xr-x 1 root root 182959 Jul 17 15:22 rdkafka.h
2. 开发实践
包含头文件 #include "librdkafka/rdkafka.h"
编译过程中出错:
/usr/bin/ld: cannot find -lz
解决:
yum install zlib-devel
gcc my_consumer.c -o my_consumer -lrdkafka -lz -lpthread -lrt
gcc my_producer.c -o my_producer -lrdkafka -lz -lpthread -lrt
./my_producer localhost:9092 test_topic
./my_consumer -b localhost:9092 -t test_topic