kafka使用

下载 wget ftp://ftp.kddilabs.jp/infosystems/apache/kafka/2.2.1/kafka_2.11-2.2.1.tgz

 

配置文件

broker.id=0

port=9092

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/opt/kafka/logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1

transaction.state.log.replication.factor=1

transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=192.168.10.21:2181

zookeeper.connection.timeout.ms=6000

host.name=pmaster

group.initial.rebalance.delay.ms=0

 

 

 

nohup /home/devel/tz/kafka/kafka/bin/kafka-server-start.sh /home/devel/tz/kafka/kafka/config/server.properties >/dev/null 2>&1 &

 

 

 

 查看当前服务器中的所有topic bin/kafka-topics.sh --list --zookeeper

zk01:2181

•   创建topic

bin/kafka-topics.sh --create --zookeeper 192.168.10.13:21812 --replication-factor 3 --partitions 3 --topic boray

•   删除topic

bin/kafka-topics.sh --delete --zookeeper pmaster:2181 --topic test

需要 server.properties 中设置 delete.topic.enable=true 否则只是标记删除或者直接重启。

•   通过shell命令发送消息

bin/kafka-console-producer.sh --broker-list 192.168.10.11:9092 --topic haha

•   通过shell消费消息

bin/kafka-console-consumer.sh --zookeeper 192.168.10.13:21812 --from-beginning --topic boray

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper pmaster:2181 --group

testGroup

 

shell生成

bin/kafka-console-producer.sh --broker-list 192.168.10.15:9092 --topic test

shell消费

bin/kafka-console-consumer.sh --bootstrap-server 192.168.10.13:9092 --topic boray

查看所有消费组

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --list

查看偏移量

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group

查看某个Topic的详情

bin/kafka-topics.sh --topic test --describe --zookeeper zk01:2181

对分区数进行修改

bin/kafka-topics.sh --zookeeper zk01 --alter --partitions 2 --topic test

 

 

scp -r 分发到所有的节点