Kafka常用工具

1. 创建topic

 kafka-topics --zookeeper datanode01.isesol.com:2181,datanode01.isesol.com:2181,datanode02.isesol.com:2181,datanode03.isesol.com:2181,datanode04.isesol.com:2181  --create --partitions 2 --replication-factor 2 --topic test

2. 查询topic最大及最小offset  (time: -2表示最小，-1表示最大)

 /opt/cloudera/parcels/KAFKA/lib/kafka/bin/kafka-run-class.sh  kafka.tools.GetOffsetShell --broker-list  namenode01.isesol.com:9092,namenode02.isesol.com:9092,datanode03.isesol.com:9092 --topic 2001 --time -1

 3.  手动测试消费消息

kafka-console-consumer  --zookeeper datanode01.isesol.com:2181,datanode01.isesol.com:2181,datanode02.isesol.com:2181,datanode03.isesol.com:2181,datanode04.isesol.com:2181 --topic 2005 --from-beginning

4. 查询__consumer_offset 的数据， config文件的值为：exclude.internal.topics=false

/opt/cloudera/parcels/KAFKA/lib/kafka/bin/kafka-console-consumer.sh --topic __consumer_offsets --zookeeper datanode01.isesol.com:2181,datanode01.isesol.com:2181,datanode02.isesol.com:2181,datanode03.isesol.com:2181,datanode04.isesol.com:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config --from-beginning 

5. 查询__consumer_offset某个分区的数据

 /opt/cloudera/parcels/KAFKA/lib/kafka/bin/kafka-simple-consumer-shell.sh --broker-list namenode01.isesol.com:9092,namenode02.isesol.com:9092,datanode03.isesol.com:9092 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --topic __consumer_offsets  --partition 0

6. __consumer_offset保存new consumer的offset, 如果数据仍然保存在zk，使用如下命令查看消息的offset

/usr/bin/kafka-consumer-groups --zookeeper zk01.example.com:2181 --describe --group flume

还有重新分布分区，查看log等工具

kafka-reassign-partitions.sh

kafka-replay-log-producer.sh 

0.11 :

bin/kafka-console-consumer.sh --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties --from-beginning

目前我碰到的问题是，new consumer查询为空，实际上我使用了new consumer去获取数据，而且查看__consumer_offset也的确有数据，但是查询不出来到底有多少个new consumer group,  另外有些用老的consumer来消费的，数据保存在ZK，名字格式为console-consumer-xxxxxx，