Kafka查询topic以及消息内容

基于kafka_2.11-0.10.0.1版本:

查看kafka的安装目录

find / -name kafka* -type d

查看redis安装目录:

ps -ef|grep redis

查询所有group信息:

./bin/kafka-topics.sh --list --zookeeper localhost:2181

查询指定topic的详细信息: 

./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic paymentmgt_download_asyn_deal

示例结果: 

Topic:paymentmgt_download_asyn_deal     PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: paymentmgt_download_asyn_deal    Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: paymentmgt_download_asyn_deal    Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: paymentmgt_download_asyn_deal    Partition: 2    Leader: 0       Replicas: 0     Isr: 0

 

创建topic:paymentmgt_download_asyn_deal

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic paymentmgt_download_asyn_deal

注:执行后报错:WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
意思是,由于度量名称的限制，带有句点（'.'）或下划线的主题可能产生冲突.

topic的名称必须由大小写字母、数字、“.”、“-”、“_”组成，不能为空、不能为“.”、不能为“…”，且长度不能超过249。
topic的命名同样不推荐（虽然可以这样做）使用双下划线“__”开头，因为以双下划线开头的topic一般看作是kafka的内部topic，比如__consumer_offsets和__transaction_state。

查询指定group的topic信息:

./bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group paymentmgt

查询指定topic的消息内容:

./kafka-console-consumer.sh --zookeeper localhost:2181 --topic paymentmgt_download_asyn_deal --from-beginning

查询匹配指定内容的消息内容:

./bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181,127.0.0.1:2182 --topic paymentmgt_download_asyn_deal --from-beginning | grep "1291248658964021250"

消息发送与消息消费示例:

消息发送:

./bin/kafka-console-producer.sh --broker-list 192.168.1.94:9092 --topic paymentmgt_download_asyn_deal 

消息消费: 

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic paymentmgt_download_asyn_deal

通过查看消费者组信息如下: CURRENT-OFFSET是指当前偏移量,  LOG-END-OFFSET是指结束的偏移量， LAG是指落后的偏移量.

GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER 

paymentmgt                     paymentmgt_download_asyn_deal  0          4               4               0               paymentmgt_192.168.1.94_gfx-0
paymentmgt                     paymentmgt_download_asyn_deal  1          0               1               1               paymentmgt_192.168.1.94_gfx-0
paymentmgt                     paymentmgt_download_asyn_deal  2          3               3               0               paymentmgt_192.168.2.33_quw-0 

---