启动命令:
1、启动Zookper
bin/zookeeper-server-start.sh config/zookeeper.properties &
2、现在启动Kafka服务(集群时有多个,分别启动)
bin/kafka-server-start.sh config/server.properties
集群时(kafka配置多个broker,配置多个server.properties,并设置里面的ip和端口以及日志输出路径):
bin/kafka-server-start.sh config/server-1.properties &
3、(首次需要创建topic)
bin/kafka-topics.sh --create --zookeeper 192.168.66.124:2181 --replication-factor 1 --partitions 1 --topic topicName
通过list命令查看创建的topic:
bin/kafka-topics.sh --list --zookeeper 192.168.66.124:2181
4、启动producer,发送消息到topic
bin/kafka-console-producer.sh --broker-list 192.168.66.124:9092 --topic topicName
5、启动consumer(查看订阅的topic消息)
bin/kafka-console-consumer.sh --zookeeper 192.168.66.124:2181 --topic topicName --from-beginning
以下命令中使用的zookeeper配置地址为127.0.0.1:2181,bootstrap--server(即broker)地址为: 127.0.0.1:9292
1,查看kafka topic列表,使用--list参数
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
__consumer_offsets
lx_test_topic
test
2,查看kafka特定topic的详情,使用--topic与--describe参数
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic lx_test_topic --describe Topic:lx_test_topic PartitionCount:1 ReplicationFactor:1 Configs: Topic: lx_test_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
列出了lx_test_topic的parition数量、replica因子以及每个partition的leader、replica信息
3,查看consumer group列表,使用--list参数
查看consumer group列表有新、旧两种命令,分别查看新版(信息保存在broker中)consumer列表和老版(信息保存在zookeeper中)consumer列表,因而需要区分指定bootstrap--server和zookeeper参数:
bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server 127.0.0.1:9292 --list lx_test
bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --list console-consumer-86845 console-consumer-11967
4,查看特定consumer group 详情,使用--group与--describe参数
同样根据新/旧版本的consumer,分别指定bootstrap-server与zookeeper参数:
bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server 127.0.0.1:9292 --group lx_test --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER lx_test lx_test_topic 0 465 465 0 kafka-python-1.3.1_/127.0.0.1
bin/kafka-consumer-groups.sh --zookeeper 127.0.0.1:2181 --group console-consumer-11967 --describe GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER Could not fetch offset from zookeeper for group console-consumer-11967 partition [lx_test_topic,0] due to missing offset data in zookeeper. console-consumer-11967 lx_test_topic 0 unknown 465 unknown console-consumer-11967_aws-lx-1513787888172-d3a91f05-0
其中依次展示group名称、消费的topic名称、partition id、consumer group最后一次提交的offset、最后提交的生产消息offset、消费offset与生产offset之间的差值、当前消费topic-partition的group成员id(不一定包含hostname)
上面示例中console-consumer-11967是为了测试临时起的一个console consumer,缺少在zookeeper中保存的current_offset信息。