Kafka常用操作

1.Kafka监控
kafka-manager: https://github.com/yahoo/kafka-manager

2.启动Kafka
## 启动zookeeper
bin/zkServer.sh start
## 打开zk客户端查询
bin/zkCli.sh -server dsjcx1:2181,dsjcx2:2181,dsjcx3:2181
## 启动kafka
bin/kafka-server-start.sh -daemon ./config/server.properties

## 现在kafka的消费者offser都存在topic为__consumer_offsets中

3.Kafka基本操作
## create topic
sh /kafka/bin/kafka-topics.sh --create --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka --replication-factor 2 --partitions 1 --topic FLINK-PAUSE-TOPIC

## producer
sh /kafka/bin/kafka-console-producer.sh --broker-list dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --topic test0916
sh /kafka/bin/kafka-console-producer.sh --broker-list dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --topic testTd < testTd.json

## consumer
sh /kafka/bin/kafka-console-consumer.sh --bootstrap-server dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --topic FLINK-PAUSE-TOPIC
sh /kafka/bin/kafka-console-consumer.sh --bootstrap-server dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --from-beginning --topic test123 > test123.json
sh /kafka/bin/kafka-console-consumer.sh --bootstrap-server dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --topic test123 --partition 0 --offset 6269200 > test123.json

## offset 分区：最小值 分区：最大值 其中最大值为当前offset
sh /kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --topic test --time -2
sh /kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --topic test --time -1

## offset提交情况(新旧版本的命令)
bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server localhost:9092 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties
bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server localhost:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties

## 查看topic
sh /kafka/bin/kafka-topics.sh --list  --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka
## 查看topic详细
sh /kafka/bin/kafka-topics.sh --describe  --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka --topic test123
## 删除topic
sh /kafka/bin/kafka-topics.sh --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka --delete --topic test123

## 查看consumer
sh /kafka/bin/kafka-consumer-groups.sh --new-consumer  --bootstrap-server dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --list
sh /kafka/bin/kafka-consumer-groups.sh --new-consumer  --bootstrap-server dsjcx1:9092,dsjcx2:9092,dsjcx3:9092 --group sx_bank_flink_project --describe
sh /kafka/bin/kafka-consumer-groups.sh --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka --list
sh /kafka/bin/kafka-consumer-groups.sh --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka --group sx_bank_flink_project

## 完全删除topic及数据
1.如果配置了delete.topic.enable=true
1.1 sh /kafka/bin/kafka-topics.sh --zookeeper dsjcx1:2181,dsjcx2:2181,dsjcx3:2181/kafka --delete --topic test
1.2 zookeeper上执行rmr /kafka/brokers/topics/test
1.3 删除kafka的数据目录 rm -rf /tmp/kafka-log/test-*
如果想重新建这个topic，必须重启kafka再重建，否则新建的topic会marked for deletion