kafka相关命令

• 创建topic

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-topics.sh --create --zookeeper master.hive.com:2181,node01.hive.com:2181,node02.hive.com:2181 --replication-factor 1 --partitions 1 --topic data_test
  ----------------------------------
  --zookeeper：为zk服务器地址，已逗号分割配置多个 
  --replication-factor：分区leader副本数,1代表没有副本即分区本身，建议为2 
  --partitions：分区数 
  --topic：topic名称
  

• 查看topic列表

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-topics.sh --list --zookeeper master.hive.com:2181,node01.hive.com:2181,node02.hive.com:2181
  

• 查看指定topic信息

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-topics.sh --describe --zookeeper master.hive.com:2181,node01.hive.com:2181,node02.hive.com:2181 --topic data_test
  -------------------
  leader:负责处理消息的读和写，leader是从所有节点中随机选择的.
  Replicas:列出了所有的副本节点，不管节点是否在服务中.
  Lsr:是正在服务中的节点.
  

• 生产者

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-console-producer.sh --broker-list master.hive.com:9092,node01.hive.com:9092,node02.hive.com:9092 --topic news_data_2W_realtime
  

• 消费者

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-console-consumer.sh --bootstrap-server master.hive.com:9092,node01.hive.com:9092,node02.hive.com:9092 --topic data_test --from-beginning
  ----------------------------
  from-beginning：每次从头开始消费
  

• 删除topic

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-topics.sh --delete --zookeeper master.hive.com:2181,node01.hive.com:2181,node02.hive.com:2181 --topic data_test	
  

• 查看topic的分区和副本信息

  /opt/cluster/kafka_2.12-2.3.0/bin/kafka-topics.sh --describe --zookeeper master.hive.com:2181,node01.hive.com:2181,node02.hive.com:2181  --topic news_data_2W_realtime
  

• 看下group消费情况

  /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server master:6667,node01:6667,node02:6667 --group "consumer_data_to_topic" --describe
  

• 查询消费者组

  /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --bootstrap-server master:6667,node01:6667,node02:6667 --list
  

• 删除消费者组

  /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --bootstrap-server master:6667,node01:6667,node02:6667 --delete --group 7d84f064353611ebab99fa163e15af0c