kafka各种shell操作

######################### 1、启动集群每个节点的进程 ###################################
nohup kafka-server-start.sh
/home/hadoop/apps/kafka_2.11-1.1.0/config/server.properties
1>~/kafkalogs/kafka_std.log
2>kafkalogs/kafka_err.log &

######################## 2、创建topic ################################################
kafka-topics.sh
–create
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–replication-factor 3
–partitions 10
–topic kafka_test

kafka-topics.sh
–create
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–replication-factor 1
–partitions 1
–topic weblog

参数解释：
–create 创建kafka topic
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181 指定kafka的zookeeper地址
–partitions 指定分区的个数
–replication-factor 指定每个分区的副本个数

######################## 3、查看已经创建的所有kafka topic #############################
kafka-topics.sh
–list
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181

######################## 4、查看某个指定的kafka topic的详细信息 #######################
kafka-topics.sh
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–describe
–topic kafka_test

######################### 4、开启生产者模拟生成数据 ####################################
kafka-console-producer.sh
–broker-list hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092
–topic kafka_test

########################## 5、开启消费者模拟消费数据 ###############################
kafka-console-consumer.sh
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–from-beginning
–topic kafka_test

kafka-console-consumer.sh
–bootstrap-server hadoop02:9092,hadoop03:9092,hadoop04:9092
–from-beginning
–topic kafka_test

########################### 6、查看某topic某个分区的偏移量最大值和最小值 ##################
kafka-run-class.sh
kafka.tools.GetOffsetShell
–topic kafka_test
–time -1
–broker-list hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092
–partitions 0

############################# 7、增加topic分区数 ###########################################
kafka-topics.sh
–alter
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–topic kafka_test
–partitions 15

这个操作是不被允许的。
kafka-topics.sh
–alter
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–topic kafka_test
–replication-factor 2

##################### 8、获取节点的kafka消息 ########################################
kafka-topics.sh
–describe
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–topic kafka_test

结果：
Topic:kafka_test PartitionCount:15 ReplicationFactor:1 Configs:
Topic: kafka_test Partition: 0 Leader: 5 Replicas: 5 Isr: 5
Topic: kafka_test Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: kafka_test Partition: 2 Leader: 3 Replicas: 3 Isr: 3
Topic: kafka_test Partition: 3 Leader: 4 Replicas: 4 Isr: 4
Topic: kafka_test Partition: 4 Leader: 5 Replicas: 5 Isr: 5
Topic: kafka_test Partition: 5 Leader: 2 Replicas: 2 Isr: 2
Topic: kafka_test Partition: 6 Leader: 3 Replicas: 3 Isr: 3
Topic: kafka_test Partition: 7 Leader: 4 Replicas: 4 Isr: 4
Topic: kafka_test Partition: 8 Leader: 5 Replicas: 5 Isr: 5
Topic: kafka_test Partition: 9 Leader: 2 Replicas: 2 Isr: 2
Topic: kafka_test Partition: 10 Leader: 3 Replicas: 3 Isr: 3
Topic: kafka_test Partition: 11 Leader: 4 Replicas: 4 Isr: 4
Topic: kafka_test Partition: 12 Leader: 5 Replicas: 5 Isr: 5
Topic: kafka_test Partition: 13 Leader: 2 Replicas: 2 Isr: 2
Topic: kafka_test Partition: 14 Leader: 3 Replicas: 3 Isr: 3

第一行是对所有分区的一个描述，然后每个分区都会对应一行
Topic：topic名称
Partition：topic的分区编号
leader：负责处理消息的读和写，leader是从所有节点中随机选择的。
replicas：列出了所有的副本节点，不管节点是否在服务中。
isr：正在服务中的节点。

##################### 9、删除Topic ###################################################
kafka-topics.sh
–delete
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–topic kafka_test
–delete-config delete.topic.enable=true

kafka-topics.sh
–delete
–zookeeper hadoop02:2181,hadoop03:2181,hadoop04:2181
–topic weblog
–delete-config delete.topic.enable=true