Kafka非常实用的操作命令总结

Kafka常用操作命令

查看当前服务器中的所有topic

kafka-topics.sh --list --zookeeper  0.0.0.0:2181

创建topic

• 这里的副本数--replication-factor=1 表示这个topic一共有几份数据，这里表示只有一份数据，没有副本
• --config retention.ms=2592000000表示topic数据保存时间（ms）

kafka-topics.sh --create --zookeeper 0.0.0.0:2181 --replication-factor 3 --partitions 5 --topic  --config retention.ms=2592000000

topic级别修改

修改topic数据保存时间

kafka-configs.sh --zookeeper 0.0.0.0:2181 --alter --entity-type topics --entity-name xxx --add-config retention.ms=86400000

对分区数进行修改

//kafka 0.10版本与SparkStreaming结合支持新增分区检测，已经实测过
kafka-topics.sh --zookeeper  0.0.0.0:2181 --alter --partitions 15 --topic topicName

topic分区重分配

新增的broker是不会自动地分担已有topic的负载的，它只会对增加broker后新创建的topic生效，如果要让新增broker为已有的topic服务，用户必须手动地调整已有topic的分区分布，将一部分分区搬移到新broker上，这就是所谓的分区重分配操作partition reassignment

• 用户一定要谨慎的发起分区重分配操作，因为分区在不同的broker间进行数据迁移会极大的占用broker机器的带宽资源，从而显著的影响clients端业务应用的性能。尽量在非高峰时段执行重分配操作
• 现有两个broker，broker113，broker81
• 创建两个topic，--replication-factor 2 --partitions 3 --topic hello，world 两个副本，三个分区

//查看topic详情
[www@xxx kafka]$ ./bin/kafka-topics.sh --describe --zookeeper 0.0.0.0:2181 --topic hello
Topic:hello	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: hello	Partition: 0	Leader: 81	Replicas: 81,113	Isr: 81,113
	Topic: hello	Partition: 1	Leader: 113	Replicas: 113,81	Isr: 113,81
	Topic: hello	Partition: 2	Leader: 81	Replicas: 81,113	Isr: 81,113

[www@xxx kafka]$ ./bin/kafka-topics.sh --describe --zookeeper 0.0.0.0:2181 --topic world
Topic:world	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: world	Partition: 0	Leader: 113	Replicas: 113,81	Isr: 113,81
	Topic: world	Partition: 1	Leader: 81	Replicas: 81,113	Isr: 81,113
	Topic: world	Partition: 2	Leader: 113	Replicas: 113,81	Isr: 113,81

• 现需要增加一个broker98，并将分区和副本平均分配到三个节点上
  • 构造一个json文件

vim topics-to-move3.json
{"topics":[{"topic": "hello"},{"topic": "world"}],"version":1}

• 启动新增的broker98
• 执行 kafka/bin/kafka-reassign-partitions.sh 脚本生成分配方案

#执行
./bin/kafka-reassign-partitions.sh --zookeeper 0.0.0.0:2181 --topics-to-move-json-file topics-to-move3.json --broker-list "113,81,98" --generate

[www@xxx kafka]$ ./bin/kafka-reassign-partitions.sh --zookeeper 0.0.0.0:2181 --topics-to-move-json-file topics-to-move3.json --broker-list "113,81,98" --generate
#这个是现有分区情况，可以先保存下来，留着后续rollback
Current partition replica assignment
{"version":1,"partitions":[{"topic":"hello","partition":2,"replicas":[81,113]},{"topic":"hello","partition":0,"replicas":[81,113]},{"topic":"world","partition":1,"replicas":[81,113]},{"topic":"world","partition":2,"replicas":[113,81]},{"topic":"world","partition":0,"replicas":[113,81]},{"topic":"hello","partition":1,"replicas":[113,81]}]}

#这个是kafka推荐的分区情况，复制下来
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"hello","partition":2,"replicas":[113,81]},{"topic":"hello","partition":0,"replicas":[81,98]},{"topic":"world","partition":1,"replicas":[98,113]},{"topic":"world","partition":2,"replicas":[113,81]},{"topic":"world","partition":0,"replicas":[81,98]},{"topic":"hello","partition":1,"replicas":[98,113]}]}

• 创建另一个json文件 expand-cluster-reassignment.json 将推荐的分配方案复制进去

vim expand-cluster-reassignment3.json
{"version":1,"partitions":[{"topic":"test","partition":1,"replicas":[81]},{"topic":"test","partition":2,"replicas":[81]},{"topic":"test","partition":0,"replicas":[81]}]}

• 执行重分配

./bin/kafka-reassign-partitions.sh --zookeeper 0.0.0.0:2181 --reassignment-json-file expand-cluster-reassignment3.json --execute

• 查看执行进度,所有都已成功分配

./bin/kafka-reassign-partitions.sh --zookeeper 0.0.0.0:2181 --reassignment-json-file expand-cluster-reassignment3.json --verify

Status of partition reassignment:
Reassignment of partition [world,1] completed successfully
Reassignment of partition [world,0] completed successfully
Reassignment of partition [hello,0] completed successfully
Reassignment of partition [hello,2] completed successfully
Reassignment of partition [hello,1] completed successfully
Reassignment of partition [world,2] completed successfully

• 在查看topic详情,可见副本和分区已经均匀的分配到三个节点，此刻重分区就已经完成了，这是系统推荐的分配方案，我们自己也可自定义分配方案，下面我们继续

./bin/kafka-topics.sh --describe --zookeeper 0.0.0.0:2181 --topic hello
Topic:hello	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: hello	Partition: 0	Leader: 81	Replicas: 81,98	    Isr: 81,98
	Topic: hello	Partition: 1	Leader: 113	Replicas: 98,113	Isr: 113,98
	Topic: hello	Partition: 2	Leader: 81	Replicas: 113,81	Isr: 81,113

./bin/kafka-topics.sh --describe --zookeeper 0.0.0.0:2181 --topic hello
Topic:world	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: world	Partition: 0	Leader: 81	Replicas: 81,98	Isr: 81,98
	Topic: world	Partition: 1	Leader: 98	Replicas: 98,113	Isr: 113,98
	Topic: world	Partition: 2	Leader: 113	Replicas: 113,81	Isr: 113,81

• 自定义分配方案
  • 将hello分区0的两个副本移到broker113上（目前分别在broker81、broker98），
    将world分区1的两个副本分别移到broker81,broker113上（目前分别在broker113、broker98）
  • 现在不需要--generate参数了

1. 构建一个json文件

vim custom-reassignment.json
{ "version": 1, "partitions": [{ "topic": "hello", "partition": 0, "replicas": [113] }, { "topic": "world", "partition": 1, "replicas": [81, 113] }] }

2. 执行重分配

kafka-reassign-partitions.sh --zookeeper 0.0.0.0:2181 --reassignment-json-file custom-reassignment.json --execute

3. 查看执行进度

kafka-reassign-partitions.sh --zookeeper 0.0.0.0:2181 --reassignment-json-file custom-reassignment.json --verify

Status of partition reassignment:
Reassignment of partition [hello,0] completed successfully
Reassignment of partition [world,1] completed successfully
[www@xxx kafka]$ vim custom-reassignment.json

4. 再次查看topic详情

Topic:hello	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: hello	Partition: 0	Leader: 113	Replicas: 113	Isr: 113
	Topic: hello	Partition: 1	Leader: 98	Replicas: 98,113	Isr: 113,98
	Topic: hello	Partition: 2	Leader: 113	Replicas: 113,81	Isr: 81,113

Topic:world	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: world	Partition: 0	Leader: 81	Replicas: 81,98	Isr: 81,98
	Topic: world	Partition: 1	Leader: 81	Replicas: 81,113	Isr: 113,81
	Topic: world	Partition: 2	Leader: 113	Replicas: 113,81	Isr: 113,81

删除topic

#正在使用中的topic不能被删除

kafka-topics.sh --delete --zookeeper 0.0.0.0:2181 --topic

通过shell命令发送消息

kafka-console-producer.sh --broker-list 0.0.0.0:9092 --topic

通过shell消费消息

--property print.key=true 打印key

kafka-console-consumer.sh --zookeeper 0.0.0.0:2181 --property print.key=true --from-beginning --topic

消费制定offset的消息

//这里是消费0号分区的offset为172的消息，`| more +n 1`是为了停下方便查看那个位置的消息，不然会一直消费
kafka-console-consumer.sh --bootstrap-server localhost:9092 --property print.key=true --partition 0 --offset 172 --topic nginx_log | more +n 1

查看消费位置

./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper 0.0.0.0:2181 --group

查看Topic

查看某个Topic的详情

kafka-topics.sh --describe --zookeeper 0.0.0.0:2181 --topic

查看topic分区以及分区的offset，-1表示显示最大条数(初始offset为0时)，-2查看每个分区的最早offset，之前的过期了，比如0:100表示0号分区的0-99的offset过期了：

kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 0.0.0.0:9092 --time -1 --topic 

查询该topics的特殊配置

kafka-configs.sh --describe --zookeeper 0.0.0.0:2181 --entity-type topics --entity-name topicName

查看消费者组的消费情况

kafka-consumer-groups.sh --bootstrap-server 0.0.0.0:9092 --describe --group 

性能测试

producer性能测试

#发送500000条记录，每条200字节
kafka-producer-perf-test.sh --topic test --num-records 500000 --record-size 200 --throughput -1 --producer-props bootstrap.servers=0.0.0.0:9092 acks=-1

#每秒大概14万条、26.7M/sec，平均延迟743.47ms，最大延迟1120ms，50%的数据平均每条用了801ms，95%的数据1094ms，99%的数据1107ms，99.9%的数据1113ms
500000 records sent, 139977.603583 records/sec (26.70 MB/sec), 743.47 ms avg latency, 1120.00 ms max latency, 801 ms 50th, 1094 ms 95th, 1107 ms 99th, 1113 ms 99.9th.

consumer性能测试

kafka-consumer-perf-test.sh --broker-list 0.0.0.0:9092 --topic test --messages 1   --threads 3

查看__consumer_offsets

kafka-simple-consumer-shell.sh --topic __consumer_offsets --broker-list 0.0.0.0:9092 --partition 3 --format "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter"

后台启动kafka

kafka-server-start.sh -daemon config/server.properties