Kafka常用指令
Kafka环境配置:
- Kafka的安装目录:/usr/local/kafka
- Kafka的服务配置文件:/usr/local/kafka/config/server.properties
- 客户端id:broker.id
- 数据目录路径:log.dirs
- Zookeeper HOST:zookeeper.connect
- 数据存储的最大时间: log.retention.hours(默认24小时)
- 控制日志segment文件的大小:log.segment.bytes
- Kafka的数据目录:/usr/local/kafka/data
- 存放topic消息数据
- 消费offset
- Kafka的日志目录:/var/log/hems/kafka
- 主要看服务器运行日志server.log
查看kafka集群状态
目的:查看kafka集群中的所有topic信息
方法:
/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181
注:testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181:zookeeper节点
Topic:主
PartitionCount:分区数
ReplicationFactor:复制数
创建topic
目的:创建topic
方法:
/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --replication-factor 2 --partitions 3 --topic SprintKPITopic1
注:testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181:zookeeper节点
查看topic
目的:查看kafka中已存在的topic
方法:
bin/kafka-topics.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --list
注:testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181:zookeeper节点
查看消费组
目的:查看kafka中已存在的消费组
方法:
bin/kafka-consumer-groups.sh --bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 -list
注:testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092:kafka节点
查看组消费情况
目的:查看消息消费情况
方法:
bin/kafka-consumer-groups.sh --bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 --describe --group testGroup
bin/kafka-consumer-groups.sh --bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 --describe --group prtsGroup
注:testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092:kafka节点
TOPIC:主题
PARTITION:分区id
CURRENT-OFFSET: 当前已消费的条数
LOG-END-OFFSET: 总条数
LAG:未消费的条数
查看kafka消息内容
目的:查看kpi原始报文
方法:
找到topic(kpi TOPIC为:SprintKPITopic)
获取数据文件路径:/usr/local/kafka/data/SprintKPITopic-X(X:0,1,2)/*********.log
执行命令:
/usr/local/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /usr/local/kafka/data/SprintKPITopic-0/00000000000000336835.log --print-data-log > kpi-original.log
注:红色字体替换为步骤2获取的路径,由于报文比较大,防止文件很大,可按 CTRL+C终止,获取部分
查看kpi-original.log查看报文
分区重置
目的:修改分区存放节点,或者扩容再分配
方法:
[root@testems01lkpi5 kafka]# vi topics-to-move.json
{"topics": [{"topic": "__consumer_offsets"}],
"version":1
}
:wq
[root@testems01lkpi5 kafka]#bin/kafka-reassign-partitions.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2" --generate
注:testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181:zookeeper节点
Current partition replica assignment
{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":2,"replicas":[2],"log_dirs":["any"]}]}
[root@testems01lkpi5 kafka]#vi result.json
{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":2,"replicas":[2],"log_dirs":["any"]}]}
:wq
[root@testems01lkpi5 kafka]#bin/kafka-reassign-partitions.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --reassignment-json-file result.json –execute
注:testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181:zookeeper节点
Current partition replica assignment
{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
[root@testems01lkpi5 data]#/usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --reassignment-json-file kafka_partion_reset.json --verify
注:testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181:zookeeper节点
查看topic节点信息
目的:所有kafka消息日志目录以及目录下所有topic信息
方法:
bin/kafka-log-dirs.sh -bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 -describe -topic-list SprintKPITopic