Kafka常用指令

Kafka常用指令

Kafka环境配置：

1. Kafka的安装目录：/usr/local/kafka
2. Kafka的服务配置文件：/usr/local/kafka/config/server.properties

1. 客户端id：broker.id
2. 数据目录路径：log.dirs
3. Zookeeper HOST：zookeeper.connect
4. 数据存储的最大时间: log.retention.hours(默认24小时)
5. 控制日志segment文件的大小：log.segment.bytes

1. Kafka的数据目录：/usr/local/kafka/data
   1. 存放topic消息数据
   2. 消费offset
2. Kafka的日志目录：/var/log/hems/kafka
   1. 主要看服务器运行日志server.log

查看kafka集群状态

目的：查看kafka集群中的所有topic信息

方法：

/usr/local/kafka/bin/kafka-topics.sh --describe --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181

注：testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181：zookeeper节点

 

Topic:主

PartitionCount：分区数

ReplicationFactor：复制数

创建topic

目的：创建topic

方法：

/usr/local/kafka/bin/kafka-topics.sh --create --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --replication-factor 2 --partitions 3 --topic SprintKPITopic1

注：testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181：zookeeper节点

查看topic

目的：查看kafka中已存在的topic

方法：

bin/kafka-topics.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --list

注：testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181：zookeeper节点

 

查看消费组

目的：查看kafka中已存在的消费组

方法：

bin/kafka-consumer-groups.sh --bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 -list

注：testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092：kafka节点

 

查看组消费情况

目的：查看消息消费情况

方法：

bin/kafka-consumer-groups.sh --bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 --describe --group testGroup

bin/kafka-consumer-groups.sh --bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 --describe --group prtsGroup

注：testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092：kafka节点

 

TOPIC：主题          

PARTITION：分区id 

CURRENT-OFFSET: 当前已消费的条数 

LOG-END-OFFSET: 总条数 

LAG：未消费的条数

查看kafka消息内容

目的：查看kpi原始报文

方法：

找到topic(kpi TOPIC为：SprintKPITopic)

获取数据文件路径：/usr/local/kafka/data/SprintKPITopic-X(X:0,1,2)/*********.log

 

执行命令：

/usr/local/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /usr/local/kafka/data/SprintKPITopic-0/00000000000000336835.log --print-data-log > kpi-original.log

注：红色字体替换为步骤2获取的路径，由于报文比较大，防止文件很大，可按 CTRL+C终止，获取部分

查看kpi-original.log查看报文

分区重置

目的：修改分区存放节点，或者扩容再分配

方法：

[root@testems01lkpi5 kafka]# vi  topics-to-move.json

{"topics": [{"topic": "__consumer_offsets"}],

 "version":1

}

:wq

[root@testems01lkpi5 kafka]#bin/kafka-reassign-partitions.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --topics-to-move-json-file  topics-to-move.json --broker-list "0,1,2" --generate

注：testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181：zookeeper节点

Current partition replica assignment

{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]}]}

 

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":2,"replicas":[2],"log_dirs":["any"]}]}

 

[root@testems01lkpi5 kafka]#vi result.json

{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":2,"replicas":[2],"log_dirs":["any"]}]}

:wq

[root@testems01lkpi5 kafka]#bin/kafka-reassign-partitions.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --reassignment-json-file result.json –execute

注：testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181：zookeeper节点

Current partition replica assignment

 

{"version":1,"partitions":[{"topic":"TOPIC_KPI","partition":2,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"TOPIC_KPI","partition":0,"replicas":[0],"log_dirs":["any"]}]}

 

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions.

 

[root@testems01lkpi5 data]#/usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181 --reassignment-json-file kafka_partion_reset.json --verify

注：testems01lredis1:2181,testems01lredis2:2181,testems01lredis3:2181：zookeeper节点

 

查看topic节点信息

目的：所有kafka消息日志目录以及目录下所有topic信息

方法：

bin/kafka-log-dirs.sh -bootstrap-server testems01lkpi4:9092,testems01lkpi5:9092,testems01lkpi6:9092 -describe -topic-list SprintKPITopic