常用kafka shell

胖胖胖胖胖虎

已于 2023-10-23 15:55:58 修改

阅读量229

点赞数

分类专栏： Kafka 文章标签： kafka

于 2021-12-22 16:35:59 首次发布

本文链接：https://blog.csdn.net/qq_15138049/article/details/122088875

版权

Kafka 专栏收录该内容

16 篇文章 1 订阅

订阅专栏

Kafka shell

Kafka全网最全最详细运维命令合集(精品强烈建议保存)

新建topic

kafka-topic --create --zookeeper kafka-zookeeper-headless:2181 --topic test --replication-factor 3 --partitions 3

修改kafka 分区数

kafka-topics --bootstrap-server broker_host:port --alter --topic test --partitions 2

###
root@kafka-1:/opt/kafka/data/logs/aaa-0# kafka-topics --zookeeper kafka-zookeeper-headless:2181 --alter --topic aaa --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!

root@kafka-1:/opt/kafka/data/logs# ls -l | grep aaa
drwxr-sr-x. 2 root 2004 4096 Feb  8 07:44 aaa-0
drwxr-sr-x. 2 root 2004 4096 Feb  8 07:56 aaa-1
drwxr-sr-x. 2 root 2004 4096 Feb  8 07:56 aaa-2

删除topic

server.property:
delete.topic.enable=true
##
kafka-topics.sh --zookeeper kafka-zookeeper-headless:2181 --delete --topic test

在这里插入图片描述

消息生产

kafka-console-producer --broker-list kafka-headless:9092 --topic test

主题消费

kafka-console-consumer --bootstrap-server kafka-headless:9092 --topic test --from-beginning

消费者组

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --list

删除消费者组

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --delete --group <groupname>

kafka-run-class.sh 查看分区消费偏移量

kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list kafka-headless:9092 \
  --time -1 \
  --topic test1 \
  --offsets 100 \
  # 指定分区
  --partitions 61

###
test1:2:20
test1:1:20
test1:0:20

指定主题分区进行消费

kafka-simple-consumer-shell --topic __consumer_offsets --partition 19 --broker-list kafka-headless:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

消费偏移量

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --describe --group nufront.loganalysis.test.01

重置消费偏移量

重置策略
–to-earliest：把位移调整到分区当前最小位移
–to-latest：把位移调整到分区当前最新位移
–to-current：把位移调整到分区当前位移
–to-offset ：把位移调整到指定位移处
–shift-by N：把位移调整到当前位移 + N处，注意N可以是负数，表示向前移动
–to-datetime ：把位移调整到大于给定时间的最早位移处，datetime格式是yyyy-MM-ddTHH:mm:ss.xxx，比如2017-08-04T00:00:00.000

https://blog.csdn.net/u010003835/article/details/83314766

 kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group nufront.loganalysis.test.01  --reset-offsets --topic test --to-earliest --execute
 

###

kafka-consumer-groupsh --bootstrap-server kafka-headless:9092 --group group --reset-offsets --all-topics --to-datetime 2018-10-23T18:50:00.000+08:00 --execute

更新到当前group最初的offset位置
kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group test-group --reset-offsets --topics <topic_name> --to-earliest --execute
更新到指定的offset位置

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group test-group --reset-offsets --all-topics --to-offset 500000 --execute
更新到当前offset位置（解决offset的异常）

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group test-group --reset-offsets --all-topics --to-current --execute
offset位置按设置的值进行位移

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group test-group --reset-offsets --all-topics --shift-by -100000 --execute
offset设置到指定时刻开始

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group test-group --reset-offsets --all-topics --to-datetime 2017-08-04T14:30:00.000
查看消费组状态

kafka-consumer-groups --bootstrap-server kafka-headless:9092 --group source_nginx --describe
查看kafka topic消息数量

#time -1 当前topic 当前分区最大位移 总消息
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic topic_name --time -1
#time -2 当前topic 当前分区最早位移 历史消息
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic topic_name --time -2
#分区当前的消息总数 time-1] - [time-2]

kafka-run-class.sh

主题日志文件

[root@bigdata-kafka-01 v2xRTData-0]# ll
total 3155104
-rw-r--r--. 1 root root    1681880 Oct 23 05:22 00000000002293357422.index
-rw-r--r--. 1 root root 1073741401 Oct 23 05:22 00000000002293357422.log
-rw-r--r--. 1 root root    2487492 Oct 23 05:22 00000000002293357422.timeindex
-rw-r--r--. 1 root root    1474448 Oct 23 13:04 00000000002295695595.index
-rw-r--r--. 1 root root 1073734624 Oct 23 13:04 00000000002295695595.log
-rw-r--r--. 1 root root         10 Oct 23 05:22 00000000002295695595.snapshot
-rw-r--r--. 1 root root    2166228 Oct 23 13:04 00000000002295695595.timeindex
-rw-r--r--. 1 root root   10485760 Oct 23 15:31 00000000002298108844.index
-rw-r--r--. 1 root root  613704731 Oct 23 15:31 00000000002298108844.log
-rw-r--r--. 1 root root         10 Oct 23 13:04 00000000002298108844.snapshot
-rw-r--r--. 1 root root   10485756 Oct 23 15:31 00000000002298108844.timeindex

查看kafka 日志文件内容

[root@bigdata-kafka-01 v2xRTData-0]# kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000002293357422.index --print-data-log
Dumping 00000000002293357422.index
offset: 2293357443 position: 7843
offset: 2293357457 position: 15235
offset: 2293357470 position: 20490
offset: 2293357496 position: 26593
offset: 2293357498 position: 32649

[root@bigdata-kafka-01 v2xRTData-0]# kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000002293357422.timeindex --print-data-log
Dumping 00000000002293357422.timeindex
timestamp: 1697957849928 offset: 2293357439
timestamp: 1697957850028 offset: 2293357456
timestamp: 1697957850128 offset: 2293357468
timestamp: 1697957850328 offset: 2293357496


[root@bigdata-kafka-01 v2xRTData-0]# kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000002293357422.log --print-data-log

Dumping 00000000002293357422.log
Starting offset: 2293357422
offset: 2293357422 position: 0 CreateTime: 1697957849828 isvalid: true keysize: -1 valuesize: 415 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] payload: {"device":"p13r","ip":"10.100.17.20","payload":{"id":"0","type":"rtd","packagesn":362306,"mac":"B8:BB:23:80:19:64","content":{"longitude":1165847468,"latitude":397788335,"vehicle_speed":0,"vehicle_accel":0,"altitude":275,"vehicle_angle":8274,"angle_speed":0,"slope":0,"star_num":25,"acc_state":1,"timestamp":1697957849900,"avg_speed":40,"vehicle_state":0,"gps_angle":8275,"mileage":217871706}},"time":1697957849749}
offset: 2293357423 position: 0 CreateTime: 1697957849828 isvalid: true keysize: -1 valuesize: 417 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] payload: {"device":"p13r","ip":"10.100.17.14","payload":{"id":"0","type":"rtd","packagesn":362306,"mac":"B8:BB:23:80:19:67","content":{"longitude":1165232575,"latitude":398030805,"vehicle_speed":0,"vehicle_accel":0,"altitude":331,"vehicle_angle":32390,"angle_speed":0,"slope":0,"star_num":28,"acc_state":1,"timestamp":1697957849900,"avg_speed":39,"vehicle_state":0,"gps_angle":32391,"mileage":187839132}},"time":1697957849749}
offset: 2293357424 position: 0 CreateTime: 1697957849828 isvalid: true keysize: -1 valuesize: 427 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] payload: {"device":"p13r","ip":"10.100.17.2","payload":{"id":"0","type":"rtd","packagesn":362307,"mac":"B8:BB:23:80:19:2B","content":{"longitude":1165010893,"latitude":398148875,"vehicle_speed":6193,"vehicle_accel":-205,"altitude":323,"vehicle_angle":15963,"angle_speed":-54,"slope":47,"star_num":29,"acc_state":1,"timestamp":1697957849900,"avg_speed":6198,"vehicle_state":1,"gps_angle":15963,"mileage":146900431}},"time":1697957849749}

查看主题各个分区offset

[root@bigdata-kafka-01 v2xRTData-0]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list bigdata-kafka-01:9092 --topic euhtTerminalData
euhtTerminalData:0:41318
euhtTerminalData:1:59238
euhtTerminalData:2:8689

kafka-configs

压测

kafka-producer-perf-test --topic test --num-records 100 --record-size 1 --throughput 100  --producer-props bootstrap.servers=kafka-headless:9092

kafka-verifiable-producer --broker-list kafka-headless:9092 --topic test --max-messages 100000

—>>>重分区

kafka-reassign-partitions

https://blog.csdn.net/feelwing1314/article/details/81047613

修改topic 副本

root@kafka-1:/# kafka-topics  --describe --topic aaa --zookeeper kafka-zookeeper-headless:2181
Topic:aaa       PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: aaa      Partition: 0    Leader: 1       Replicas: 1     Isr: 1
...
root@kafka-1:/# cat << EOF > increate_replica_aaa.sh
> {"version": 1,"partitions": [{"topic": "aaa","partition": 0,"replicas": [0, 1, 2]}]}
> EOF
root@kafka-1:/# kafka-reassign-partitions --zookeeper kafka-zookeeper-headless:2181 --reassignment-json-file increate_replica_aaa.                                                                            sh --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"aaa","partition":0,"replicas":[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
root@kafka-1:/# kafka-topics  --describe --topic aaa --zookeeper kafka-zookeeper-headless:2181
Topic:aaa       PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: aaa      Partition: 0    Leader: 1       Replicas: 0,1,2 Isr: 1,0,2