kafka常用命令摘选

以 kafka 2.2.x 为基础，假定当前已位于kafka工作目录下。

新老版本区别

若果是老版本的消费者，将 –bootstrap-server 换成 –zookeeper

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list

查看topic分区信息

• –time：-1表示显示获取当前offset最大值，-2表示offset的最小值。

bin/kafka-run-class.sh \
kafka.tools.GetOffsetShell  \
--broker-list localhost:9092 \
--topic my-topic \
--time -1

添加topic

bin/kafka-topics.sh \
--bootstrap-server broker_host:port \
--create --topic my_topic_name \
--partitions 20 \
--replication-factor 3 \
--config x=y

删除topic

bin/kafka-topics.sh \
--bootstrap-server broker_host:port \
--delete \
--topic my_topic_name

topic增加分区

bin/kafka-topics.sh \
--bootstrap-server broker_host:port \
--alter --topic my_topic_name \ 
--partitions 40

topic添加配置

bin/kafka-configs.sh \
--bootstrap-server broker_host:port \
--entity-type topics \
--entity-name my_topic_name \
--alter --add-config x=y

topic删除配置

bin/kafka-configs.sh \
--bootstrap-server broker_host:port \
--entity-type topics \
--entity-name my_topic_name \
--alter --delete-config x

查看已有的消费者组

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

consumer-group-1
consumer-group-2
consumer-group-3

查看消费者组下所有消费者的消费情况

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
 
TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
topic3          0          241019          395308          154289          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2
topic2          1          520678          803288          282610          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2
topic3          1          241018          398817          157799          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2
topic1          0          854144          855809          1665            consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1      consumer1
topic2          0          460537          803290          342753          consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1      consumer1
topic3          2          243655          398812          155157          consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1      consumer4

查看消费者组下的所有消费者

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members

CONSUMER-ID                                    HOST            CLIENT-ID       #PARTITIONS
consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /127.0.0.1      consumer1       2
consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /127.0.0.1      consumer4       1
consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /127.0.0.1      consumer2       3
consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee /127.0.0.1      consumer3       0

删除消费者组

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group

Deletion of requested consumer groups ('my-group', 'my-other-group') was successful.

重置消费者位移

多种重置方式可选：

• -to-datetime <String: datetime>
  Reset offsets to offsets from datetime. Format: 'YYYY-MM-DDTHH:mm:SS.sss'
• –to-earliest
  Reset offsets to earliest offset.
• –to-latest
  Reset offsets to latest offset.
• –shift-by <Long: number-of-offsets>
  Reset offsets shifting current offset by 'n', where 'n' can be positive or negative.
• –from-file
  Reset offsets to values defined in CSV file.
• –to-current
  Resets offsets to current offset.
• –by-duration <String: duration>
  Reset offsets to offset by duration from current timestamp. Format: 'PnDTnHnMnS'
• –to-offset
  Reset offsets to a specific offset.

# 重置到最新的偏移
bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--group consumergroup1 \
--topic topic1 \
--reset-offsets \
--to-latest \
--execute