背景说明
kafka客户端程序异常导致消费积压,再恢复时遗留的历史消息太多,而追赶到最新消息需费时太久,无法贴合业务使用场景;
和业务部门协商确定: 可以丢弃中间未消费的数据,将指定offset(指定消费组消费topic)调整到最新值;
【kafka-消费堆积】
告警时间: 2020.04.02 10:52:22
主机名称: kafka001
[主题] : heartbeat_sync
消费组: heart-consumer
触发器: >20000
最近值: 36726011
操作过程
查看主题、消费组列表
# 打印group列表
/usr/local/confluent-5.0.0/bin/kafka-consumer-groups --bootstrap-server 172.19.94.232:9092 --list
auth-consumer
heart-consumer
wxfile-uploadoss
pdd-consumer
c_api20_merchant_file_download
# 打印topic列表
/usr/local/confluent-5.0.0/bin/kafka-topics --zookeeper localhost:2181 --list
AuthStatisticTopic
ChatMessage
ChatRoom
FileUploadOss
Friend
GameAuthTopic
HeartSyncTopic
MerchantTopic
Moments
PinDuoDuoTopic
PinDuoDuoTopic1.0
#查看消费组offset
/usr/local/confluent-5.0.0/bin/kafka-consumer-groups --describe --group heart-consumer --bootstrap-server 172.19.94.232:9092
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
HeartSyncTopic 3 - 860806 - rdkafka-f49a06fe-55c5-43d4-a743-07b02a8cddd0 /172.19.94.235 rdkafka
HeartSyncTopic 4 - 977386 - rdkafka-f49a06fe-55c5-43d4-a743-07b02a8cddd0 /172.19.94.235 rdkafka
HeartSyncTopic 5 - 855648 - rdkafka-f49a06fe-55c5-43d4-a743-07b02a8cddd0 /172.19.94.235 rdkafka
HeartSyncTopic 0 - 1085257 - rdkafka-04413f4f-85d9-4ac2-85d1-8e631ee4ff3e /172.18.20.66 rdkafka
HeartSyncTopic 1 - 918835 - rdkafka-04413f4f-85d9-4ac2-85d1-8e631ee4ff3e /172.18.20.66 rdkafka
HeartSyncTopic 2 - 946388 - rdkafka-04413f4f-85d9-4ac2-85d1-8e631ee4ff3e /172.18.20.66 rdkafka
heartbeat_sync_pdd_game 0 6821 719589 712768 - - -
注:【重设位移必须要停止消费者】查看到GROUP还有在消费其它topic消息,此时直接修改offset会提示如下报错(kafka防护机制),必须要先停掉客户端(以上HOST机器)的GROUP所有的消费才能修改。
Error: Assignments can only be reset if the group 'flumeGroup' is inactive, but the current state is Stable.
GROUP: heart-consumer
TOPIC: HeartSyncTopic
TOPIC: heartbeat_sync_pdd_game
调整消费组offset
# 修改到最新offset
/usr/local/confluent-5.0.0/bin/kafka-consumer-groups --bootstrap-server 172.19.94.232:9092 --group heart-consumer --reset-offsets --topic heartbeat_sync_pdd_game --to-latest --execute
Consumer group 'heart-consumer' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
heartbeat_sync_pdd_game 0 6821 720390 713569 - - -
重设位移有几种选项:
--to-earliest: 设置到最早位移处,也就是0
--to-latest: 设置到最新处,也就是主题分区HW的位置
--to-offset NUM: 指定具体的位移位置
--shift-by NUM: 基于当前位移向前回退多少
--by-duration: 回退到多长时间
# 查看当前GROUP消费offset
/usr/local/confluent-5.0.0/bin/kafka-consumer-groups --describe --group heart-consumer --bootstrap-server 172.19.94.232:9092
Consumer group 'heart-consumer' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
heartbeat_sync_pdd_game 1 720393 720402 9 - - -
heartbeat_sync_pdd_game 0 720393 720401 8 - - -
heartbeat_sync_pdd_game 2 720393 720401 8 - - -
消费组参数说明:
CURRENT-OFFSET: 当前消费者位移
LOG-END-OFFSET: 分区最新位移
LAG: LOG-END-OFFSET减去CURRENT-OFFSET的值,表示积压量
CONSUMER-ID: 是Kafka自己生成的
CLIENT-ID: 是消费者代码里写的CLIENT ID,用于区分同消费者组中的不同客户端