kafka rebalance与数据重复消费问题

最新推荐文章于 2024-06-30 13:56:29 发布

Mrgray

最新推荐文章于 2024-06-30 13:56:29 发布

阅读量2.4k

点赞数

分类专栏： java 文章标签： kafka

本文链接：https://blog.csdn.net/Ntozot/article/details/105275039

版权

在消费Kafka数据时，程序由于enable.auto.commit配置为true导致数据重复消费。问题源于消费者max.poll.interval.ms配置，当消费间隔超过300秒，消费者会被认为失败并触发再平衡。解决方案包括手动提交偏移量、调整提交间隔或优化处理速度。如果消费者组内多消费者，rebalance可能导致JVM运行时错误，建议不同任务使用不同消费者组。

摘要由CSDN通过智能技术生成

问题和现象：

某个程序在消费kafka数据时，总是重复消费相关数据，仿佛在数据消费完毕之后，没有提交相应的偏移量。然而在程序中设置了自动提交：enable.auto.commit为true
检查日志，发现日志提示：

2020-03-26 17:20:21.414  WARN 28800 --- [ntainer#2-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, 
groupId=test-consumer-group] Synchronous auto-commit of offsets
 {E2C-GDFS-0=OffsetAndMetadata{offset=9632, leaderEpoch=8, metadata=''}} failed: 
Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. 
This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing.
 You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.