当使用Flume的KafkaSource时遇到CommitFailedException，原因是两次poll操作间超过了max.poll.interval.ms设定的时间，导致组重新平衡。报错可能由于消息处理时间过长。解决方案包括减小max.poll.records或增大max.poll.interval.ms。可通过调整Flume配置文件中的相关参数来避免此问题，例如调整batchSize、batchDurationMillis、auto.commit.interval.ms、max.poll.interval.ms和session.timeout.ms。
flume系列之：Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member
一、完整报错
- KafkaSource EXCEPTION, {}
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
二、参数理解
max.poll.interval.ms：
- 两次poll操作允许的最大时间间隔。默认5分钟。两次poll超过此时间间隔，会进行rebalance操作，导致客户端连接失效，无法提交offset信息ÿ
订阅专栏 解锁全文
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
