org.apache.kafka.clients.consumer.CommitFailedException

最新推荐文章于 2024-11-04 21:20:47 发布

优优我心

最新推荐文章于 2024-11-04 21:20:47 发布

阅读量1.5k

点赞数

本文链接：https://blog.csdn.net/weixin_41608066/article/details/111714189

版权

本文介绍了在迁移Spark Streaming任务至Flink过程中遇到的`org.apache.kafka.clients.consumer.CommitFailedException`异常。问题根源在于Spark的subscribe模式与Flink的assign模式不兼容。解决方法是创建新groupid，复制旧groupid的offset，并用Flink以新groupid消费。详细解决方案包含使用python-kafka脚本进行offset复制。

摘要由CSDN通过智能技术生成

最近遇到一个kafka分区提异常如下：

throwable:org.apache.flink.kafka.shaded.org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
at org.apache.flink.kafka.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:1151)
at org.apache.flink.kafka.shaded.org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.