kafka报错整理

kafka报错整理

基本都是内网报错，这里纯手敲，老爷要是懂了点个赞👍再走呗，砰砰砰

分区消费再平衡

1.表现：自动提交偏移量消费时，有的成功有的失败。失败时报错：
Auto offset commit failed for group XXXXX-test-1: Commit cannot be complete since the group has already rebalanced and assigned the partitions to another member.This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms which typically implies that the poll loop is spending to much time message processing.You can address this either by increasing this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
原因：
kafka集群中，每个broker都有一个coordinator，它用来实现消费者的初始化和分区分配。在初始化完成后，消费者组的每个消费者都会和coordinator保持心跳（默认3s），一旦超时（session.timeout.ms=45）或者消费者处理时间太长了（max.poll.interval.ms=5min），该消费者会被干掉，消费者组执行再平衡。平衡后的partition需要分配给各其他消费者进行消费，被干掉的消费者也肯定不能提交偏移量。
结合测试环境的操作来看的话，之前不是压测，消费的速度很快，这次秒提交1000条大报文，下游数据消费的慢了，就报出以上问题，按照报错提示提高session.timeout.ms可以解决问题