kafka异常处理20181128

异常信息

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.

---

异常原因

consumer.poll()的调用之间的最大延迟，由参数max.poll.interval.ms来决定，当我们从kafkaServer端poll消息时，这提供了消费者在获取更多记录之前可以空闲的时间量的上限。 如果在此超时到期之前未调用poll()，则认为使用者失败，并且消费者组将重新平衡以便将分区重新分配给其他消费者，因此发生rebalanced，consumer线程rejoin，出现大量警告和异常。

---

处理方式

1.增大max.poll.interval.ms
2.增大session.timeout.ms
3.减小max.poll.records
4.减小consumer.poll(n)中的参数n