kafka消费慢的问题解决方案

背景  

kafka消费的过程中需要很耗时的IO请求处理，会导致两个问题

（1）消费者的被误认为是挂了，导致rebalance，

（2）同步处理，导致很多数据积压，

（3）若是异步处理，大的并发导致IO请求的集群性能达到瓶颈，大量IO请求失败。

问题记录探讨

1. 若同一个group offset之后的已经提交，之前的没有提交，kafka如何处理？
2. kafka消费慢导致异常Commit cannot be completed since the group has already rebalanced and assigned the partitions，是如何产生，有如何解决的
3. 显示异常

   CommitFailedError: Commit cannot be completed since the group has already

               rebalanced and assigned the partitions to another member.

               This means that the time between subsequent calls to poll()

               was longer than the configured max_poll_interval_ms, which

               typically implies that the poll loop is spending too much

               time message processing. You can address this either by

               increasing the rebalance timeout with max_poll_interval_ms,

               or by reducing the maximum size of batches returned in poll()

               with max_poll_records.

rebalanced

解决方案：

整理后给出

参考https://www.cnblogs.com/hapjin/archive/2019/06/01/10926882.html