记一次kakfa consumer join group失败导致OOM处理过程

最新推荐文章于 2024-08-20 00:20:13 发布

tracyjuantong

最新推荐文章于 2024-08-20 00:20:13 发布

阅读量8k

点赞数 2

文章标签： kafka java zookeeper

本文链接：https://blog.csdn.net/wwpwly/article/details/122087868

版权

[] 12-22 15:07:16 [org.springframework.kafka.KafkaListenerEndpointContainer#1-1-C-1] [] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-add.tag-17, groupId=add.tag] Join group failed with org.apache.kafka.common.errors.MemberIdRequiredException: The group member needs to have a valid member id before actually entering a consumer group
[] 12-22 15:07:14 [org.springframework.kafka.KafkaListenerEndpointContainer#5-1-C-1] [] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-group_sensitive_action-11, groupId=group_sensitive_action] (Re-)joining group
[] 12-22 15:40:52 [org.springframework.kafka.KafkaListenerEndpointContainer#2-1-C-1] [] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-agent.push-2, groupId=agent.push] Group coordinator IP:9092 (id: 2147483465 rack: null) is unavailable or invalid, will attempt rediscovery
[] 12-22 15:40:45 [org.springframework.kafka.KafkaListenerEndpointContainer#3-0-C-1] [] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-add.friend-4, groupId=add.friend] Group coordinator IP:9092 (id: 2147483464 rack: null) is unavailable or invalid, will attempt rediscovery
[] 12-22 15:07:06 [org.springframework.kafka.KafkaListenerEndpointContainer#4-2-C-1] [] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-group_wework_chat-9, groupId=group_wework_chat] Discovered group coordinator IP:9092 (id: 2147483463 rack: null)
[] 12-22 15:40:08 [kafka-coordinator-heartbeat-thread | group_wework_chat] [] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-group_wework_chat-8, groupId=group_wework_chat] Group coordinator IP:9092 (id: 2147483463 rack: null) is unavailable or invalid, will attempt rediscovery
[] 12-22 15:35:53 [http-nio-54001-exec-10] [] ERROR o.a.coyote.http11.Http11NioProtocol - Failed to complete processing of a request
java.lang.OutOfMemoryError: Metaspace
[] 12-22 15:35:42 [http-nio-54001-exec-34] [] ERROR o.a.coyote.http11.Http11NioProtocol - Failed to complete processing of a request
java.lang.OutOfMemoryError: Metaspace

原因：由于项目数据量大，使用原来的group id进行消费，新进来的消费端，kafka在进行consumer group 分配时候，由于数据量问题，kafka server端重新分配客户端的consumer offset超时导致无法加入之前的消费组。
解决方案：新的服务使用新的group id，启动新的消费端，成功消费数据。
我觉得应该是kafka使用新的group if，消费端最后一个位置进行消费，无需参与跟老的group id重新分配的下标进行消费，应该是老的group id数据积压，导致数据量比较大，我使用命令去访问连接zk获取topic消费数据指标，显示超时。

谁有新的想法，留下评论。