Kafka 网络中断导致leader,follower不断进行协调,导致无法处理消息

1.spark streaming 持续如下打印日志,无法进行任务处理,并无报错

AbstractCoordinator: (Re-)joining group testp

AbstractCoordinator: (Re-)joining group testp

AbstractCoordinator: (Re-)joining group testp

 

 2.查看zookeeper.out发现consumer消费offset 出现问题,部分贴出:

2018-04-20 14:39:10,158 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c72 zxid:0x23ba txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/6/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/6/state
2018-04-20 14:39:10,161 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c74 zxid:0x23bb txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/10/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/10/state
2018-04-20 14:39:10,164 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c76 zxid:0x23bc txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/20/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/20/state
2018-04-20 14:39:10,167 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c78 zxid:0x23bd txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/4/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/4/state
 

 

 

原因:kafka中间出现过网络中断问题,导致leader,follower,follower不断进行协调消息,offset管理

解决: 确保网络正常后,  重启kafka broker ,kafka producer,用到的spark-streaming程序,

hdfs集群(如有必要)