1.spark streaming 持续如下打印日志,无法进行任务处理,并无报错
AbstractCoordinator: (Re-)joining group testp
AbstractCoordinator: (Re-)joining group testp
AbstractCoordinator: (Re-)joining group testp
2.查看zookeeper.out发现consumer消费offset 出现问题,部分贴出:
2018-04-20 14:39:10,158 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c72 zxid:0x23ba txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/6/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/6/state
2018-04-20 14:39:10,161 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c74 zxid:0x23bb txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/10/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/10/state
2018-04-20 14:39:10,164 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c76 zxid:0x23bc txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/20/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/20/state
2018-04-20 14:39:10,167 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@651] - Got user-level KeeperException when processing sessionid:0x162e0b4e63b001f type:setData cxid:0x2c78 zxid:0x23bd txntype:-1 reqpath:n/a Error Path:/kafka10/brokers/topics/__consumer_offsets/partitions/4/state Error:KeeperErrorCode = BadVersion for /kafka10/brokers/topics/__consumer_offsets/partitions/4/state
原因:kafka中间出现过网络中断问题,导致leader,follower,follower不断进行协调消息,offset管理
解决: 确保网络正常后, 重启kafka broker ,kafka producer,用到的spark-streaming程序,
hdfs集群(如有必要)