zookeeper频繁异常问题分析

最新推荐文章于 2023-08-05 17:07:25 发布

xjping0794

最新推荐文章于 2023-08-05 17:07:25 发布

阅读量10w+

点赞数 10

分类专栏： zookeeper 文章标签： zookeeper 异常

本文链接：https://blog.csdn.net/xjping0794/article/details/77784171

版权

博客分析了Zookeeper服务频繁异常的原因，主要问题在于Zookeeper在同步日志时耗时过长，导致session过期和服务端shutdown。此外，磁盘IO过高也是问题之一，尤其是在进行大数据作业时。建议关注磁盘IO性能，并考虑关闭forceSync参数以减少延迟，同时注意这可能导致的数据一致性风险。已知的CancelledKeyException在新版本Zookeeper中已被修复。

摘要由CSDN通过智能技术生成

1.1 操作系统信息

1.1.2 内存信息

CM监控界面并无内存信息，无法提供图片。

但问题期间，监控机器内存，发现还有剩余，故排除内存问题。

1.2 ZKFAILOVER日志信息

2017-09-0109:45:57,390 INFO org.apache.zookeeper.ClientCnxn: Client session timed out,have not heard from server in 1668ms for sessionid 0x0, closing socketconnection and attempting reconnect

2017-09-01 09:45:58,224 ERRORorg.apache.hadoop.ha.ActiveStandbyElector: Connection timed out: couldn'tconnect to ZooKeeper in 5000 milliseconds

2017-09-0109:45:58,450 INFO org.apache.zookeeper.ZooKeeper: Session: 0x0 closed

2017-09-0109:45:58,450 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down

2017-09-0109:45:58,451 WARN org.apache.hadoop.ha.ActiveStandbyElector:org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =ConnectionLoss

2017-09-0109:46:03,453 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection,connectString=test-ssps-s-02:2181,test-ssps-s-03:2181,test-ssps-s-04:2181sessionTimeout=5000watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@7f0d36c

2017-09-0109:46:03,455 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection toserver test-ssps-s-04/10.117.210.216:2181. Will not attempt to authenticateusing SASL (unknown error)

2017-09-0109:46:03,463 INFO org.apache.zookeeper.ClientCnxn: Socket connectionestablished to test-ssps-s-04/10.117.210.216:2181, initiating session

2017-09-0109:46:05,131 INFO org.apache.zookeeper.ClientCnxn: Client session timed out,have not heard from server in 1668ms for sessionid 0x0, closing socketconnection and attempting reconnect

2017-09-0109:46:05,885 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection toserver test-ssps-s-03/10.51.20.155:2181. Will not attempt to authenticate usingSASL (unknown error)

2017-09-0109:46:06,626 INFO org.apache.zookeeper.ClientCnxn: Socket connectionestablished to test-ssps-s-03/10.51.20.155:2181, initiating session

2017-09-0109:46:08,293 INFO org.apache.