记录一次程序与zookeeper断开session连接的问题

在项目开发过程中，需要与zookeeper建立连接，但是在程序运行过程中，遇到了程序与zookeeper断开连接的问题：

[2019-10-24 10:02:00.531] [Thread-7-SendThread(10.128.138.178:2181)] WARN  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 6354ms for sessionid 0x1023f975777000a
[2019-10-24 10:02:00.531] [Thread-8-SendThread(10.128.138.178:2181)] WARN  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 5919ms for sessionid 0x1023f9757770009
[2019-10-24 10:02:00.531] [Thread-8-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 5919ms for sessionid 0x1023f9757770009, closing socket connection and attempting reconnect
[2019-10-24 10:02:00.531] [Thread-7-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 6354ms for sessionid 0x1023f975777000a, closing socket connection and attempting reconnect
[2019-10-24 10:02:00.633] [Thread-8-EventThread] INFO  cn.cloudwalk.dist.base.handler.AbstractHandler - WatchedEvent state:Disconnected type:None path:null
[2019-10-24 10:02:00.633] [Thread-7-EventThread] INFO  cn.cloudwalk.dist.base.handler.MasterHandler - master exist watch, WatchedEvent is: WatchedEvent state:Disconnected type:None path:null
[2019-10-24 10:02:00.633] [Thread-7-EventThread] INFO  cn.cloudwalk.dist.base.handler.AbstractHandler - WatchedEvent state:Disconnected type:None path:null
[2019-10-24 10:02:00.633] [Thread-8-EventThread] INFO  cn.cloudwalk.dist.base.handler.SlaveHandler - slaveExistsWatcher, WatchedEvent: WatchedEvent state:Disconnected type:None path:null
[2019-10-24 10:02:01.942] [Thread-7-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.128.138.178/10.128.138.178:2181. Will not attempt to authenticate using SASL (unknown error)
[2019-10-24 10:02:01.943] [Thread-7-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 10.128.138.178/10.128.138.178:2181, initiating session
[2019-10-24 10:02:01.949] [Thread-7-SendThread(10.128.138.178:2181)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x1023f975777000a has expired
[2019-10-24 10:02:01.949] [Thread-7-EventThread] INFO  cn.cloudwalk.dist.base.handler.MasterHandler - master exist watch, WatchedEvent is: WatchedEvent state:Expired type:None path:null
[2019-10-24 10:02:01.949] [Thread-7-EventThread] INFO  cn.cloudwalk.dist.base.handler.AbstractHandler - WatchedEvent state:Expired type:None path:null
[2019-10-24 10:02:01.949] [Thread-7-EventThread] ERROR cn.cloudwalk.dist.base.handler.AbstractHandler - Session expired
[2019-10-24 10:02:01.949] [Thread-7-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x1023f975777000a has expired, closing socket connection
[2019-10-24 10:02:01.950] [Thread-7-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x1023f975777000a
[2019-10-24 10:02:02.434] [Thread-8-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.128.138.178/10.128.138.178:2181. Will not attempt to authenticate using SASL (unknown error)
[2019-10-24 10:02:02.434] [Thread-8-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 10.128.138.178/10.128.138.178:2181, initiating session
[2019-10-24 10:02:02.435] [Thread-8-SendThread(10.128.138.178:2181)] WARN  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x1023f9757770009 has expired
[2019-10-24 10:02:02.435] [Thread-8-EventThread] INFO  cn.cloudwalk.dist.base.handler.AbstractHandler - WatchedEvent state:Expired type:None path:null
[2019-10-24 10:02:02.435] [Thread-8-SendThread(10.128.138.178:2181)] INFO  org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x1023f9757770009 has expired, closing socket connection
[2019-10-24 10:02:02.435] [Thread-8-EventThread] ERROR cn.cloudwalk.dist.base.handler.AbstractHandler - Session expired
[2019-10-24 10:02:02.435] [Thread-8-EventThread] INFO  cn.cloudwalk.dist.base.handler.SlaveHandler - slaveExistsWatcher, WatchedEvent: WatchedEvent state:Expired type:None path:null

调查相关资料，发现可能产生问题的原因：

Your client can no longer talk to the ZooKeeper server. The first thing that happened was there was no answer to the heartbeats within the negotiated session timeout:

o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 23789ms for sessionid 0x252f862028c0083, closing socket connection and attempting reconnect

Then when it tried to reconnect, it got a connection refused:

o.a.s.z.ClientCnxn [WARN] Session 0x252f862028c0083 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused

This means either your ZooKeeper server:

• Is not reachable (network connection down)
• Is dead (so nothing is listening on the socket)
• Is GCing itself to death and cannot communicate (although that might have issued a connection timeout error, I'm not sure)

To tell more you will need to check the ZooKeeper server logs on your zookeeper cluster.