ZK的Error contacting service. It is probably not running问题

记录测试的时候遇到了一个bug

发起请求后返回502 bad gateway（nginx/1.1.19）查看到nginx正常启动后查看后端的日志发现了报错

127.0.0.1       localhost
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

报错显示：dubbo链接zookeeper链接不上,服务不可用

1.排查问题

查看hosts配置正常

root@ubuntu:~# cat /etc/hosts
127.0.0.1       localhost

查看zk的状态

● zk.service - Zookeeper Service
   Loaded: loaded (/etc/systemd/system/zk.service; enabled; vendor preset: enabled)
   Active: activating (start-post) (Result: exit-code) since Mon 2021-10-11 20:17:54 CST; 2s ago
  Process: 9433 ExecStop=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh stop (code=exited, status=0/SUCCESS)
  Process: 9448 ExecStart=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh start (code=exited, status=0/SUCCESS)
 Main PID: 9458 (code=exited, status=1/FAILURE);         : 9485 (sleep)
    Tasks: 1
   Memory: 256.0K
      CPU: 3.695s
   CGroup: /system.slice/zk.service
           └─control
             └─9485 /bin/sleep 4
             
Oct 11 20:17:54 ubuntu systemd[1]: Starting Zookeeper Service...
Oct 11 20:17:54 ubuntu zkServer.sh[9448]: ZooKeeper JMX enabled by default
Oct 11 20:17:54 ubuntu zkServer.sh[9448]: Using config: /opt/zk1/zookeeper-3.4.10/bin/../conf/zoo.cfg
Oct 11 20:17:55 ubuntu zkServer.sh[9448]: Starting zookeeper ... STARTED
Oct 11 20:17:56 ubuntu systemd[1]: zk.service: Main process exited, code=exited, status=1/FAILURE

发现是因为zk没有启动成功导致的

2. 解决问题

找到zookeeper的bin目录 使用./zkServer.sh start 可以正常启动
但是使用./zkServer.sh status的时候发现报错Error contacting service. It is probably not running.

ZooKeeper JMX enabled by default
Using config: /opt/zk1/zookeeper-3.4.10/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.

使用systemctl status zk命令查看状态（灰色小圆点）

● zk.service - Zookeeper Service
   Loaded: loaded (/etc/systemd/system/zk.service; enabled; vendor preset: enabled)
   Active: activating (start-post) (Result: exit-code) since Mon 2021-10-11 20:17:54 CST; 2s ago
  Process: 9433 ExecStop=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh stop (code=exited, status=0/SUCCESS)
  Process: 9448 ExecStart=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh start (code=exited, status=0/SUCCESS)
 Main PID: 9458 (code=exited, status=1/FAILURE);         : 9485 (sleep)
    Tasks: 1
   Memory: 256.0K
      CPU: 3.695s
   CGroup: /system.slice/zk.service
           └─control
             └─9485 /bin/sleep 4
             
Oct 11 20:17:54 ubuntu systemd[1]: Starting Zookeeper Service...
Oct 11 20:17:54 ubuntu zkServer.sh[9448]: ZooKeeper JMX enabled by default
Oct 11 20:17:54 ubuntu zkServer.sh[9448]: Using config: /opt/zk1/zookeeper-3.4.10/bin/../conf/zoo.cfg
Oct 11 20:17:55 ubuntu zkServer.sh[9448]: Starting zookeeper ... STARTED
Oct 11 20:17:56 ubuntu systemd[1]: zk.service: Main process exited, code=exited, status=1/FAILURE

尝试启动systemctl start zk

Job for zk.service failed because the control process exited with error code. See "systemctl status zk.service" and "journalctl -xe" for details.

再次查看systemctl status zk

● zk.service - Zookeeper Service
   Loaded: loaded (/etc/systemd/system/zk.service; enabled; vendor preset: enabled)
   Active: activating (start-post) since Mon 2021-10-11 20:18:16 CST; 1s ago
  Process: 9894 ExecStop=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh stop (code=exited, status=0/SUCCESS)
  Process: 9910 ExecStart=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh start (code=exited, status=0/SUCCESS)
 Main PID: 9920 (java);         : 9953 (sleep)
    Tasks: 26
   Memory: 170.6M
      CPU: 3.268s
   CGroup: /system.slice/zk.service
           ├─9920 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp /opt/zk1/zookeeper-3.4.10/bin/../build/classes:/opt/zk1/zookeeper-3.4.10/bin/../build/lib/*.jar:/
           └─control
             └─9953 /bin/sleep 4

Oct 11 20:18:16 ubuntu systemd[1]: Starting Zookeeper Service...
Oct 11 20:18:16 ubuntu zkServer.sh[9910]: ZooKeeper JMX enabled by default
Oct 11 20:18:16 ubuntu zkServer.sh[9910]: Using config: /opt/zk1/zookeeper-3.4.10/bin/../conf/zoo.cfg
Oct 11 20:18:17 ubuntu zkServer.sh[9910]: Starting zookeeper ... STARTED

发现还是不行。
查看配置文件less zoo.cfg 找到日志路径

dataLogDir=/opt/zk1/zookeeper-3.4.10/datalog

进入/opt/zk1/zookeeper-3.4.10/datalog/version-2 发现有一个日志疑似损坏（log.220770）

-rw-r--r-- 1 root root 67108880 Sep 22 00:21 log.1ce8b6
-rw-r--r-- 1 root root 67108880 Sep 22 02:30 log.1e0dbe
-rw-r--r-- 1 root root 67108880 Sep 22 11:15 log.1e4a05
-rw-r--r-- 1 root root 67108880 Sep 22 20:26 log.1f407d
-rw-r--r-- 1 root root 67108880 Sep 23 02:30 log.20427c
-rw-r--r-- 1 root root 67108880 Sep 23 12:32 log.20ecc9
-rw-r--r-- 1 root root        0 Sep 23 12:33 log.220770

删除此日志 rm log.220770再次启动并查看状态
systemctl start zk systemctl status zk （绿色小圆点）

● zk.service - Zookeeper Service
   Loaded: loaded (/etc/systemd/system/zk.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-10-11 20:19:30 CST; 5s ago
  Process: 11032 ExecStop=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh stop (code=exited, status=0/SUCCESS)
  Process: 11088 ExecStartPost=/bin/sleep 4 (code=exited, status=0/SUCCESS)
  Process: 11047 ExecStart=/opt/zk1/zookeeper-3.4.10/bin/zkServer.sh start (code=exited, status=0/SUCCESS)
 Main PID: 11057 (java)
    Tasks: 28
   Memory: 222.5M
      CPU: 4.229s
   CGroup: /system.slice/zk.service
           └─11057 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp /opt/zk1/zookeeper-3.4.10/bin/../build/classes:/opt/zk1/zookeeper-3.4.10/bin/../build/lib/*.jar:

Oct 11 20:19:25 ubuntu systemd[1]: Starting Zookeeper Service...
Oct 11 20:19:25 ubuntu zkServer.sh[11047]: ZooKeeper JMX enabled by default
Oct 11 20:19:25 ubuntu zkServer.sh[11047]: Using config: /opt/zk1/zookeeper-3.4.10/bin/../conf/zoo.cfg
Oct 11 20:19:26 ubuntu zkServer.sh[11047]: Starting zookeeper ... STARTED
Oct 11 20:19:30 ubuntu systemd[1]: Started Zookeeper Service.

重启后端服务，发现错误解决了。

3.预防

因为某些原因会导致服务异常关闭就会出现同样的问题，可以在重启服务器时可以将zookeeper下data和datalog 目录下的文件清除。
执行一下脚本

rm -rf /opt/zk1/zookeeper-3.4.10/data/*
rm -rf /opt/zk1/zookeeper-3.4.10/datalog/*
systemctl restart zk.service

除此以外还有其他情况会导致此问题

1.防火墙没有关闭。
ubuntu
centOS
2.端口被占用。
netstat -anp | grep 端口号
kill -9 id
3.等问题