zookeeper无法启动：出现Unable to load database on disk报错

“Unable to load database on disk” 错误的解决办法

我的hadoop集群是利用VMware Workstation部署的，今天在启动的时候，发现slave2节点的zookeeper启动出现异常，使用zkServer.sh start 命令启动，显示启动成功，但是使用jps查看进程，发现没有zookeeper的进程QuorumPeerMain。


查看日志发现报了如下错误：

[hadoop@slave2 ~]$ tail -n 300 zookeeper.out 
2021-06-10 21:01:17,121 [myid:] - INFO  [main:QuorumPeerConfig@103] - Reading configuration from: /home/hadoop/app/zookeeper/bin/../conf/zoo.cfg
2021-06-10 21:01:17,126 [myid:] - INFO  [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2021-06-10 21:01:17,130 [myid:3] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2021-06-10 21:01:17,130 [myid:3] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2021-06-10 21:01:17,130 [myid:3] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2021-06-10 21:01:17,142 [myid:3] - INFO  [main:QuorumPeerMain@127] - Starting quorum peer
2021-06-10 21:01:17,154 [myid:3] - INFO  [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2021-06-10 21:01:17,176 [myid:3] - INFO  [main:QuorumPeer@959] - tickTime set to 2000
2021-06-10 21:01:17,176 [myid:3] - INFO  [main:QuorumPeer@979] - minSessionTimeout set to -1
2021-06-10 21:01:17,177 [myid:3] - INFO  [main:QuorumPeer@990] - maxSessionTimeout set to -1
2021-06-10 21:01:17,177 [myid:3] - INFO  [main:QuorumPeer@1005] - initLimit set to 10
2021-06-10 21:01:17,189 [myid:3] - INFO  [main:FileSnap@83] - Reading snapshot /home/hadoop/data/zookeeper/zkdata/version-2/snapshot.500000002
2021-06-10 21:01:17,690 [myid:3] - ERROR [main:QuorumPeer@497] - Unable to load database on disk
java.io.IOException: The current epoch, 6, is older than the last zxid, 30064771083
	at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:479)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:440)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2021-06-10 21:01:17,692 [myid:3] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server 
	at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:498)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:440)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: The current epoch, 6, is older than the last zxid, 30064771083
	at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:479)
	... 4 more

然后在老外的一个网站中找到了答案：https://issues.apache.org/jira/browse/ZOOKEEPER-1546
他的解决办法是：

Removing data from /var/zookeeper/version-2 then restart seems to “fix” the problem (it gets a snapshot from one of the other nodes in the quorum).

This is Zookeeper 3.3.5+19.5-1~squeeze-cdh3, i.e. from Cloudera’s distribution.

根据他的描述，要删掉version-2这个目录，但是我忘记了这个目录在哪，就看了一下zoo.cfg的配置文件，找到datadir的位置，就能找到version-2的位置：

我的version-2在/home/hadoop/data/zookeeper/zkdata目录下，在这个目录下找到version-2这个目录，我没有删掉他，给他做了个副本，把之前version-2这个文件重命名为version-2.bak，然后重新建了一个version-2的目录，然后再次启动zookeeper，就启动成功了，使用jps也能查到zookeeper的进程QuorumPeerMain。