“Unable to load database on disk” 错误的解决办法
我的hadoop集群是利用VMware Workstation部署的,今天在启动的时候,发现slave2节点的zookeeper启动出现异常,使用zkServer.sh start 命令启动,显示启动成功,但是使用jps查看进程,发现没有zookeeper的进程QuorumPeerMain。
查看日志发现报了如下错误:
[hadoop@slave2 ~]$ tail -n 300 zookeeper.out
2021-06-10 21:01:17,121 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /home/hadoop/app/zookeeper/bin/../conf/zoo.cfg
2021-06-10 21:01:17,126 [myid:] - INFO [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2021-06-10 21:01:17,130 [myid:3] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2021-06-10 21:01:17,130 [myid:3] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2021-06-10 21:01:17,130 [myid:3] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2021-06-10 21:01:17,142 [myid:3] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
2021-06-10 21:01:17,154 [myid:3] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2021-06-10 21:01:17,176 [myid:3] - INFO [main:QuorumPeer@959] - tickTime set to 2000
2021-06-10 21:01:17,176 [myid:3] - INFO [main:QuorumPeer@979] - minSessionTimeout set to -1
2021-06-10 21:01:17,177 [myid:3] - INFO [main:QuorumPeer@990] - maxSessionTimeout set to -1
2021-06-10 21:01:17,177 [myid:3] - INFO [main:QuorumPeer@1005] - initLimit set to 10
2021-06-10 21:01:17,189 [myid:3] - INFO [main:FileSnap@83] - Reading snapshot /home/hadoop/data/zookeeper/zkdata/version-2/snapshot.500000002
2021-06-10 21:01:17,690 [myid:3] - ERROR [main:QuorumPeer@497] - Unable to load database on disk
java.io.IOException: The current epoch, 6, is older than the last zxid, 30064771083
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:479)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:440)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2021-06-10 21:01:17,692 [myid:3] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:498)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:440)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.io.IOException: The current epoch, 6, is older than the last zxid, 30064771083
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:479)
... 4 more
然后在老外的一个网站中找到了答案:https://issues.apache.org/jira/browse/ZOOKEEPER-1546
他的解决办法是:
Removing data from /var/zookeeper/version-2 then restart seems to “fix” the problem (it gets a snapshot from one of the other nodes in the quorum).
This is Zookeeper 3.3.5+19.5-1~squeeze-cdh3, i.e. from Cloudera’s distribution.
根据他的描述,要删掉version-2这个目录,但是我忘记了这个目录在哪,就看了一下zoo.cfg的配置文件,找到datadir的位置,就能找到version-2的位置:
我的version-2在/home/hadoop/data/zookeeper/zkdata目录下,在这个目录下找到version-2这个目录,我没有删掉他,给他做了个副本,把之前version-2这个文件重命名为version-2.bak,然后重新建了一个version-2的目录,然后再次启动zookeeper,就启动成功了,使用jps也能查到zookeeper的进程QuorumPeerMain。