Hadoop数据节点DataNode异常

1、问题背景：

操作系统：CentOS7.6 64位 
Hadoop版本：Hadoop2.8.5 
JDK：1.8.0_201

2、问题描述
使用start-dfs.sh、start-yarn.sh脚本启动Hadoop后，通过jps命令检测DataNode节点进程发现没有DataNode进程，只有NameNode，SecondaryNameNode，NodeManager，ResourceManager，Jps5个进程。简而言之就是DataNode进程异常，由于某种原因导致无法启动或者启动后数据节点DataNode进程自动关闭。

3、问题原因

通过查看/opt/modules/hadoop-2.8.5/logs路径下的hadoop-hadoop-datanode-EddieCentOS130.log文件，报如下异常：

2019-06-14 01:06:01,244 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /opt/modules/hadoop-2.8.5/dfsdata/data/in_use.lock acquired by nodename 14087@EddieCentOS130
2019-06-14 01:06:01,245 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/opt/modules/hadoop-2.8.5/dfsdata/data/
java.io.IOException: Incompatible clusterIDs in /opt/modules/hadoop-2.8.5/dfsdata/data: namenode clusterID = CID-1fcd3829-f52d-4544-9d4b-1c0069cbe52a; datanode clusterID = CID-61539443-43aa-48dd-af20-1745cfef2e0b
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:760)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:293)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:409)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:388)
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:556)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1574)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1535)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:382)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:266)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:750)
	at java.lang.Thread.run(Thread.java:748)
2019-06-14 01:06:01,247 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN, trace:
java.lang.Exception
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:210)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.hasBlockPoolId(BPOfferService.java:220)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shouldRetryInit(BPOfferService.java:826)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.shouldRetryInit(BPServiceActor.java:792)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:755)
	at java.lang.Thread.run(Thread.java:748)
2019-06-14 01:06:01,248 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid 2e2dd3bf-a304-4b14-b7cf-9e638a846344) service to EddieCentOS130/192.168.30.130:9000. Exiting. 
java.io.IOException: All specified directories are failed to load.
	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:557)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1574)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1535)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:382)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:266)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:750)
	at java.lang.Thread.run(Thread.java:748)
2019-06-14 01:06:01,248 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid 2e2dd3bf-a304-4b14-b7cf-9e638a846344) service to EddieCentOS130/192.168.30.130:9000
2019-06-14 01:06:01,350 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN, trace:
java.lang.Exception
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:210)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.hasBlockPoolId(BPOfferService.java:220)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolManager.remove(BlockPoolManager.java:90)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1490)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:465)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:527)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:787)
	at java.lang.Thread.run(Thread.java:748)
2019-06-14 01:06:01,351 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid 2e2dd3bf-a304-4b14-b7cf-9e638a846344)
2019-06-14 01:06:01,351 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool ID needed, but service not yet registered with NN, trace:
java.lang.Exception
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:210)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.hasBlockPoolId(BPOfferService.java:220)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdownBlockPool(DataNode.java:1491)
	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.shutdownActor(BPOfferService.java:465)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.cleanUp(BPServiceActor.java:527)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:787)
	at java.lang.Thread.run(Thread.java:748)
2019-06-14 01:06:03,356 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2019-06-14 01:06:03,357 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2019-06-14 01:06:03,433 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at EddieCentOS130/192.168.30.130
************************************************************/

日志文件分析如下： 
首先日志警告，“Failed to add storage directory [DISK]file:/opt/modules/hadoop-2.8.5/dfsdata/data/”添加存储文件失败，失败路劲位于“/opt/modules/hadoop-2.8.5/dfsdata/data/”。紧接着报Java IO异常“java.io.IOException”，异常信息里显示的是，namenode和datanode的clusterID不一致，这些异常就导致了后面提示初始化失败、DataNode进程退出等异常、警告信息。这是因为多次对namenode进行format导致的。将namenode和datanode的clusterID和namespaceID修改一致即可。的确，进行了两次format操作，但是修改过后仍然报同样的问题，通过对比”/opt/modules/hadoop-2.8.5/dfsdata/data/current/VERSION”文件发现VERSION文件里clusterID都是一致的，其他的信息也一致。

4、解决方案
由于无法通过直接修改VERSION文件里clusterID、namespaceID解决问题。于是尝试重命名了一下各个数据节点DataNode“/opt/modules/hadoop-2.8.5/dfsdata/data/”目录下current文件夹，再重新启动hadoop，发现hadoop运行起来了。各个数据节点的DataNode、NodeManager等进程信息正常，查看各个节点的日志，一切显示正常。然后查看了一下“/opt/modules/hadoop-2.8.5/dfsdata/data/”目录，发现自动创建了一个current目录。

5、总结
该问题因为多次对namenode进行format，每一次format主节点NameNode产生新的clusterID、namespaceID，于是导致主节点的clusterID、namespaceID与各个子节点DataNode不一致。当format过后再启动hadoop，hadoop尝试创建新的current目录，但是由于已存在current目录，导致创建失败，最终引起DataNode节点的DataNode进程启动失败，从而引起hadoop集群完全启动失败。因此可以通过直接删除数据节点DataNode的current文件夹，进行解决该问题。