[hadoop]namenode单节点启动成功后自动消失/格式化失败/fsimage加载失败

问题描述；

集群 start-all.sh 启动后，发现少了一个namenode。

单节点namenode启动成功，但是启动其他节点后，jps发现namenode消失了

格式化namenode报错

问题排查与解析

解决思路

---

1、单节点启动排查。

整体关闭stop-all.sh，

单节点启动 hadoop-daemon.sh start namenode；jps namenode单个节点启动成功，

再次启动下一个hadoop-daemon.sh start  datanode;jps后，发现namenode 消失了

namenode启动失败

2、 通过logs查看问题

2022-06-02 06:30:39,722 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.io.IOException: Failed to load FSImage file, see error(s) above for more info.

        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)

        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)

报错的原因：fsimage的镜像文件加载失败

基本定位到问题

3、解决方案

试图通过格式化 hadoop namenode -format解决，结果仍旧报错

 通namenode(active)的节点，拷贝 dfs的文件到挂掉的节点。scp -r dfs/ hadoop001:`pwd`

[root@hadoop002 data]# scp -r dfs/ hadoop001:`pwd`

root@hadoop001's password: (输入密码)

再次start-all.sh

[root@hadoop001 logs]# start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [hadoop001 hadoop002]

hadoop001: starting namenode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop001.out

hadoop002: starting namenode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop002.out

hadoop001: starting datanode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop001.out

hadoop003: starting datanode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop003.out

hadoop002: starting datanode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop002.out

Starting journal nodes [hadoop001 hadoop002 hadoop003]

hadoop003: starting journalnode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-journalnode-hadoop003.out

hadoop001: starting journalnode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-journalnode-hadoop001.out

hadoop002: starting journalnode, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-journalnode-hadoop002.out

Starting ZK Failover Controllers on NN hosts [hadoop001 hadoop002]

hadoop001: starting zkfc, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-zkfc-hadoop001.out

hadoop002: starting zkfc, logging to /opt/module/hadoop-2.7.3/logs/hadoop-root-zkfc-hadoop002.out

starting yarn daemons

starting resourcemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-resourcemanager-hadoop001.out

hadoop002: starting nodemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop002.out

hadoop001: starting nodemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop001.out

hadoop003: starting nodemanager, logging to /opt/module/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop003.out

通过jps查看

[root@hadoop001 logs]# jps

106723 DFSZKFailoverController

105860 NameNode

106052 DataNode

107065 NodeManager

106427 JournalNode

106939 ResourceManager

12126 QuorumPeerMain

107182 Jps

问题解决！