今天启动hadoop集群后,查看进程发现名称节点的NameNode进程启动了,但是数据节点的DataNode进程没有启动,如下:
master上
# jps
27585 Jps
26874 ResourceManager
26622 NameNode
slave1上
# jps
25792 SecondaryNameNode
25988 Jps
25862 NodeManager
slave2上
# jps
25276 NodeManager
25581 Jps
原因为多次格式化namenode导致的namenode与datanode之间的不一致导致。
解决方法
(1)停止集群
# stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
slave2: no datanode to stop
slave1: no datanode to stop
Stopping secondary namenodes [slave1]
slave1: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
slave2: stopping nodemanager
slave1: stopping nodemanager
no proxyserver to stop
(2)删除datanode存放的路径的数据
可以在hdfs-site.xml中查看存放的路径,如下:
# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
# rm -rf /usr/local/hadoop/data/* #三个节点都操作
(3)重新格式化
# hadoop namenode -format
(4)启动集群
# start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-namenode-master.out
slave2: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-slave2.out
slave1: starting datanode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-slave1.out
Starting secondary namenodes [slave1]
slave1: starting secondarynamenode, logging to /usr/local/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-slave1.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-resourcemanager-master.out
slave2: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-slave2.out
slave1: starting nodemanager, logging to /usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-slave1.out
(4)查看进程
master 上
# jps -m
28708 Jps -m
28380 ResourceManager
28126 NameNode
slave1上
# jps -m
26848 Jps -m
26626 NodeManager
26557 SecondaryNameNode
26462 DataNode
slave2上
# jps -m
25810 NodeManager
25996 Jps -m
25709 DataNode
(5)查看集群状态
# hadoop fsadmin -report
Error: Could not find or load main class fsadmin
[root@master hadoop]# hadoop dfsadmin -report
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Configured Capacity: 33335099392 (31.05 GB)
Present Capacity: 26244313088 (24.44 GB)
DFS Remaining: 26244263936 (24.44 GB)
DFS Used: 49152 (48 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 192.168.22.129:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 16667549696 (15.52 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 3545563136 (3.30 GB)
DFS Remaining: 13121961984 (12.22 GB)
DFS Used%: 0.00%
DFS Remaining%: 78.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Aug 27 00:02:44 CST 2019
Name: 192.168.22.130:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 16667549696 (15.52 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 3545223168 (3.30 GB)
DFS Remaining: 13122301952 (12.22 GB)
DFS Used%: 0.00%
DFS Remaining%: 78.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Tue Aug 27 00:02:44 CST 2019
查看进程,DataNode进程已正常启动,集群状态正常。