Hadoop 启动/停止集群和节点的命令

 

 

集群

启动/停止Hadoop集群：start-all.sh    stop-all.sh

1) 启动NameNode, DataNode

2) 启动JournalNode, JournalNode在hdfs-site.xml中指定edits log存储的位置，主备NameNode共享数据，方便同步

3) 启动DFSZKFailoverController, HA会启用ZooKeeper Failover Controller

4) 启动YARN守护进程ResourceManager, NodeManager

启动/停止HDFS集群：start-dfs.sh    stop-dfs.sh

1) 启动NameNode, DataNode

2) 启动JournalNode, JournalNode在hdfs-site.xml中指定edits log存储的位置，主备NameNode共享数据，方便同步

3) 启动DFSZKFailoverController, HA会启用ZooKeeper Failover Controller

启动/停止YARN集群：start-yarn.sh    stop-yarn.sh

据说如果ResourceManager和NameNode不在同一台机器上，那NameNode上无法直接使用start-yarn.sh启动ResourceManager，但NodeManager可以启动，ResourceManager需要在ResourceManager节点上启动，这一点待考证

 

节点

启动NameNode: hadoop-daemon.sh start namenode

启动DataNode: hadoop-daemon.sh start datanode

启动JournalNode节点: hadoop-daemon.sh start journalnode

启动zkfc节点：hadoop-daemon.sh start zkfc

启动ResourceManager: yarn-daemon.sh start resourcemanager

启动NodeManager: yarn-daemon.sh start nodemanager

启动ZooKeeper节点：zkServer.sh start    启动QuorumPeerMain进程

查看ZooKeeper节点状态：zkServer.sh status    查看leader和follower状态

格式化HDFS: hdfs namenode -format

格式化ZKFC：hdfs zkfc -formatZK

查看NameNode和DataNode信息： hdfs dfsadmin -report

查看高可用NameNode状态： hdfs haadmin -getServiceState nn1 (active/standby)

 

Web界面端口

NameNode 50070

DataNode 50075

ResourceManager 8088

NodeManager 8042

更多端口参考

 

安装，查看等常用命令 

查看进程：jps

kill进程：kill -9 pid

内容追加到文件：echo content > file, 如 echo 1 > myid

SSH拷贝：scp,如scp -r /usr/local/hadoop master:/usr/local

关闭防火墙：

查看状态systemctl status firewalld

关闭防火墙systemctl stop firewalld

开机禁用防火墙systemctl disable firewalld

配置免登录：

ssh-keygen -t rsa

ssh-copy-id -i host01

启动集群的过程：

[root@hadoop01 sbin]# sh start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
19/03/12 09:22:53 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop01 hadoop02]
hadoop01: starting namenode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-namenode-hadoop01.out
hadoop02: starting namenode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-namenode-hadoop02.out
hadoop05: starting datanode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop05.out
hadoop07: starting datanode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop07.out
hadoop06: starting datanode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop06.out
hadoop04: starting datanode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop04.out
hadoop08: starting datanode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-datanode-hadoop08.out
Starting journal nodes [hadoop04 hadoop05 hadoop06 hadoop07 hadoop08]
hadoop06: starting journalnode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop06.out
hadoop05: starting journalnode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop05.out
hadoop04: starting journalnode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop04.out
hadoop08: starting journalnode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop08.out
hadoop07: starting journalnode, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop07.out
19/03/12 09:23:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting ZK Failover Controllers on NN hosts [hadoop01 hadoop02]
hadoop01: starting zkfc, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-zkfc-hadoop01.out
hadoop02: starting zkfc, logging to /usr/software/hadoop-2.7.1/logs/hadoop-root-zkfc-hadoop02.out
starting yarn daemons
starting resourcemanager, logging to /usr/software/hadoop-2.7.1/logs/yarn-root-resourcemanager-hadoop01.out
hadoop07: starting nodemanager, logging to /usr/software/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop07.out
hadoop06: starting nodemanager, logging to /usr/software/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop06.out
hadoop08: starting nodemanager, logging to /usr/software/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop08.out
hadoop04: starting nodemanager, logging to /usr/software/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop04.out
hadoop05: starting nodemanager, logging to /usr/software/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop05.out
 

停止集群的过程：

[root@hadoop01 sbin]# sh stop-all.sh 
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
19/03/12 09:19:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [hadoop01 hadoop02]
hadoop01: stopping namenode
hadoop02: stopping namenode
hadoop07: stopping datanode
hadoop08: stopping datanode
hadoop05: stopping datanode
hadoop06: stopping datanode
hadoop04: stopping datanode
Stopping journal nodes [hadoop04 hadoop05 hadoop06 hadoop07 hadoop08]
hadoop06: stopping journalnode
hadoop07: stopping journalnode
hadoop08: stopping journalnode
hadoop05: stopping journalnode
hadoop04: stopping journalnode
19/03/12 09:20:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping ZK Failover Controllers on NN hosts [hadoop01 hadoop02]
hadoop01: stopping zkfc
hadoop02: stopping zkfc
stopping yarn daemons
stopping resourcemanager
hadoop06: stopping nodemanager
hadoop05: stopping nodemanager
hadoop07: stopping nodemanager
hadoop04: stopping nodemanager
hadoop08: stopping nodemanager
no proxyserver to stop
 

 

 

Hadoop Shell命令