前言
Hadoop单节点伪分布部署参考这篇,本篇来进行真正多节点部署。
角色分配
cdh01,cdh02,cdh03
node | cdh01 | cdh02 | cdh03 |
---|---|---|---|
HDFS | NameNode、DataNode | SecondNameNode、DataNode | DataNode |
YARN | ResourceManager、NodeManager | NodeManager | NodeManager |
解压安装及配置
- cdh01:
tar -zxvf hadoop-2.6.0-cdh5.15.1.tar.gz -C /opt/
!!!修改配置文件
1.修改HDFS配置文件 /opt/hadoop-2.6.0-cdh5.15.1/etc/hadoop路径下面
hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_261
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://cdh01:8020</value> # 写namenode的ip和端口,8020和9000都可以,只要其他组件对应
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-2.6.0-cdh5.15.1/tmp</value> #创建一个tmp文件夹用来存储临时文件
</property>
hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>3</value> # 副本数量最大为datanode数量,3副本就够了
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cdh02:50090</value>
</property>
slaves #slaves文件,hdfs启动时,通过这个文件来启动datanode,只要启动节点有这个文件就可以
cdh01
cdh02
cdh03
修改YARN配置文件 /opt/hadoop-2.6.0-cdh5.15.1/etc/hadoop路径下面
yarn-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_261
yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>cdh01</value>
</property>
- cdh02-03
# 把hadoop文件传输到其他两台机器上
xsync /opt/hadoop-2.6.0-cdh5.15.1
- 三台机器添加配置文件
vi /etc/profile
# 添加
export HADOOP_HOME=/opt/hadoop-2.6.0-cdh5.15.1
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile
- 启动
在NameNode
机器上
# 第一次启动前,在cdh01上执行初始化
hdfs namenode -format
# 启动
[root@cdh01 sbin] ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [cdh01]
cdh01: starting namenode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-namenode-cdh01.out
cdh03: starting datanode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-datanode-cdh03.out
cdh02: starting datanode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-datanode-cdh02.out
cdh01: starting datanode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-datanode-cdh01.out
Starting secondary namenodes [cdh02]
cdh02: starting secondarynamenode, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/hadoop-root-secondarynamenode-cdh02.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-resourcemanager-cdh01.out
cdh03: starting nodemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-nodemanager-cdh03.out
cdh02: starting nodemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-nodemanager-cdh02.out
cdh01: starting nodemanager, logging to /opt/hadoop-2.6.0-cdh5.15.1/logs/yarn-root-nodemanager-cdh01.out
# jps看一下各节点的进程
[root@cdh01 sbin]# jps
8055 Jps
7454 NameNode
7806 ResourceManager
7902 NodeManager
7551 DataNode
[root@cdh02 ~]# jps
3318 SecondaryNameNode
3223 DataNode
3384 NodeManager
3498 Jps
[root@cdh03 ~]# jps
2480 NodeManager
2599 Jps
2379 DataNode
# 各个节点的进程都启动成功了
看一下webUI:cdh01:50070
# 可以通过命令看一下
hdfs dfsadmin -report
Configured Capacity: 119101992960 (110.92 GB)
Present Capacity: 88014856192 (81.97 GB)
DFS Remaining: 88014819328 (81.97 GB)
DFS Used: 36864 (36 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: xxx.xxx.xxx.xxx:50010 (cdh01)
Hostname: cdh01
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 20862103552 (19.43 GB)
DFS Remaining: 18838548480 (17.54 GB)
DFS Used%: 0.00%
DFS Remaining%: 47.45%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Dec 06 22:08:04 CST 2020
Name: xxx.xxx.xxx.xxx:50010 (cdh02)
Hostname: cdh02
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 5113217024 (4.76 GB)
DFS Remaining: 34587435008 (32.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Dec 06 22:08:05 CST 2020
Name: xxx.xxx.xxx.xxx:50010 (cdh03)
Hostname: cdh03
Decommission Status : Normal
Configured Capacity: 39700664320 (36.97 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 5111816192 (4.76 GB)
DFS Remaining: 34588835840 (32.21 GB)
DFS Used%: 0.00%
DFS Remaining%: 87.12%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Dec 06 22:08:05 CST 2020
# 关闭hadoop
[root@cdh01 sbin]# ./stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [cdh01]
cdh01: stopping namenode
cdh01: stopping datanode
cdh03: stopping datanode
cdh02: stopping datanode
Stopping secondary namenodes [cdh02]
cdh02: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
cdh03: stopping nodemanager
cdh02: stopping nodemanager
cdh01: stopping nodemanager
no proxyserver to stop