1.集群部署规划
hadoop1 | hadoop2 | hadoop3 | |
---|---|---|---|
HDFS | NameNode,DataNode | DataNode | SecondaryNameNode,DataNode |
YARN | NodeManager | ResourceManager,NodeManager | NodeManager |
- NameNode和SecondaryNameNode不在同一台服务器上
- ResourceManager和NameNode,SecondaryNameNode不在同一台服务器上
2.在hadoop1上配置core-site.xml
hadoop-3.2.3/etc/hadoop/core-site.xml
<configuration>
<!--指定NN的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop1:8020</value>
</property>
<!--指定hadoop数据存储目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/software/hadoop-3.2.3/data</value>
</property>
</configuration>
3.在hadoop1上配置hdfs-site.xml
hadoop-3.2.3/etc/hadoop/hdfs-site.xml
<configuration>
<!-- nn web端访问地址 -->
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop1:9870</value>
</property>
<!-- 2nn web端访问地址 -->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop3:9868</value>
</property>
</configuration>
4.在hadoop1上配置yarn-site.xml
hadoop-3.2.3/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- 指定MR走shuffle -->
<property>
<name>yarn.namemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop2</value>
</property>
</configuration>
5.在hadoop1上配置mapred-site.xml
hadoop-3.2.3/etc/hadoop/yarn-site.xml
<configuration>
<!-- 指定MR程序运行在yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
6.配置workers
hadoop-3.2.3/etc/hadoop/workers
hadoop1
hadoop2
hadoop3
主机名称后不能有空格
分发workers
xsync workers
7.集群第一次启动格式化NameNode
hdfs namenode -format
8.启动HDFS
start-dfs.sh
9.启动YARN
在hadoop2上启动YARN
start-yarn.sh
10.集群崩溃的恢复
先杀进程(先yarn再dfs),再删所有的data与logs,最后格式化即可
11.配置历史服务器
11.1.在mapred-site.xml中配置历史服务器相关信息
<!-- 历史服务器地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop1:10020</value>
</property>
<!-- 历史服务器web端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop1:19888</value>
</property>
11.2.分发mapred-site.xml
xsync mapred-site.xml
11.3.启动历史服务器
bin/mapred --daemon start historyserver