hadoop完全分布式搭建(2)

1.集群部署规划

	hadoop1	hadoop2	hadoop3
HDFS	NameNode,DataNode	DataNode	SecondaryNameNode,DataNode
YARN	NodeManager	ResourceManager,NodeManager	NodeManager

• NameNode和SecondaryNameNode不在同一台服务器上
• ResourceManager和NameNode,SecondaryNameNode不在同一台服务器上

2.在hadoop1上配置core-site.xml

hadoop-3.2.3/etc/hadoop/core-site.xml

<configuration>
  <!--指定NN的地址 -->
  <property>
     <name>fs.defaultFS</name>
     <value>hdfs://hadoop1:8020</value>
  </property>
  
  <!--指定hadoop数据存储目录 -->
  <property>
     <name>hadoop.tmp.dir</name>
     <value>/opt/software/hadoop-3.2.3/data</value>
  </property>
</configuration>

3.在hadoop1上配置hdfs-site.xml

hadoop-3.2.3/etc/hadoop/hdfs-site.xml

<configuration>
   <!-- nn web端访问地址 -->
  <property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop1:9870</value>
  </property>

 <!-- 2nn web端访问地址 -->
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop3:9868</value>
  </property>
</configuration>

4.在hadoop1上配置yarn-site.xml

hadoop-3.2.3/etc/hadoop/yarn-site.xml

<configuration>
<!-- Site specific YARN configuration properties -->
  <!-- 指定MR走shuffle  -->
  <property>
    <name>yarn.namemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  
   <!-- 指定RM的地址 -->
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop2</value>
  </property>
</configuration>

5.在hadoop1上配置mapred-site.xml

hadoop-3.2.3/etc/hadoop/yarn-site.xml

<configuration>
   <!-- 指定MR程序运行在yarn上  -->
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

6.配置workers

hadoop-3.2.3/etc/hadoop/workers

hadoop1
hadoop2
hadoop3

主机名称后不能有空格

分发workers

xsync  workers

7.集群第一次启动格式化NameNode

hdfs namenode -format

8.启动HDFS

start-dfs.sh

9.启动YARN

在hadoop2上启动YARN

start-yarn.sh

10.集群崩溃的恢复

先杀进程(先yarn再dfs)，再删所有的data与logs，最后格式化即可

11.配置历史服务器

11.1.在mapred-site.xml中配置历史服务器相关信息

 <!-- 历史服务器地址 -->
   <property>
    <name>mapreduce.jobhistory.address</name>
    <value>hadoop1:10020</value>
  </property> 
  
  <!-- 历史服务器web端地址 -->
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop1:19888</value>
  </property>

11.2.分发mapred-site.xml

xsync mapred-site.xml

11.3.启动历史服务器

bin/mapred --daemon start historyserver