YARN集群部署

集群部署规划

 一、MapReduce配置文件

先进入hadoop里面看配置文件

1.配置mapred-env.sh 

 填入如下内容

2.配置mapred-site.xml 

配置代码：

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <description>MapReduce的运行框架设置为YARN</description>
  </property>

  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>hadoop1:10020</value>
    <description>历史服务器通讯端口为hadoop1:10020</description>
  </property>


  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>hadoop1:19888</value>
    <description>历史服务器web端口hadoop1的19888</description>
  </property>


  <property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>/data/mr-history/tmp</value>
    <description>历史信息在hdfs的记录临时路径</description>
  </property>


  <property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>/data/mr-history/done</value>
    <description>历史信息在HDFS的记录路径</description>
  </property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
  <description>MapReduce_HOME设置为HADOOP_HOME</description>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
  <description>MapReduce_HOME设置为HADOOP_HOME</description>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
  <description>MapReduce_HOME设置为HADOOP_HOME</description>
</property>
</configuration>

 在此处填入上述代码：

 3.配置yarn-env.sh文件

 4.配置yarn-site.xml文件

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.log.server.url</name>
    <value>http://node1:19888/jobhistory/logs</value>
    <description>历史服务器URL</description>
</property>

  <property>
    <name>yarn.web-proxy.address</name>
    <value>hadoop1:8089</value>
    <description>代理服务器主机和端口</description>
  </property>


  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
    <description>开启日志聚合</description>
  </property>

  <property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
    <description>程序日志HDFS的存储路径</description>
  </property>


<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop1</value>
    <description>resourceManager设置在hadoop1节点</description>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description>选择公平调度器</description>
  </property>

  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/data/nm-local</value>
    <description>NodeManager中间数据本地存储路径</description>
  </property>


  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/data/nm-log</value>
    <description>NodeManager数据日志本地存储路径</description>
  </property>


  <property>
    <name>yarn.nodemanager.log.retain-seconds</name>
    <value>10800</value>
    <description>Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.</description>
  </property>



  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    <description>为mapreduce程序开发开启shuffle服务</description>
  </property>
</configuration>

在红框处插入上述代码

5.分发配置文件

 6.启动相关进程

 历史服务器启动

7.查看YARN  WEBUI界面 

hadoop:8088