Hadoop HA + Federation + YARN HA配置实现

集群规划

配置步骤
一、 前提准备
1） 安装好Hadoop2.0集群 《==参考《HADOOP HA+Federation集群实现（七）》及以前的文章
2） 安装好zookeeper集群
二、修改如下配置文件
1） yarn-site.xml
2） mapred-site.xml
3） yarn-env.sh
三、启动服务
1） 启动HDFS
2） 启动YARN，通过yarn-daemons.sh来启动
四、测试验证
1） 查看状态
yarn rmadmin -getServiceState rm1/rm2
2） 模拟故障切换


实施步骤
1） 编辑yarn-env.sh
[hadoop@master ~]$ cd /opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ vi yarn-env.sh
export JAVA_HOME=/usr/java/default


2）编辑mapred-site.xml
[hadoop@master hadoop]$ vi mapred-site.xml
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>


3）编辑yarn-site.xml
[hadoop@master hadoop]$ vi yarn-site.xml

<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
  </property>
<!--YARN HA parameter-->
<property>
   <name>yarn.resourcemanager.cluster-id</name>
   <value>hd260-yarn</value>
  </property>
<property>
   <name>yarn.resourcemanager.ha.enabled</name>
   <value>true</value>
  </property>
  <property>
   <name>yarn.resourcemanager.ha.rm-ids</name>
   <value>rm1,rm2</value>
  </property>
  <property>
   <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
   <value>true</value>
  </property>
  <property>
   <name>yarn.resourcemanager.hostname.rm1</name>
   <value>master</value>
  </property>
  <property>
   <name>yarn.resourcemanager.hostname.rm2</name>
   <value>hadoop04</value>
  </property>
<!--zk store parameter parameter-->
  <property>
   <name>yarn.resourcemanager.recovery.enabled</name>
   <value>true</value>
  </property>
  <property>
   <name>yarn.resourcemanager.store.class</name>
   <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  </property>
  <property>
      <name>yarn.resourcemanager.zk-address</name>
      <value>hadoop04:2181,slave1:2181,slave2:2181</value>
   </property>

<!--RM1-->
  <property>
      <name>yarn.resourcemanager.address.rm1</name>
      <value>master:8032</value>
   </property>
  <property>
      <name>yarn.resourcemanager.scheduler.address.rm1</name>
      <value>master:8030</value>
   </property>
  <property>
      <name>yarn.resourcemanager.webapp.address.rm1</name>
      <value>master:8088</value>
   </property>
  <property>
      <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
      <value>master:8031</value>
   </property>
  <property>
      <name>yarn.resourcemanager.admin.address.rm1</name>
      <value>master:8033</value>
   </property>

<!--RM2-->
  <property>
      <name>yarn.resourcemanager.address.rm2</name>
      <value>hadoop04:8032</value>
   </property>
  <property>
      <name>yarn.resourcemanager.scheduler.address.rm2</name>
      <value>hadoop04:8030</value>
   </property>
  <property>
      <name>yarn.resourcemanager.webapp.address.rm2</name>
      <value>hadoop04:8088</value>
   </property>
  <property>
      <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
      <value>hadoop04:8031</value>
   </property>
  <property>
      <name>yarn.resourcemanager.admin.address.rm2</name>
      <value>hadoop04:8033</value>
   </property>

4）同步配置文件
[hadoop@master hadoop]$ scp yarn-env.sh slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-env.sh slave2:/opt/hadoop/etc/hadoop/  
[hadoop@master hadoop]$ scp yarn-env.sh hadoop04:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-site.xml slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-site.xml slave2:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-site.xml hadoop04:/opt/hadoop/etc/hadoop/   
[hadoop@master hadoop]$ scp mapred-site.xml hadoop04:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp mapred-site.xml slave1:/opt/hadoop/etc/hadoop/    
[hadoop@master hadoop]$ scp mapred-site.xml slave2:/opt/hadoop/etc/hadoop/
5）启动zookeeper
[hadoop@slave1 ~]$ zkServer.sh start
[hadoop@slave2 ~]$ zkServer.sh start
[hadoop@hadoop04 ~]$ zkServer.sh start


6）启动journalnode
[hadoop@master hadoop]$ hadoop-daemons.sh --hostnames 'slave1 slave2 hadoop04' start journalnode


7）启动namenode  《==如果没有配置hdfs ha，那就不用执行第六步，直接启动namenode
[hadoop@master hadoop]$ hadoop-daemons.sh --hostnames 'master slave1 slave2 hadoop04' start namenode


8）启动datanode
[hadoop@master ~]$ hadoop-daemons.sh start datanode


9）启动 zkfc
[hadoop@master ~]$ hadoop-daemons.sh --hostnames 'master slave1 slave2 hadoop04' start zkfc


10）分别启动resource manager
[hadoop@master ~]$ yarn-daemon.sh start resourcemanager
启动时报错，部分报错内容：
************************************************************/
2017-03-27 03:26:23,310 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered UNIX signal handlers 
for [TERM, HUP, INT]
2017-03-27 03:26:24,688 INFO org.apache.hadoop.conf.Configuration: found resource core-site.xml at file:/opt/hadoop-2.6.0/et
c/hadoop/core-site.xml
2017-03-27 03:26:25,580 FATAL org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream@4a44bd92
org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 'include' failed, and no 'fallback' element was found.
        at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
        at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
        at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2352)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2425)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2374)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281)
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:888)
        at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:907)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2068)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2098)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(Groups.java:295)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
2017-03-27 03:26:25,595 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state INITED; caus
e: java.lang.RuntimeException: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; 
An 'include' failed, and no 'fallback' element was found.
java.lang.RuntimeException: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 
'include' failed, and no 'fallback' element was found.
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2106)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(Groups.java:295)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
Caused by: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 'include' failed,
 and no 'fallback' element was found.


11）修改core-site.xml         <==解决报错内容
<!--
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
 <xi:include href="cmt.xml"/>
-->
<configuration>
 <property>
  <name>fs.defaultFS</name>
  <value>viewfs://nsX</value>
 </property>

 <property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/hadoop/tmp</value>
 </property>

  <property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/opt/hadoop/journalnode/data</value>
 </property>
 <property>
  <name>ha.zookeeper.quorum</name>
  <value>hadoop04:2181,slave1:2181,slave2:2181</value>
 </property>
 
<property>
  <name>fs.viewfs.mounttable.nsX.link./view_ns1</name>
  <value>hdfs://ns1</value>
  <description>Abasefor other temporary directories.</description>
 </property>

 <property>
  <name>fs.viewfs.mounttable.nsX.link./view_ns2</name>
  <value>hdfs://ns2</value>
  <description>Abasefor other temporary directories.</description>
 </property>


12）同步core-site.xml文件
[hadoop@master hadoop]$ scp core-site.xml slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp core-site.xml slave2:/opt/hadoop/etc/hadoop/   
[hadoop@master hadoop]$ scp core-site.xml hadoop04:/opt/hadoop/etc/hadoop/
13）再次启动resourcemanager
[hadoop@master hadoop]$ yarn-daemon.sh start resourcemanager


14）检查rm1的状态
[hadoop@master hadoop]$ yarn rmadmin -getServiceState rm1
active
[hadoop@master hadoop]$


15）Hadoop04节点启动resourcemanager
[hadoop@hadoop04 ~]$ yarn-daemon.sh start resourcemanager
[hadoop@hadoop04 ~]$ yarn rmadmin -getServiceState rm2
standby
[hadoop@hadoop04 ~]$


16）启动nodemanager
[hadoop@master hadoop]$ yarn-daemons.sh start nodemanager

自此 Hadoop HA + Federation + YARN HA配置实现完成