集群规划
配置步骤
一、 前提准备
1) 安装好Hadoop2.0集群 《==参考《HADOOP HA+Federation集群实现(七)》及以前的文章
2) 安装好zookeeper集群
二、修改如下配置文件
1) yarn-site.xml
2) mapred-site.xml
3) yarn-env.sh
三、启动服务
1) 启动HDFS
2) 启动YARN,通过yarn-daemons.sh来启动
四、测试验证
1) 查看状态
yarn rmadmin -getServiceState rm1/rm2
2) 模拟故障切换
实施步骤
1) 编辑yarn-env.sh
[hadoop@master ~]$ cd /opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ vi yarn-env.sh
export JAVA_HOME=/usr/java/default
2)编辑mapred-site.xml
[hadoop@master hadoop]$ vi mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
3)编辑yarn-site.xml
[hadoop@master hadoop]$ vi yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--YARN HA parameter-->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>hd260-yarn</value>
</property>
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>hadoop04</value>
</property>
<!--zk store parameter parameter-->
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>hadoop04:2181,slave1:2181,slave2:2181</value>
</property>
<!--RM1-->
<property>
<name>yarn.resourcemanager.address.rm1</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm1</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm1</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm1</name>
<value>master:8033</value>
</property>
<!--RM2-->
<property>
<name>yarn.resourcemanager.address.rm2</name>
<value>hadoop04:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address.rm2</name>
<value>hadoop04:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>hadoop04:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address.rm2</name>
<value>hadoop04:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address.rm2</name>
<value>hadoop04:8033</value>
</property>
4)同步配置文件
[hadoop@master hadoop]$ scp yarn-env.sh slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-env.sh slave2:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-env.sh hadoop04:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-site.xml slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-site.xml slave2:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp yarn-site.xml hadoop04:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp mapred-site.xml hadoop04:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp mapred-site.xml slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp mapred-site.xml slave2:/opt/hadoop/etc/hadoop/
5)启动zookeeper
[hadoop@slave1 ~]$ zkServer.sh start
[hadoop@slave2 ~]$ zkServer.sh start
[hadoop@hadoop04 ~]$ zkServer.sh start
6)启动journalnode
[hadoop@master hadoop]$ hadoop-daemons.sh --hostnames 'slave1 slave2 hadoop04' start journalnode
7)启动namenode 《==如果没有配置hdfs ha,那就不用执行第六步,直接启动namenode
[hadoop@master hadoop]$ hadoop-daemons.sh --hostnames 'master slave1 slave2 hadoop04' start namenode
8)启动datanode
[hadoop@master ~]$ hadoop-daemons.sh start datanode
9)启动 zkfc
[hadoop@master ~]$ hadoop-daemons.sh --hostnames 'master slave1 slave2 hadoop04' start zkfc
10)分别启动resource manager
[hadoop@master ~]$ yarn-daemon.sh start resourcemanager
启动时报错,部分报错内容:
************************************************************/
2017-03-27 03:26:23,310 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered UNIX signal handlers
for [TERM, HUP, INT]
2017-03-27 03:26:24,688 INFO org.apache.hadoop.conf.Configuration: found resource core-site.xml at file:/opt/hadoop-2.6.0/et
c/hadoop/core-site.xml
2017-03-27 03:26:25,580 FATAL org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream@4a44bd92
org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 'include' failed, and no 'fallback' element was found.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2352)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2425)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2374)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:888)
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:907)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2068)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2098)
at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(Groups.java:295)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
2017-03-27 03:26:25,595 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state INITED; caus
e: java.lang.RuntimeException: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30;
An 'include' failed, and no 'fallback' element was found.
java.lang.RuntimeException: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An
'include' failed, and no 'fallback' element was found.
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2106)
at org.apache.hadoop.security.Groups.<init>(Groups.java:70)
at org.apache.hadoop.security.Groups.<init>(Groups.java:66)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(Groups.java:295)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)
Caused by: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 'include' failed,
and no 'fallback' element was found.
11)修改core-site.xml <==解决报错内容
<!--
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
<xi:include href="cmt.xml"/>
-->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>viewfs://nsX</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop/tmp</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/opt/hadoop/journalnode/data</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop04:2181,slave1:2181,slave2:2181</value>
</property>
<property>
<name>fs.viewfs.mounttable.nsX.link./view_ns1</name>
<value>hdfs://ns1</value>
<description>Abasefor other temporary directories.</description>
</property>
<property>
<name>fs.viewfs.mounttable.nsX.link./view_ns2</name>
<value>hdfs://ns2</value>
<description>Abasefor other temporary directories.</description>
</property>
12)同步core-site.xml文件
[hadoop@master hadoop]$ scp core-site.xml slave1:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp core-site.xml slave2:/opt/hadoop/etc/hadoop/
[hadoop@master hadoop]$ scp core-site.xml hadoop04:/opt/hadoop/etc/hadoop/
13)再次启动resourcemanager
[hadoop@master hadoop]$ yarn-daemon.sh start resourcemanager
14)检查rm1的状态
[hadoop@master hadoop]$ yarn rmadmin -getServiceState rm1
active
[hadoop@master hadoop]$
15)Hadoop04节点启动resourcemanager
[hadoop@hadoop04 ~]$ yarn-daemon.sh start resourcemanager
[hadoop@hadoop04 ~]$ yarn rmadmin -getServiceState rm2
standby
[hadoop@hadoop04 ~]$
16)启动nodemanager
[hadoop@master hadoop]$ yarn-daemons.sh start nodemanager
自此 Hadoop HA + Federation + YARN HA配置实现完成
---------------------
作者:forever19870418
来源:CSDN
原文:https://blog.csdn.net/forever19870418/article/details/67632682
版权声明:本文为博主原创文章,转载请附上博文链接!