配置hadoop yarn高可用

前提

1. 三台服务器分别为hadoop01,hadoop02,hadoop03且已经在hosts中配置
2. 三台服务器网络连通
3. 三台服务器都已经安装好jdk和zookeeper

1.配置yarn-site.xml
进入配置文件目录：

cd ./hadoop-3.1.2/etc/hadoop

修改配置文件：

vi yarn-site.xml

<configuration>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<!--是否启动yarn的ha-->
<property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
</property>

<!--yarn的ha的虚拟服务名-->
<property>
    <name>yarn.resoucemanager.cluster-id</name>
    <value>qfyarn</value>
</property>

<!--yarn的ha的虚拟服务名下的具体的rm-->
<property>
    <name>yarn.resoucemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
</property>

<!--指定rm1所启动的主机-->
<property>
    <name>yarn.resoucemanager.hostname.rm1</name>
    <value>hadoop01</value>
</property>

<!--指定rm2所启动的主机-->
<property>
    <name>yarn.resoucemanager.hostname.rm2</name>
    <value>hadoop02</value>
</property>

<!--指定rm1的web ui的通信地址-->
<property>
    <name>yarn.resoucemanager.webapp.address.rm1</name>
    <value>hadoop01:8088</value>
</property>

<!--指定rm2的web ui的通信地址-->
<property>
    <name>yarn.resoucemanager.webapp.address.rm2</name>
    <value>hadoop02:8088</value>
</property>

<!--指定rm2的web ui的通信地址-->
<property>
    <name>yarn.resoucemanager.zk-address</name>
    <value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
</property>

</configuration>

2. 配置mapred-site.xml

vi mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    <final>true</final>
</property>
</configuration>

3. 在hadoop01上启动yarn

start-yarn.sh

4. 在hadoop02上启动reourcemanager

yarn-daemon.sh start resourcemanager

5. 查看reourcemanager的启动状态


6. 测试
创建文本 vi /home/words

hadoop is good
hadoop is nice
hadoop is better

把文件上传到hdfs文件系统

hdfs dfs -put /home/words /

然后执行如下命令计数：

yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /words /out/01

会报如下错误：
[2019-10-03 13:44:24.140]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

[2019-10-03 13:44:24.141]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

查阅资料后发现是要在yarn-site.xml里加入一个配置

首先输入hadoop classpath获取配置：

vi yarn-site.xml]加入如下配置：

<property>
    <name>yarn.application.classpath</name>
    <value>/home/hadoop/app/hadoop-3.1.2/etc/hadoop:/home/hadoop/app/hadoop-3.1.2/share/hadoop/common/lib/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/common/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/hdfs:/home/hadoop/app/hadoop-3.1.2/share/hadoop/hdfs/lib/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/hdfs/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/mapreduce/lib/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/mapreduce/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/yarn:/home/hadoop/app/hadoop-3.1.2/share/hadoop/yarn/lib/*:/home/hadoop/app/hadoop-3.1.2/share/hadoop/yarn/*</value>
</property>

然后重启yarn

stop-yarn.sh

执行如下命令即可：

yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /words /out/01

查看输出结果：

hdfs dfs -cat /out/01/part-r-00000

然后杀掉第一台上的resourcemanager

再次执行如下命令：

yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /words /out/02

查看结果