本文是我学习Hadoop搭建过程中的各种笔记,内容来自于各种公开的教程,起点非常低,从Linux基础开始,直至在PC上搭建Hadoop成功,是真正的从零开始。
感谢过程中帮助我的各位认识的和不认识的老师。
53、Yarn的ha配置和测试:
官网:http://hadoop.apache.org/docs/current/
YARN模块,ResourceManager HA (HA一般都是老大的HA)网址:http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html
架构:
yarn的HA:
规划:
IP地址 | 主机名称 | 规划 |
192.168.216.111 | hadoop01 | resourcemanager、nodemanager、quroumPeerMain |
192.168.216.112 | hadoop02 | resourcemanager、nodemanager、quroumPeerMain |
192.168.216.113 | hadoop03 | nodemanager、quroumPeerMain |
[没有自动失败转移的东西,内部有切换的机制]
简单的配置:
yarn-site.xml
<configuration>
<!--是否启动yarn的HA -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!--yarn的HA虚拟服务名 -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster1</value>
</property>
<!--yarn的HA虚拟服务名下的具体的rm -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!--指定rm1的所启动的主机 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>hadoop01</value>
</property>
<!--指定rm2的所启动的主机 -->
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>hadoop02</value>
</property>
<!--指定rm1的web ui的通信地址 -->
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>hadoop01:8088</value>
</property>
<!--指定rm2的web ui的通信地址 -->
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>hadoop02:8088</value>
</property>
<!--指定resourcemanager的 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>
</property>
到上面的yarn的完成,跑作业还需要加点东西:
mapred-site.xml
<configuration>
<!--指定mr的shuffle-->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!--指定mapreduce运行框架-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<final>true</final>
</property >
</configuration>
远程发送:
[root@hadoop01 hadoop-2.7.1]# scp -r ./etc/hadoop/mapred-site.xml. ./etc/hadoop/yarn-site.xml hadoop02:/usr/local/hadoop-2.7.1/etc/hadoop/
mapred-site.xml 100% 893 0.9KB/s 00:00
yarn-site.xml 100% 1647 1.6KB/s 00:00
[root@hadoop01 hadoop-2.7.1]# scp -r ./etc/hadoop/mapred-site.xml. ./etc/hadoop/yarn-site.xml hadoop03:/usr/local/hadoop-2.7.1/etc/hadoop/
mapred-site.xml 100% 893 0.9KB/s 00:00
yarn-site.xml 100% 1647 1.6KB/s 00:00
[还需要做ssh免登陆,前面做了,这就不用做了]
直接启动:start-yarn.sh
[root@hadoop01 hadoop-2.7.1]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-resourcemanager-hadoop01.out
hadoop03: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop03.out
hadoop02: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop02.out
hadoop01: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop0.out
手动的HA:yarn-daemon.sh start resourcemanager
hadoop02上,手动启动resourcemanager:
[root@hadoop02 ~]# yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root- resourcemanager-hadoop03.out
在hadoop01,02上查看jps,都多了ResourceManager和NodeManager;在hadoop03上查看jps,多了NodeManager
查看web ui监控,能看到:
http://192.168.216.111:8088/ 处于active状态
http://192.168.216.112:8088/ 处于standby状态
两台上面都有resourcemanager,要不然web ui进不来
测试:
yara的高可用测试:yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount/words /out/00
[关闭active的rm,然后查看是否正常切换]
跑一个作业:
到hadoop01,在home下建一个words vi /home/words
[root@hadoop01 hadoop-2.7.1]# vi /home/words
hadoop is nice
hadoop good
hadoop is better
建好,下面统计单词
把/home/words放到 / 根目录下来:hdfs dfs -put /home/words /
[root@hadoop01 hadoop-2.7.1]# hdfs dfs -put /home/words /
用yarn的集群测试能不能跑,能跑,yarn也是OK的
[root@hadoop01 hadoop-2.7.1]# yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount/words /out/00
跑完,查看00的结果:
[root@hadoop01 hadoop-2.7.1]# hdfs dfs -cat /out/00/part-r-00000
better 1
good 1
hadoop 3
is 2
nice 1
关闭hadoop01的ResourceManager
[root@hadoop01 hadoop-2.7.1]# kill -9 20444
在网址web ui监控上查看,原来是standby状态的切换成了active
再次跑结果,改成01:
[root@hadoop01 hadoop-2.7.1]# yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount/words /out/01
因为高可用切过去了,所以依然能跑出结果
查看jps,hadoop01,02,03的NodeManager都在,没有问题
查看01的结果,跑出来了:
[root@hadoop01 hadoop-2.7.1]# hdfs dfs -cat /out/01/part-r-00000
better 1
good 1
hadoop 3
is 2
nice 1
到此yarn的高可用搭建好↑
实体机
1、 将centos写入U盘,按F8,使用U盘启动,安装虚拟机;
2、 添加网卡驱动:
网址:https://blog.csdn.net/tidaomazi/article/details/79345746 按照此步骤执行即可;
注意:
① 下载:e1000e-3.4.0.2.tar.gz制到centos中的Computer Filesystem中的srv中;
② 将U盘中的Packages,复制到centos中的Computer Filesystem中
③ 进入network-scripts目录:
[root@hadoop001 Desktop]# cd /etc/sysconfig/network-scripts/
④ 新建一个eth0:
[root@localhost network-scripts]# vim ifcfg-eth0
⑤ 查看HWaddr:(每台机子都有固定的一个,硬件地址,唯一标识)
[root@hadoop001 network-scripts]#cat /etc/udev/rules.d/70-persistent-net.rules 或
[root@hadoop001 network-scripts]#ifconfig
⑥ 配置eth0(编辑eth0):
[root@localhost network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=b06ea46c-aced-4e34-aaf6-5e57e9b3015d
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static (static静态IP ;dhcp动态IP;none无(不指定))
HWADDR=00:0C:29:B0:A4:A5 (要改)
IPADDR=192.168.216.111 (要改)
PREFIX=24
GATEWAY=192.168.216.2
DNS1=192.168.216.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
⑦ 新建:(读取:cat network)
[root@hadoop001 network-scripts]#vim network
NETWORKING=yes
HOSTNAME=hadoop001
3、 连接外网:
[root@hadoop001 Desktop]#route –n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.216.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0
0.0.0.0 192.168.216.2 0.0.0.0 UG 0 0 0 eth0
[root@hadoop001 Desktop]# route add default gw 192.168.216.2
[root@hadoop001 Desktop]# vim /etc/resolv.conf
nameserver 114.114.114.114
nameserver 8.8.8.8 (添加)
[root@localhost network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
UUID=b06ea46c-aced-4e34-aaf6-5e57e9b3015d
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp
HWADDR=00:0C:29:B0:A4:A5
#IPADDR=192.168.216.111
#PREFIX=24
#GATEWAY=192.168.216.2
#DNS1=192.168.216.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
4、 连接XShell