模拟环境:四台虚拟机node1、node2、node3、node4
node1 | node2 | node3 | node4 |
---|---|---|---|
NameNode-1 | NameNode-2 | ||
DataNode-1 | DataNode-2 | DataNode-3 | |
Zookeeper-1 | Zookeeper-2 | Zookeeper-3 | |
zkfc-1 | zkfc-2 | ||
JournalNode-1 | JournalNode-2 | JournalNode-3 | |
ResourceManager-1 | ResourceManager-2 | ||
NodeManager-1 | NodeManager-2 | NodeManager-3 |
在Hadoop HA 集群搭建的基础上进行搭建
1. 配置文件
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
指定mr作业运行的框架:要么本地运行,要么使用MRv1,要么使用yarn
- local (本地运行)
- classic (使用MRv1,hadoop2.x之前)
- yarn (使用yarn)
yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- 让yarn的容器支持mapreduce的洗牌,开启shuffle服务 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 启用resourcemanager的HA -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 给两个resourcemanager组成的HA命名 -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>cluster1</value>
</property>
<!-- RM HA的两个resourcemanager的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 指定rm1的reourcemanager进程所在的主机名称 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>node3</value>
</property>
<!-- 指定rm2的reourcemanager进程所在的主机名称 -->
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>node4</value>
</property>
<!-- 指定zookeeper集群的各个节点地址和端口号 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>node2:2181,node3:2181,node4:2181</value>
</property>
</configuration>
将两个配置文件拷贝到node2、node3、node4上
2. 启动
- 启动zookeeper集群
- 启动hadoop集群
- 启动yarn
在node3(或node4)上执行命令:start-yarn.sh
在node4(或node3)上执行命令:yarn-daemon.sh start resourcemanager
访问resourcemanager的web页面(默认端口8088):
http://node3:8088
http://node4:8088
3. 运行自带wordcount
-
生成数据文件
for num in `seq 10000`; do echo "hello world $num" >> ~/wordcount.txt; done
-
将文件上传到hdfs
hdfs dfs -put ~/wordcount.txt /user/root/wordcount.txt
-
运行wordcount
cd $JAVA_HOME/share/hadoop/mapreduce hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /user/root/wordcount.txt /mr/wordcount/output
-
查看输出
hdfs dfs -ls /mr/wordcount/output
hdfs dfs -cat /mr/wordcount/output/part-r-00000
4. 停止
在node3(或者node4)上执行:stop-yarn.sh
在node4(或者node3)上执行:yarn-deamon.sh stop resourcemanager
停止hadoop集群
停止zookeeper集群