黑猴子的家：Hadoop 伪分布式Yarn运行MapReduce

1、配置集群

（1）配置yarn-env.sh

获取jdk 的安装路径
[victor@node1 hadoop-2.8.2]$ echo $JAVA_HOME
/opt/module/jdk1.8.0_144

修改JAVA_HOME 路径
[victor@node1 hadoop-2.8.2]$ vim etc/hadoop/yarn-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144

（2）配置yarn-site.xml

<!-- reducer获取数据的方式 -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>node1</value>
</property>
<property>
    <name>yarn.resourcemanager.nodes.include-path</name>
    <value>/opt/module/hadoop-2.8.2/etc/hadoop/dnrm</value>
</property>

（3）配置dnrm

[victor@node1 hadoop]$ vim dnrm
node1
node2
node3

（4）配置mapred-env.sh

修改JAVA_HOME 路径
[victor@node1 hadoop-2.8.2]$ vim etc/hadoop/mapred-env.sh
export JAVA_HOME=/opt/module/jdk1.8.0_144

（5）配置mapred-site.xml

对mapred-site.xml.template重命名为mapred-site.xml 
重命名mapred-site.xml文件
[victor@node1 hadoop-2.8.2]$ cd etc/hadoop/
[victor@node1 hadoop-2.8.2]$ pwd
/opt/module/hadoop-2.8.2/etc/hadoop
[victor@node1 hadoop]$ mv mapred-site.xml.template mapred-site.xml

添加配置
<!-- 指定mr运行在yarn上 -->
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

2、启动集群

（1）启动resourcemanager

[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh start resourcemanager

（2）启动nodemanager

[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh start nodemanager

(3)集群操作

（1）yarn的浏览器页面查看

$ http://node1:8088/cluster

image.png

（2）删除文件系统上的output文件

[victor@node1 hadoop-2.8.2]$ bin/hdfs dfs -rm -R /user/victor/wordcount/output

（3）执行mapreduce程序

[victor@node1 hadoop-2.8.2]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.2.jar wordcount /user/victor/wordcount/input  /user/victor/wordcount/output

（4）查看运行结果

[victor@node1 hadoop-2.8.2]$ bin/hdfs dfs -cat /user/victor/wordcount/output/*
![image.png](https://upload-images.jianshu.io/upload_images/9193428-2364994192887eee.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)