Hadoop伪分布式安装二(MapReduce+Yarn)

上一篇我们已经介绍了HDFS的部署，接着我们进行Hadoop伪分布式MapReduce+Yarn的部署

1.Yarn部署

• 修改mapred-site.xml 文件

[hadoop@zydatahadoop001 hadoop]$ cd        /opt/software/hadoop/etc/hadoop/
[hadoop@zydatahadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[hadoop@zydatahadoop001 hadoop]$ vi mapred-site.xml

添加下面的内容在<configuration>中
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

• 修改yarn-site.xml文件

[hadoop@zydatahadoop001 hadoop]$ vi yarn-site.xml

添加下面的内容在<configuration>中
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>

• 开启服务

[hadoop@zydatahadoop001 hadoop]$ cd /opt/software/hadoop
[hadoop@zydatahadoop001 hadoop]$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-zydatahadoop001.out
zydatahadoop001: starting nodemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-zydatahadoop001.out
[hadoop@zydatahadoop001 hadoop]$ jps
24439 ResourceManager
24840 Jps
24073 SecondaryNameNode
24539 NodeManager
23788 NameNode

查看8088端口
[hadoop@zydatahadoop001 hadoop]$ netstat -nlp|grep 8088
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 :::8088                     :::*                        LISTEN      24439/java   

web界面进行查看: http://自己的ip:8088/

2.MR Job测试

MapReduce: java代码写的，map(映射)函数和reduce(归约)函数

这里可能出现错误：

WARN hdfs.DataStreamer: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException):

解决办法参考这篇博客

• 计算圆周率

[hadoop@zydatahadoop001 hadoop]$ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 5 10

计算结果（选取的参数问题出现误差）：
Job Finished in 749.765 seconds
Estimated value of Pi is 3.28000000000000000000

3.停止Yarn

[hadoop@zydatahadoop001 hadoop]$ sbin/stop-yarn.sh 
stopping yarn daemons
stopping resourcemanager
zydatahadoop001: stopping nodemanager
zydatahadoop001: nodemanager did not stop gracefully after 5 seconds: killing with kill -9
no proxyserver to stop