安装服务
单机版Yarn:ResourceManager、NodeManager
环境准备:
节点:10.1.253.178(hostname:cdh1)
节点已经安装完成HDFS服务,参见:
hadoop单机部署伪分布式系列1:HDFS
安装包:
hadoop-2.0.0-cdh4.2.1.tar.gz(使用已经安装完成后的HDFS对应的包,Yarn与HDFS集成在一个包内)
部署步骤:
1. 修改以下文件
yarn文件1:/home/liulu/app/hadoop-2.0.0-cdh4.2.1/etc/hadoop/mapred-site.xml
mv ~/app/hadoop-2.0.0-cdh4.2.1/etc/hadoop/mapred-site.xml.templat ~/app/hadoop-2.0.0-cdh4.2.1/etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
yarn文件2:/home/liulu/app/hadoop-2.0.0-cdh4.2.1/etc/hadoop/yarn-site.xml:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
yarn文件3:/home/liulu/app/hadoop-2.0.0-cdh4.2.1/etc/hadoop/conf/slaves:
localhost
localhost
2. 启动yarn集群
cd /home/liulu/app/hadoop-2.0.0-cdh4.2.1/sbin
./start-yarn.sh
3. 检查yarn集群
看进程(ResourceManager、NodeManager):
[liulu@cdh1 sbin]$ jps -m
25751 DataNode
25610 NameNode
9304 ResourceManager
9423 NodeManager
25936 SecondaryNameNode
看yarn监控页面:
http://10.1.253.178:8088/cluster
4. yarn操作
执行wordcount例子
cd /home/liulu/app/hadoop-2.0.0-cdh4.2.1/bin
./hdfs dfs -mkdir /in
./hdfs dfs -put ~/testfile /in/(准备源数据)
./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.2.1.jar wordcount /in /out(执行wordcount)
./hdfs dfs -ls /out(查看输出结果)
5. 关闭yarn
cd /home/liulu/app/hadoop-2.0.0-cdh4.2.1/sbin
./stop-yarn.sh
参考文档: