可以进行验证Hadoop的伪分布式的配置是否都成功了
cd /usr/local/hadoop/share/hadoop/mapreduce/
hadoop jar ./hadoop-mapreduce-examples-3.0.0.jar pi 10 10
会得到结果:3.2
这都不重要
前一篇文章中完成了Hadoop的伪分布式配置,接下来安装Hadoop安装手册中的配置进行Yarn的启动。
首先修改配置文件 mapred-site.xml(3.0.0版本中是直接有这个文件的,其他版本可能是临时文件,需要copy一份更名为):
vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
如果不想启动yarn,以后将配置项删去或注释即可。
接着修改配置文件 yarn-site.xml(参考文章里面我不知道怎么实现任务调度的,反正按照参考文章的方式没成功):
我查询了许多文章,最后在这种方式下成功的实现了任务调度。
粗体的value值可以通过指令获取:
hadoop classpath
vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- <property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property> -->
<property>
<name>yarn.application.classpath</name>
<value>/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*</value>
</property>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>127.0.0.1:18141</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>127.0.0.1:8088</value>
</property>
</configuration>
/usr/local/hadoop/sbin/start-yarn.sh
./sbin/mr-jobhistory-daemon.sh start historyserver
jps查看进程:
至此,完成配置,我们需要验证是否配置成功:
hadoop jar ./hadoop-mapreduce-examples-3.0.0.jar pi 10 10
可以如之前一样查询到结果位:3.2
而且,我们打开localhost:8088
参考文章:
1.HADOOP2.0,Exception java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster
http://blog.csdn.net/bxyz1203/article/details/8095042
2.Hadoop安装教程_单机/伪分布式配置_Hadoop2.6.0/Ubuntu14.04
http://www.powerxing.com/install-hadoop/