当我们使用Hadoop的MapReduce框架来运行一个示例程序的时候,例如:
hadoop jar hadoop-mapreduce-examples-3.4.0.jar grep input output 'dfs[a-z.]+'
报出以下错误:
一开始还以为是电脑的配置不太行,然后找了很多东西,发现只需要在yarn-site.xml文件中加入以下配置:
首先你通过hadoop classpath获取到这个位置
然后将这个位置复制,写入yarn-site.xml中
<property>
<name>yarn.application.classpath</name>
<value>/opt/hadoop-3.4.0/etc/hadoop:/opt/hadoop-3.4.0/share/hadoop/common/lib/*:/opt/hadoop-3.4.0/share/hadoop/common/*:/opt/hadoop-3.4.0/share/hadoop/hdfs:/opt/hadoop-3.4.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.4.0/share/hadoop/hdfs/*:/opt/hadoop-3.4.0/share/hadoop/mapreduce/*:/opt/hadoop-3.4.0/share/hadoop/yarn:/opt/hadoop-3.4.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.4.0/share/hadoop/yarn/*
</value>
</property>
同时yarn-site.xml文件中要对外暴露端口,我完整的yarn-site.xml文件如下:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>node1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>node1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>node1:8031</value>
</property>
<property>
<name>yarn.nodemanager.memory-mb</name>
<value>4GB</value>
</property>
<property>
<name>yarn.nodemanager.cpu-vcores</name>
<value>4</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>/opt/hadoop-3.4.0/etc/hadoop:/opt/hadoop-3.4.0/share/hadoop/common/lib/*:/opt/hadoop-3.4.0/share/hadoop/common/*:/opt/hadoop-3.4.0/share/hadoop/hdfs:/opt/hadoop-3.4.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.4.0/share/hadoop/hdfs/*:/opt/hadoop-3.4.0/share/hadoop/mapreduce/*:/opt/hadoop-3.4.0/share/hadoop/yarn:/opt/hadoop-3.4.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.4.0/share/hadoop/yarn/*
</value>
</property>
</configuration>
提供参考
重新start-yarn.sh
结果如下,发现成功了:
然后访问自己的 ip地址+8088端口
发现没有问题
然后查看Browse Directory查看是否生成output文件
同时通过命令:hdfs dfs -cat /user/root/output/part-r-00000查看运行后的内容
至此,问题解决