hadoop伪分布式下yarn搭建中遇到的问题

虚拟机连不上网

虚拟机网络设置改为NAT

Retrying connect to server: 0.0.0.0/0.0.0.0:8032

在只配置了dfs的情况下，MapReduce任务能够执行下列任务，开启yarn以后报错，如果没有配置yarn的情况下也不能运行，那肯定不是yarn的错
MapReduce任务

$./sbin/start-dfs.sh
$./bin/hdfs dfs -mkdir -p /user/hadoop
$./bin/hdfs dfs -mkdir input
$./bin/hdfs dfs -put ./etc/hadoop/*.xml input
$./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs[a-z.]+'

查看运行结果的命令

./bin/hdfs dfs -cat output/*

在配置了yarn的情况下，如果不想启动 YARN，务必把配置文件 mapred-site.xml 重命名，改成 mapred-site.xml.template，需要用时改回来就行。否则在该配置文件存在，而未开启 YARN 的情况下，运行程序会提示 “Retrying connect to server: 0.0.0.0/0.0.0.0:8032” 的错误，这也是为何该配置文件初始文件名为 mapred-site.xml.template。

修改以后还是报错，搜到的都说修改yarn-site.xml，但伪分布式其实不用配置那些，默认的就好了这个东西把我坑惨了，跟同学的一一对比配置文件以后nodemanager日志终于不报Retrying connect to server: 0.0.0.0/0.0.0.0:8032了，我最终的配置文件放在最后，但是MapReduce依旧卡住，查看namenode日志文件，发现日志中 没有报错 信息，但是一直在重复以下

2020-05-08 22:25:33,530 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 78  
2020-05-08 22:25:33,531 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 79  
2020-05-08 22:25:33,534 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /usr/local/hadoop/tmp/dfs/name/current/edits_inprogress_0000000000000000489 -> /usr/local/hadoop/tmp/dfs/name/current/edits_0000000000000000489-0000000000000000490 

参考博主的文章「bigtiger1648」：https://blog.csdn.net/bigtiger1648/article/details/102842679

将mapred-site.xml中

<property> 
              <name>mapreduce.framework.name</name> 
            <value>yarn</value>
</property>

属性替换为下面，然后重启，成功

<property>        
          <name>mapreduce.job.tracker</name>        
          <value>hdfs://izbp1aiqq9qrjpvel26rx0z:8001</value>        
          <final>true</final>    
</property>

伪分布式yarn详细配置

/etc/hosts

127.0.0.1 localhost
127.0.0.1 hostname(你的主机名）

core-site.xml

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

hdfs.site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop/tmp/dfs/data</value>
    </property>
<property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:9870</value>
    </property>
</configuration>

mapred-site.xml

<configuration>
 <property>        
          <name>mapreduce.job.tracker</name>        
          <value>hdfs://izbp1aiqq9qrjpvel26rx0z:8001</value>        
          <final>true</final>    
</property>
	<property>
		<name>yarn.app.mapreduce.am.env</name>
		<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
	</property>
	<property>
		<name>mapreduce.map.env</name>
		<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
	</property>
	<property>
		<name>mapreduce.reduce.env</name>
		<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
	</property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        </property>
</configuration>