每个map 和reduce任务的物理内存限制有问题
修改/hadoop安装路径/etc/hadoop/yarn-site.xml文件 添加
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>22528</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>16384</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1500</value>
<discription>单个任务可申请最少内存,默认1024MB</discription>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>4</value>
</property>
修改同样路径下的mapred-site.xml文件
我这里忘记添加了那些 下面我的是mapred-site.xml的所有配置
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<!--#设置MapReduce的运行平台为yarn-->
</property>
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
<property>
<name>mapreduce.job.ubertask.maxbytes</name>
<value>100</value>
</property>
<property>
<name>mapred.task.timeout</name>
<value>600000</value>
<description>The number of milliseconds before a task will be terminated if it neither reads an input,
writes an output, nor updates its status string.
</description>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1500</value>
<description>每个Map任务的物理内存限制</description>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3000</value>
<description>每个Reduce任务的物理内存限制</description>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1200m</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2600m</value>
</property>
</configuration>