Hadoop四：Container is running beyond memory limits

The NodeManager is killing your container. It sounds like you are trying to use hadoop streaming which is running as a child process of the map-reduce task. The NodeManager monitors the entire process tree of the task and if it eats up more memory than the maximum set in mapreduce.map.memory.mb or mapreduce.reduce.memory.mb respectively, we would expect the Nodemanager to kill the task, otherwise your task is stealing memory belonging to other containers, which you don't want.

NodeManager监控任务的整体进程树，当某个进程耗费较多内存，超出mapreduce.map.memory.mb o或mapreduce.reduce.memory.mb中的设定值，则NodeManager会将相应任务kill掉，否则该任务会偷属于其他容器的内存。

参考链接：https://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

解决方案一：设置mapreduce内存大小（需根据你的实际情况设置）

In mapred-site.xml:

mapreduce.map.memory.mb: 4096

mapreduce.reduce.memory.mb: 8192

另外，由于每个Container的Map and Reduce 任务需要运行JVMs，因此，需要设置JVM heap size大小，且其大小需低于上一步定义的Map and Reduce内存值，才能保证在分配YARN 任务时在container的内存限制内。

In mapred-site.xml:

mapreduce.map.java.opts: -Xmx3072m

mapreduce.reduce.java.opts: -Xmx6144m

结果：问题解决。

解决方案二：在yarn-site.xml文件中添加内容如下：

 <property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
    <description>Whether virtual memory limits will be enforced for containers</description>
  </property>
 <property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>4</value>
    <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
  </property>