mapreduce在yarn上运行，Container killed by the ApplicationMaster, Exit code is 143

最新推荐文章于 2024-02-17 19:11:27 发布

彧卿丶

最新推荐文章于 2024-02-17 19:11:27 发布

阅读量1.8k

点赞数 5

分类专栏： Hadoop 文章标签： YARN 内存溢出 MapReduce 虚拟内存配置调整

本文链接：https://blog.csdn.net/Asa_Prince/article/details/115284225

版权

Hadoop 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

mapreduce 程序在yarn上运行时报错，报错信息具体如下：

.....

[2021-03-28 17:38:50.081]Container killed on request. Exit code is 143
[2021-03-28 17:38:50.085]Container exited with a non-zero exit code 143. 

2021-03-28 17:38:57,155 INFO mapreduce.Job: Task Id : attempt_1616924256514_0001_m_000000_1, Status : FAILED
[2021-03-28 17:38:55.966]Container [pid=3061,containerID=container_1616924256514_0001_01_000003] is running 288238080B beyond the 'VIRTUAL' memory limit. Current usage: 90.1 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1616924256514_0001_01_000003 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

.....

[2021-03-28 17:38:56.028]Container killed on request. Exit code is 143
[2021-03-28 17:38:56.035]Container exited with a non-zero exit code 143. 

2021-03-28 17:39:03,274 INFO mapreduce.Job: Task Id : attempt_1616924256514_0001_m_000000_2, Status : FAILED
[2021-03-28 17:39:02.076]Container [pid=3106,containerID=container_1616924256514_0001_01_000004] is running 293661184B beyond the 'VIRTUAL' memory limit. Current usage: 99.2 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1616924256514_0001_01_000004 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

.....

显示当前使用情况：使用了90.1 MB的1 GB物理内存；使用了2.4 GB的2.1 GB虚拟内存。容器被 kill 掉了。

说到底还是物理内存不够用导致的。

可以在 yarn-site.xml 文件中增加如下属性，并重启 yarn 服务解决。

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
   <description>Whether virtual memory limits will be enforced for containers</description>
</property>

<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
   <value>4</value>
   <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>