成功解决 -- Container is running beyond the ‘VIRTUAL‘ memory limit. Killing container.

问题描述

在学习 大数据组件笔记 – Hive 运行 YARN 任务时报错：Container is running beyond the ‘VIRTUAL‘ memory limit. Killing container.

报错详情

[omm@bigdata01 lib]$ yarn jar etl-1.0-SNAPSHOT.jar com.simwor.etl.VideoDriver /gulivideo/video /gulivideo/video_etl
....
2021-03-24 20:49:41,394 INFO mapreduce.Job:  map 0% reduce 0%
2021-03-24 20:49:53,666 INFO mapreduce.Job: Task Id : attempt_1616589845973_0001_m_000001_0, Status : FAILED
[2021-03-24 20:48:09.540]Container [pid=14298,containerID=container_1616589845973_0001_01_000003] is running 262203904B beyond the 'VIRTUAL' memory limit. Current usage: 58.9 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
....

解决方法

参数详解：https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

向配置文件 /opt/module/hadoop/etc/hadoop/mapred-site.xml 中添加参数并重启服务。

  <property>
  　　<name>mapreduce.map.memory.mb</name>
  　　<value>1536</value>
  </property>
  <property>
  　　<name>mapreduce.map.java.opts</name>
  　　<value>-Xmx1024M</value>
  </property>
  <property>
  　　<name>mapreduce.reduce.memory.mb</name>
  　　<value>3072</value>
  </property>
  <property>
  　　<name>mapreduce.reduce.java.opts</name>
  　　<value>-Xmx2560M</value>
  </property>