报错:
2023-11-29 22:54:36,745 INFO mapreduce.Job: Task Id : attempt_1701267209584_0004_m_000001_2, Status : FAILED
[2023-11-29 22:54:34.910]Container [pid=3211,containerID=container_1701267209584_0004_01_000032] is running 307337728B beyond the 'VIRTUAL' memory limit. Current usage: 308.2 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1701267209584_0004_01_000032 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 3211 3210 3211 3211 (bash) 0 0 9789440 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/qianyihui/appcache/application_1701267209584_0004/container_1701267209584_0004_01_000032/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1701267209584_0004/container_1701267209584_0004_01_000032 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.104 43206 attempt_1701267209584_0004_m_000001_2 32 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1701267209584_0004/container_1701267209584_0004_01_000032/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1701267209584_0004/container_1701267209584_0004_01_000032/stderr
|- 3222 3211 3211 3211 (java) 244 21 2552406016 78621 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/qianyihui/appcache/application_1701267209584_0004/container_1701267209584_0004_01_000032/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1701267209584_0004/container_1701267209584_0004_01_000032 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.104 43206 attempt_1701267209584_0004_m_000001_2 32
修改yarn配置:
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
解决Yarn在执行程序遇到超出虚拟内存限制,Container被kill