Task with the most failures(4):
-----
Task ID:
task_1562543688229_0006_m_000000
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1562543688229_0006&tipid=task_1562543688229_0006_m_000000
-----
Diagnostic Messages for this Task:
[2019-07-08 15:32:44.976]Container [pid=100107,containerID=container_1562543688229_0006_01_000005] is running 429783552B beyond the 'VIRTUAL' memory limit. Current usage:
35.8 MB of 1 GB physical memory used; 2.5 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1562543688229_0006_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 100117 100107 100107 100107 (java) 33 43 2568744960 8859 /usr/local/java/jdk1.8.0_11/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
-Xmx820m -Djava.io.tmpdir=/hadoop/pull/nm-local-dir/usercache/root/appcache/application_1562543688229_0006/container_1562543688229_0006_01_000005/tmp -Dlog4j.configuration=
container-log4j.properties -Dyarn.app.container.log.dir=/hadoop/hadoop-3.1.2/logs/userlogs/application_1562543688229_0006/container_1562543688229_0006_01_000005 -Dyarn.app.
container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.176.142 42385 attempt_1562543688229_0006_m_000000_3 5
|- 100107 100105 100107 100107 (bash) 0 0 115896320 308 /bin/bash -c /usr/local/java/jdk1.8.0_11/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
-Xmx820m -Djava.io.tmpdir=/hadoop/pull/nm-local-dir/usercache/root/appcache/application_1562543688229_0006/container_1562543688229_0006_01_000005/tmp -Dlog4j.configuration=container
-log4j.properties -Dyarn.app.container.log.dir=/hadoop/hadoop-3.1.2/logs/userlogs/application_1562543688229_0006/container_1562543688229_0006_01_000005 -Dyarn.app.container.log.filesize
=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.176.142 42385 attempt_1562543688229_0006_m_000000_3 5 1>
/hadoop/hadoop-3.1.2/logs/userlogs/application_1562543688229_0006/container_1562543688229_0006_01_000005/stdout 2>/hadoop/hadoop-3.1.2/logs/userlogs/application_1562543688229_0006/
container_1562543688229_0006_01_000005/stderr
解决方式 :
修改hadoop的yarn-site.xml,添加
yarn.nodemanager.vmem-check-enabled
false