hadoop MR 内存不足

MR内存不足:
18/07/17 15:20:21 INFO mapreduce.Job: Task Id : attempt_1530494171563_1814563_m_000022_0, Status : FAILED
Container [pid=25576,containerID=container_e06_1530494171563_1814563_01_000066] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 7.6 GB of 8.4 GB virtual memory used. Killing container.
Dump of the process-tree for container_e06_1530494171563_1814563_01_000066 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 25576 25574 25576 25576 (bash) 0 0 115843072 357 /bin/bash -c /usr/local/jdk1.8.0_131/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3072m -Djava.io.tmpdir=/data9/hadoop/local/usercache/dw/appcache/application_1530494171563_1814563/container_e06_1530494171563_1814563_01_000066/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data7/hadoop/log/yarn/application_1530494171563_1814563/container_e06_1530494171563_1814563_01_000066 -Dyarn.app.container.log.filesize=524288000 -Dyarn.app.container.log.backups=2 -Dhadoop.root.logger=INFO,CRLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.39.69.95 41124 attempt_1530494171563_1814563_m_000022_0 6597069766722 1>/data7/hadoop/log/yarn/application_1530494171563_1814563/container_e06_1530494171563_1814563_01_000066/stdout 2>/data7/hadoop/log/yarn/application_1530494171563_1814563/container_e06_1530494171563_1814563_01_000066/stderr
|- 25598 25576 25576 25576 (java) 13612 4097 8027222016 1084215 /usr/local/jdk1.8.0_131/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx3072m -Djava.io.tmpdir=/data9/hadoop/local/usercache/dw/appcache/application_1530494171563_1814563/container_e06_1530494171563_1814563_01_000066/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data7/hadoop/log/yarn/application_1530494171563_1814563/container_e06_1530494171563_1814563_01_000066 -Dyarn.app.container.log.filesize=524288000 -Dyarn.app.container.log.backups=2 -Dhadoop.root.logger=INFO,CRLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.39.69.95 41124 attempt_1530494171563_1814563_m_000022_0 6597069766722

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

增加内存可解决:
hadoop jar mds_sinadm_tblog_sport_tag-1.0-SNAPSHOT-jar-with-dependencies.jar com.weibolabel.Label -D mapreduce.map.memory.mb=10240 -D mapred.map.child.java.opts=-Xmx8192m -D isDebug=false 2018/07/12 2018/07/12 /dw/tmp/weiboLabel/mds_cnn2 false;

hadoop2.2内存控制的几个重要参数:
YARN
yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb
yarn.nodemanager.vmem-pmem-ratio
yarn.nodemanager.resource.memory.mb
Mapreuce
Map Memory
mapreduce.map.java.opts
mapreduce.map.memory.mb
Reduce Memory
mapreduce.reduce.java.opts
mapreduce.reduce.memory.mb

阅读更多
换一批

没有更多推荐了,返回首页