hive任务RMContainerAllocator: REDUCE capability required is more than the supported max container

hive任务执行时：FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-12-17 09:49:08,015 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1573730956546_59894 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

查看日志原因：REDUCE分配的大小超过集群的最大container容量

2019-12-16 20:17:24,006 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:8192, vCores:1>
2019-12-16 20:17:24,029 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:16384, vCores:1>
2019-12-16 20:17:24,029 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: REDUCE capability required is more than the supported max container capability in the cluster. Killing the Job. reduceResourceRequest: <memory:16384, vCores:1> maxContainerCapability:<memory:10150, vCores:16>
2019-12-16 20:17:24,030 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: REDUCE capability required is more than the supported max container capability in the cluster. Killing the Job. reduceResourceRequest: <memory:16384, vCores:1> maxContainerCapability:<memory:10150, vCores:16>

解决方案：

1、hive shell

hive> set mapreduce.map.memory.mb=1024

hive> set mapreduce.reduce.memory.mb=4096

2、pyhive

def get_connection_hive(db, is_test=False):
    conn = hive.Connection(host='xxx', port=10000, database=db,configuration={'mapreduce.map.memory.mb': '4096','mapreduce.reduce.m
emory.mb':'2048'})
    return conn

配置完后，可能出现另外一个问题：配置的memory可能超过实际占用的memory，导致任务被kill

Diagnostic Messages for this Task:
Container [pid=25807,containerID=container_1573730956546_59979_01_000055] is running beyond physical memory limits. Current usage: 1.1 GB of 1 GB physical memory used; 8.0 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1573730956546_59979_01_000055 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 25826 25807 25807 25807 (java) 1328 170 8507072512 285103 /usr/local/java/jdk1.8.0_131/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true -Xmx6144m -Djava.io.tmpdir=/data08/yarn/nm/usercache/root/appcache/application_1573730956546_59979/container_1573730956546_59979_01_000055/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data03/yarn/container-logs/application_1573730956546_59979/container_1573730956546_59979_01_000055 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 172.16.218.71 41398 attempt_1573730956546_59979_m_000002_3 55 
        |- 25807 25805 25807 25807 (bash) 0 0 108638208 344 /bin/bash -c /usr/local/java/jdk1.8.0_131/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Djava.net.preferIPv4Stack=true -Xmx6144m -Djava.io.tmpdir=/data08/yarn/nm/usercache/root/appcache/application_1573730956546_59979/container_1573730956546_59979_01_000055/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/data03/yarn/container-logs/application_1573730956546_59979/container_1573730956546_59979_01_000055 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 172.16.218.71 41398 attempt_1573730956546_59979_m_000002_3 55 1>/data03/yarn/container-logs/application_1573730956546_59979/container_1573730956546_59979_01_000055/stdout 2>/data03/yarn/container-logs/application_1573730956546_59979/container_1573730956546_59979_01_000055/stderr  

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

1.1 GB 任务所占的物理内存

1 G 是mapreduce.map.memory.mb 设置的

8.0 GB 是程序占用的虚拟内存

2.1 GB 是mapreduce.map.memory.db 乘以 yarn.nodemanager.vmem-pmem-ratio 得到的

其中yarn.nodemanager.vmem-pmem-ratio 是 虚拟内存和物理内存比例，在yarn-site.xml中设置，默认是2.1, 由于我本地设置的是1 G， container占用了8.0 GB的虚拟内存，但是分配给container的却只有2.1 GB。所以kill掉了这个container

解决方案：合理配置memory大小

mapreduce.map.memory.mb=2048
mapreduce.reduce.memory.mb=1024
yarn.nodemanager.vmem-pmem-ratio=4

这样map运行时的虚拟内存大小为 2048*4

参考文献：

https://blog.csdn.net/weixin_33766168/article/details/93405662

https://pypi.org/project/PyHive/

https://www.cnblogs.com/gxgd/p/9628974.html