错误:
ERROR : Ended Job = job_1631679144970_1574917 with errors
ERROR : Error during job, obtaining debugging information...
ERROR :
Task with the most failures(4):
-----
Task ID:
task_1631679144970_1574917_m_000158
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1631679144970_1574917&tipid=task_1631679144970_1574917_m_000158
-----
Diagnostic Messages for this Task:
Container [pid=18442,containerID=container_1631679144970_1574917_01_003316] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 5.8 GB of 80 GB virtual memory used. Killing container.
网上查了写资料说是
java.lang.OutOfMemoryError: Java heap space,原因是namenode内存空间不够,jvm不够新job启动导致。
解决办法是设置本地模式
set hive.exec.mode.local.auto=true;
而我这个的具体报错是running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used;
是物理内存大小不够
解决办法
set mapreduce.map.memory.mb=8192;
set mapreduce.reduce.memory.mb=8192;