java heap space内存溢出

报错日志

2024-01-26 10:22:00,947 Stage-1 map = 88%,  reduce = 9%, Cumulative CPU 5377.77 sec
2024-01-26 10:22:01,986 Stage-1 map = 92%,  reduce = 9%, Cumulative CPU 5707.94 sec
2024-01-26 10:22:03,002 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 5190.82 sec
MapReduce Total cumulative CPU time: 0 days 1 hours 26 minutes 30 seconds 820 msec
Ended Job = job_1706155077266_0202 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1706155077266_0202_m_000004 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000059 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000034 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000058 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000029 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000062 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000064 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000065 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000000 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000082 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000083 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000020 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000052 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000090 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000058 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000099 (and more) from job job_1706155077266_0202
Examining task ID: task_1706155077266_0202_m_000010 (and more) from job job_1706155077266_0202

Task with the most failures(4): 
-----
Task ID:
  task_1706155077266_0202_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1706155077266_0202&tipid=task_1706155077266_0202_m_000000
-----
Diagnostic Messages for this Task:
Error: Java heap space

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 112  Reduce: 117   Cumulative CPU: 5190.82 sec   HDFS Read: 25802611154 HDFS Write: 14663299108 FAIL
Total MapReduce CPU Time Spent: 0 days 1 hours 26 minutes 30 seconds 820 msec

解决办法

• 在mapred-site.xml文件配置如下参数

set mapreduce.map.memory.mb=10150; 
set mapreduce.map.java.opts=-Xmx6144m;
# reduce阶段内存不足
set mapreduce.reduce.memory.mb=10150; 
set mapreduce.reduce.java.opts=-Xmx8120m;