HiveSQL 系统内存分配问题处理

HiveSQL 系统内存分配问题处理

[INFO] 2021-04-26 18:18:52.147 - [taskAppld=TASK-51-2469-19327]:[127] - -> Ended Job = job 1619390628176 0017 with errors
Error during job, obtaining debugging information.
Examining task ID: task 1619390628176_0017_m_000008(and more) from job job 1619390628176_0017
Examining task ID:task_1619390628176_0017_m_000035(and more)frob job_1619390628176_0017
Task with the most failures(4)
Task ID:
task_1619390628176_0017_m_000035
URL:
http://0.0.0.0:8088/taskdetails.jspjobid=job_1619390628176_0017&tipid=task_1619390628176_0017_m_000035
Diagnostic Messages for this Task
[2021-04-26 18:18:45.475]Exception from container-launch.
Container id:container_e75_1619390628176_0017_01_000067
Exit code:1
[2021-04-26 18:18:45.477]Container exited with a non-zero exit code 1.Error file:prelaunch.err.
Last 4096 bytes of prelaunch.err
Last 4096 bytes of stderr:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os: commit memory(0x0000000000000, 309854208, 0) failed; error
[2021-04-26 18:18:45.514]Container exited with a non-zero exit code 1.Error file:prelaunch.err.
Last 4096 bytes of prelaunch.err
Last 4096 bytes of stderr:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os: commit memory(0x000000000000, 3098008, 309854208, 0) failed; error=' Cannot allocate memory
FAILED: Execution Error, return code 2 from org. apache. hadoop. hive. ql. exec. mr. MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1:Map:467 Reduce:1 Cumulative CPU:3069.14 sec HDFS Read:0 HDFS Write:0 FAIL
Total MapReduce CPU Time Spent: 51 minutes 9 seconds 140 msec

DS调度HiveSQL时遇到该问题，由于报错os: commit memory(0x000000000000, 3098008, 309854208, 0)，检查worker节点剩余内存发现剩余内存充足，判断系统内存分配策略问题，修改配置文件/etc/sysctl.conf解决

vim /etc/sysctl.conf
vm.overcommit_memory=1
sysctl -p