cdh6.2平台，oozie执行hive脚本，报 java.lang.OutOfMemoryError: Java heap space 排错的过程

最新推荐文章于 2023-06-27 10:54:37 发布

小哇666

最新推荐文章于 2023-06-27 10:54:37 发布

阅读量365

点赞数

分类专栏： # hive 文章标签： hive mapreduce jvm

本文链接：https://blog.csdn.net/qq_41712271/article/details/110388228

版权

hive 专栏收录该内容

28 篇文章 2 订阅

订阅专栏

本文探讨了Hive MapReduce作业遇到的内存溢出问题，通过调整yarn-site.xml配置、SQL设置优化，以及最终稳定的解决方案。重点在于解决Reduce阶段的ShuffleError和内存不足问题。

摘要由CSDN通过智能技术生成

Ended Job = job_1606699847024_0002 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1606699847024_0002_m_000008 (and more) from job job_1606699847024_0002
Examining task ID: task_1606699847024_0002_r_000000 (and more) from job job_1606699847024_0002

Task with the most failures(4): 
-----
Task ID:
  task_1606699847024_0002_r_000000

URL:
  http://bdcloud04:8088/taskdetails.jsp?jobid=job_1606699847024_0002&tipid=task_1606699847024_0002_r_000000
-----
Diagnostic Messages for this Task:
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#2
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
	at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
	at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:60)
	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:313)
	at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:303)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:532)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:342)
	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 9  Reduce: 1   Cumulative CPU: 483.42 sec   HDFS Read: 2202415491 HDFS Write: 0 HDFS EC Read: 0 FAIL
Total MapReduce CPU Time Spent: 8 minutes 3 seconds 420 msec

1 修改 yarn-site.xml 中 yarn.scheduler.minimum-allocation-mb 的值，原来是1，改成2 ，照样报错

参考的博客原文 https://www.jianshu.com/p/7d32a32a6e43

2 在执行的sql中添加一个设置，如下

set mapred.reduce.tasks=1;
set hive.auto.convert.join = false;
set hive.ignore.mapjoin.hint=false;
set hive.exec.parallel=true;
set hive.exec.dynamic.partition.mode=nonstrict;
-- 下面这行是后面添加的,意思是执行map前进行小文件合并
set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;

参考的博客原文 https://blog.csdn.net/weixin_42262699/article/details/88681616

经过上面的修改，初步已经稳定，也不知道是哪一步起了关键作用，运气好吧