HIve任务发生Java heap space

最新推荐文章于 2024-07-24 21:14:44 发布

有语忆语

最新推荐文章于 2024-07-24 21:14:44 发布

阅读量879

点赞数

分类专栏：大数据之Hive 文章标签： hbase hadoop big data

本文链接：https://blog.csdn.net/qq_44696532/article/details/125272311

版权

大数据之Hive 专栏收录该内容

24 篇文章 2 订阅

订阅专栏

报错信息：

2022-06-14 03:23:46,863 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2022-06-14 03:23:46,932 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2022-06-14 03:23:46,933 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2022-06-14 03:23:46,943 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2022-06-14 03:23:46,980 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1640883018040_4318694, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@40005471)
2022-06-14 03:23:47,014 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2022-06-14 03:23:47,423 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data0/yarndata/nodemanager/local/usercache/pgxl/appcache/application_1640883018040_4318694,/data1/yarndata/nodemanager/local/usercache/pgxl/appcache/application_1640883018040_4318694,/data2/yarndata/nodemanager/local/usercache/pgxl/appcache/application_1640883018040_4318694,/data3/yarndata/nodemanager/local/usercache/pgxl/appcache/application_1640883018040_4318694,/data4/yarndata/nodemanager/local/usercache/pgxl/appcache/application_1640883018040_4318694
2022-06-14 03:23:47,644 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2022-06-14 03:23:48,105 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2022-06-14 03:23:48,388 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/data/hive/warehouse/wedw/ods/health_record_physicalexam_ful_d/PhysicalExam.bson:1610612736+134217728,/data/hive/warehouse/wedw/ods/health_record_physicalexam_ful_d/PhysicalExam.bson:1744830464+134217728InputFormatClass: com.mongodb.hadoop.mapred.BSONFileInputFormat

2022-06-14 03:23:48,491 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://wecloud-cluster/tmp/hive/pgxl/ac6ebd5a-a7ed-4503-981a-16eed1fe3e93/hive_2022-06-14_03-21-16_203_1158185465589876377-26159/-mr-10004/a0bab00e-650f-4dad-822c-ffcc7e1a7c7b/map.xml
2022-06-14 03:23:48,491 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: ***************non-local mode***************
2022-06-14 03:23:48,492 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: local path = hdfs://wecloud-cluster/tmp/hive/pgxl/ac6ebd5a-a7ed-4503-981a-16eed1fe3e93/hive_2022-06-14_03-21-16_203_1158185465589876377-26159/-mr-10004/a0bab00e-650f-4dad-822c-ffcc7e1a7c7b/map.xml
2022-06-14 03:23:48,492 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Open file to read in plan: hdfs://wecloud-cluster/tmp/hive/pgxl/ac6ebd5a-a7ed-4503-981a-16eed1fe3e93/hive_2022-06-14_03-21-16_203_1158185465589876377-26159/-mr-10004/a0bab00e-650f-4dad-822c-ffcc7e1a7c7b/map.xml
2022-06-14 03:23:48,507 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2022-06-14 03:23:48,508 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via kryo
2022-06-14 03:23:48,794 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: </PERFLOG method=deserializePlan start=1655148228507 end=1655148228794 duration=287 from=org.apache.hadoop.hive.ql.exec.Utilities>
2022-06-14 03:23:48,814 INFO [main] com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries
2022-06-14 03:23:48,829 INFO [main] com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 97184efe294f64a51a4c5c172cbc22146103da53]
2022-06-14 03:23:48,864 INFO [main] com.mongodb.hadoop.splitter.BSONSplitter: Found split file at : FileStatus{path=hdfs://wecloud-cluster/data/hive/warehouse/wedw/ods/health_record_physicalexam_ful_d/.PhysicalExam.bson.splits; isDirectory=false; length=540; replication=3; blocksize=268435456; modification_time=1641631726384; access_time=1655148204229; owner=pgxl; group=pgxl; permission=rw-r--r--; isSymlink=false}
2022-06-14 03:23:49,219 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:pgxl (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2022-06-14 03:23:49,221 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: java.lang.reflect.InvocationTargetException
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
	at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:265)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:212)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:332)
	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:722)
	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:251)
	... 11 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.bson.BasicBSONDecoder.readFully(BasicBSONDecoder.java:65)
	at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:57)
	at com.mongodb.hadoop.splitter.BSONSplitter.getStartingPositionForSplit(BSONSplitter.java:427)
	at com.mongodb.hadoop.mapred.BSONFileInputFormat.getRecordReader(BSONFileInputFormat.java:127)
	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)
	... 16 more

2022-06-14 03:23:49,225 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task
2022-06-14 03:23:49,330 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2022-06-14 03:23:49,330 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2022-06-14 03:23:49,331 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

Error: Java heap space 解决办法

使用MR引擎时：

set mapreduce.map.memory.mb=12000;
set mapreduce.reduce.memory.mb=12000;
set mapred.map.child.java.opts=-server -Xmx10000m -Djava.net.preferIPv4Stack=true;
set io.sort.mb=100;
set mapred.reduce.child.java.opts=-server -Xmx10000m -Djava.net.preferIPv4Stack=true;

设置 set io.sort.mb=10; 默认值是100
io.sort.mb 的作用排序所使用的内存数量。默认值：100M，需要与mapred.child.java.opts相配默认：-Xmx200m。
不能超过mapred.child.java.opt设置，否则会OOM。

默认io.sort.mb为100.但是在hql执行的时候hive log提示Java heap space.

找了很久的原因，尝试如下：

1.在mapred-site.xml中增加mapreduce.task.io.sort.mb参数

mapreduce.task.io.sort.mb 10

结果：无效

2.在core-site.xml中增加io.sort.mb参数，并设置为1

io.sort.mb #指定了排序使用的内存，大的内存可以加快 job 的处理速度。 1

结果：无效

3.在hive 命令行中手动设置io.sort.mb参数

hive> set io.sort.mb=10;

结果：成功（后来尝试发现<=80均不会报错，即使设置为100看虚拟内存也应该够用，很奇怪）

后来尝试在系统参数设置中直接把 io.sort.mb改掉，发现不管是该core-site.xml还是mapred-site.xml都没用，应该是在某一个默认参数设置中修改，但是还么有找到。

补充一点：我的hadoop是2.0.0-cdh4.2.1，mapreduce是hadoop2.0.0-mr1-cdh4.2.1(MRv1)

追加一下调查结果：

HADOOP_HOME=/home/hadoop/hadoop-2.0.0-cdh4.2.1

HADOOP_MR1_HOME=/home/hadoop/hadoop-2.0.0-mr1-cdh4.2.1

在hadoop dfs配置（HADOOP_HOME/etc/hadoop）中修改io.sort.mb参数无效，但是在同一个目录的mapred-site.xml修改后，hive查看io.sort.mb生效了。也就是说，即使hadoop2.0采用MR1的时候，虽然mapreduce在启动的时候不是在这个目录，但在start-dfs.sh执行的时候也会从这个目录去读取mapreduce的参数。

解决方法：把HADOOP_HOME/etc/hadoop/mapred-site.xml mv成其他名称如 mapred-site.xml.bak。然后在HIVE_CONF_DIR中修改mapred-site.xml中对应的io.sort.mb参数即可。

相应的，在hadoop-env.sh中，把HADOOP_HEAP增大。

使用Tez引擎时：

set hive.execution.engine=tez;
set tez.am.resource.memory.mb=9216;
set hive.exec.orc.split.strategy=BI;

有语忆语

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
0
评论
HIve任务发生Java heap space

HIve任务发生Java heap space
复制链接

扫一扫

专栏目录