java hadoop优化_hive&hadoop 配置项的调优-Java heap space

hive运行详细报错WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

Query ID = root_20201212183932_270ed399-a426-4f98-88d2-3c33c5433379

Total jobs = 1

Launching Job 1 out of 1

Number of reduce tasks determined at compile time: 1

In order to change the average load for a reducer (in bytes):

set hive.exec.reducers.bytes.per.reducer=

In order to limit the maximum number of reducers:

set hive.exec.reducers.max=

In order to set a constant number of reducers:

set mapreduce.job.reduces=

Cannot run job locally: Input Size (= 94395226) is larger than hive.exec.mode.local.auto.inputbytes.max (= 50000000)

Starting Job = job_1607739316117_0009, Tracking URL = http://bigdata130:8088/proxy/application_1607739316117_0009/

Kill Command = /opt/server/hadoop-2.7.7/bin/hadoop job -kill job_1607739316117_0009

Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1

2020-12-12 18:39:35,991 Stage-1 map = 0%, reduce = 0%

2020-12-12 18:39:50,312 Stage-1 map = 100%, reduce = 100%

Ended Job = job_1607739316117_0009 with errors

Error during job, obtaining debugging information...

Examining task ID: task_1607739316117_0009_m_000000 (and more) from job job_1607739316117_0009

Task with the most failures(4):

-----

Task ID:

task_1607739316117_0009_m_000000

URL:

http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1607739316117_0009&tipid=task_1607739316117_0009_m_000000

-----

Diagnostic Messages for this Task:

Error: Java heap space

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

MapReduce Jobs Launched:

Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL

Total MapReduce CPU Time Spent: 0 msec

解决方案

方案1

设置 set io.sort.mb=10; 默认值是100

io.sort.mb 的作用 排序所使用的内存数量。 默认值:100M,需要与mapred.child.java.opts相配 默认:-Xmx200m。

不能超过mapred.child.java.opt设置,否则会OOM。

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值