hive常见报错：Execution failed with exit status: 3

最新推荐文章于 2023-07-12 23:17:49 发布
最新推荐文章于 2023-07-12 23:17:49 发布
阅读量4.1k 收藏 1
点赞数 1
分类专栏： hive
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_18838991/article/details/51841280
版权

sql：

select '$v_date','$v_prov','ps',cell_cnt,misidn_cnt,imsi_cnt,imei_cnt,total_cnt,A.rantype from 
(select /*+mapjoin(tb2)*/ 'aa' tt,count(*) cell_cnt,rantype from
snl_dwd.dwd_d_sa_basic_normal tb1
join snl_dwa.dim_all_station_cellid tb2
on tb1.month_id='$v_month' and tb1.day_id='$v_day' and tb1.prov_id='$v_prov'  and tb1.sa_type='ps' and tb1.current_lac=tb2.lac and tb1.current_ci=tb2.cellid
group by rantype) A
join 
(select 'aa' tt,sum(case when is_num(misidn)=true then  1  else     0  end) misidn_cnt,
sum( case    when is_num(imsi)=true then 1 else 0 end) imsi_cnt ,
sum( case    when is_num(imei)=true then 1 else 0 end) imei_cnt,
sum(1) total_cnt,
rantype
from snl_dwd.dwd_d_sa_basic_normal
where month_id='$v_month' and day_id='$v_day' and prov_id='$v_prov'  and sa_type='ps' 
group by rantype) B
on A.tt=B.tt and A.rantype=B.rantype;

这里小表是snl_dwa.dim_all_station_cellid ，大表是：snl_dwd.dwd_d_sa_basic_normal


报错信息：

16/07/06 17:35:21 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

16/07/06 17:35:21 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/07/06 17:35:21 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/07/06 17:35:21 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/07/06 17:35:21 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/07/06 17:35:21 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/07/06 17:35:21 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative


Logging initialized using configuration in file:/opt/beh/core/hive/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/beh/core/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/beh/core/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
OK
Time taken: 0.497 seconds
Added /data/all_signal_dev/UDF_wuy.jar to class path
Added resource: /data/all_signal_dev/UDF_wuy.jar
OK
Time taken: 0.392 seconds
Total MapReduce jobs = 9
Launching Job 1 out of 9
Number of reduce tasks not specified. Estimated from input data size: 9
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Stage-16 is selected by condition resolver.
Stage-1 is filtered out by condition resolver.
Starting Job = job_1460530329895_286504, Tracking URL = http://DSJ-signal-4T-206:23188/proxy/application_1460530329895_286504/
Kill Command = /opt/beh/core/hadoop/bin/hadoop job  -kill job_1460530329895_286504
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/beh/core/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/beh/core/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/07/06 17:35:28 WARN conf.Configuration: file:/tmp/all_signal_dev/hive_2016-07-06_17-35-22_865_2627282251966464053-1/-local-10016/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
16/07/06 17:35:28 WARN conf.Configuration: file:/tmp/all_signal_dev/hive_2016-07-06_17-35-22_865_2627282251966464053-1/-local-10016/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/07/06 17:35:28 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
Execution log at: /tmp/all_signal_dev/all_signal_dev_20160706173535_b9069bd9-dd43-4e2f-9873-c2b832d73eff.log
2016-07-06 05:35:29     Starting to launch local task to process map join;      maximum memory = 514850816
Hadoop job information for Stage-6: number of mappers: 111; number of reducers: 9
2016-07-06 17:35:30,239 Stage-6 map = 0%,  reduce = 0%
2016-07-06 05:35:30     Processing rows:        200000  Hashtable size: 199999  Memory usage:   154090568       percentage:     0.299
2016-07-06 05:35:30     Processing rows:        300000  Hashtable size: 299999  Memory usage:   64363592        percentage:     0.125
2016-07-06 05:35:30     Processing rows:        400000  Hashtable size: 399999  Memory usage:   81483296        percentage:     0.158
2016-07-06 05:35:30     Processing rows:        500000  Hashtable size: 499999  Memory usage:   91823584        percentage:     0.178
2016-07-06 05:35:31     Processing rows:        600000  Hashtable size: 599999  Memory usage:   104748984       percentage:     0.203
2016-07-06 05:35:31     Processing rows:        700000  Hashtable size: 699999  Memory usage:   117674344       percentage:     0.229
2016-07-06 05:35:31     Processing rows:        800000  Hashtable size: 799999  Memory usage:   136403224       percentage:     0.265
2016-07-06 05:35:31     Processing rows:        900000  Hashtable size: 899999  Memory usage:   149328584       percentage:     0.29
2016-07-06 05:35:31     Processing rows:        1000000 Hashtable size: 999999  Memory usage:   162253984       percentage:     0.315
2016-07-06 05:35:31     Processing rows:        1100000 Hashtable size: 1099999 Memory usage:   172594272       percentage:     0.335
2016-07-06 05:35:31     Processing rows:        1200000 Hashtable size: 1199999 Memory usage:   185519632       percentage:     0.36
2016-07-06 05:35:32     Processing rows:        1300000 Hashtable size: 1299999 Memory usage:   190954904       percentage:     0.371
2016-07-06 05:35:32     Processing rows:        1400000 Hashtable size: 1399999 Memory usage:   203699304       percentage:     0.396
2016-07-06 05:35:32     Processing rows:        1500000 Hashtable size: 1499999 Memory usage:   216443704       percentage:     0.42
2016-07-06 05:35:32     Processing rows:        1600000 Hashtable size: 1599999 Memory usage:   243416472       percentage:     0.473
2016-07-06 05:35:32     Processing rows:        1700000 Hashtable size: 1699999 Memory usage:   256160872       percentage:     0.498
2016-07-06 05:35:32     Processing rows:        1800000 Hashtable size: 1799999 Memory usage:   268905272       percentage:     0.522
2016-07-06 05:35:33     Processing rows:        1900000 Hashtable size: 1899999 Memory usage:   281649664       percentage:     0.547
2016-07-06 05:35:33     Processing rows:        2000000 Hashtable size: 1999999 Memory usage:   291845192       percentage:     0.567
Execution failed with exit status: 3
Obtaining error information


Task failed!
Task ID:
  Stage-16


Logs:


/opt/beh/logs/hive/all_signal_dev/hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
Launching Job 3 out of 9
Number of reduce tasks not specified. Estimated from input data size: 9
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_1460530329895_286507, Tracking URL = http://DSJ-signal-4T-206:23188/proxy/application_1460530329895_286507/
Kill Command = /opt/beh/core/hadoop/bin/hadoop job  -kill job_1460530329895_286507
Hadoop job information for Stage-1: number of mappers: 117; number of reducers: 9
2016-07-06 17:35:39,056 Stage-1 map = 0%,  reduce = 0%
2016-07-06 17:35:41,011 Stage-6 map = 1%,  reduce = 0%, Cumulative CPU 4.9 sec
2016-07-06 17:35:42,064 Stage-6 map = 2%,  reduce = 0%, Cumulative CPU 11.78 sec
2016-07-06 17:35:43,091 Stage-6 map = 2%,  reduce = 0%, Cumulative CPU 26.8 sec
2016-07-06 17:35:44,122 Stage-6 map = 4%,  reduce = 0%, Cumulative CPU 86.76 sec
2016-07-06 17:35:45,152 Stage-6 map = 4%,  reduce = 0%, Cumulative CPU 89.92 sec
2016-07-06 17:35:46,179 Stage-6 map = 5%,  reduce = 0%, Cumulative CPU 96.0 sec
2016-07-06 17:35:47,209 Stage-6 map = 7%,  reduce = 0%, Cumulative CPU 188.29 sec
2016-07-06 17:35:48,239 Stage-6 map = 7%,  reduce = 0%, Cumulative CPU 262.25 sec
2016-07-06 17:35:49,269 Stage-6 map = 8%,  reduce = 0%, Cumulative CPU 346.72 sec
2016-07-06 17:35:50,299 Stage-6 map = 10%,  reduce = 0%, Cumulative CPU 492.01 sec
2016-07-06 17:35:51,329 Stage-6 map = 10%,  reduce = 0%, Cumulative CPU 594.49 sec
2016-07-06 17:35:52,357 Stage-6 map = 11%,  reduce = 0%, Cumulative CPU 724.09 sec
2016-07-06 17:35:52,515 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 71.99 sec
2016-07-06 17:35:53,386 Stage-6 map = 17%,  reduce = 0%, Cumulative CPU 898.56 sec
2016-07-06 17:35:53,541 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 127.15 sec


解决方法：设置set hive.auto.convert.join=false;

默认情况下，hive是自动把左表当做小表加载到内存里，这里设置/*+mapjoin(tb2)*/，是想强制把tb2表当做小表放到内存里，但是在这里看起来不管用 。


官方FAQ解释：

Execution failed with exit status: 3

Execution failed with exit status: 3
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

Hive converted a join into a locally running and faster 'mapjoin', but ran out of memory while doing so. There are two bugs responsible for this.

Bug 1)

hives metric for converting joins miscalculated the required amount of memory. This is especially true for compressed files and ORC files, as hive uses the filesize as metric, but compressed tables require more memory in their uncompressed 'in memory representation'.

You could simply decrease 'hive.smalltable.filesize' to tune the metric, or increase 'hive.mapred.local.mem' to allow the allocation of more memory for map tasks.

The later option may lead to bug number two if you happen to have a affected hadoop version.

Bug 2)

Hive/Hadoop ignores 'hive.mapred.local.mem' ! (more exactly: bug in Hadoop 2.2 where hadoop-env.cmd sets the -xmx parameter multiple times, effectively overriding the user set hive.mapred.local.mem setting. see: https://issues.apache.org/jira/browse/HADOOP-10245

There are 3 workarounds for this bug:

  • 1) assign more memory to the local! Hadoop JVM client (this is not! mapred.map.memory) because map-join child jvm will inherit the parents jvm settings
    • In cloudera manager home, click on "hive" service,
    • then on the hive service page click on "configuration"
    • Gateway base group --(expand)--> Resource Management -> Client Java Heap Size in Bytes -> 1GB
  • 2) reduce "hive.smalltable.filesize" to ~1MB or below (depends on your cluster settings for the local JVM)
  • 3) turn off "hive.auto.convert.join" to prevent hive from converting the joins to a mapjoin.

2) & 3) can be set in Big-Bench/engines/hive/conf/hiveSettings.sql



寒郊無留影
关注
  • 1
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Hive Beeline连接报错：Could not open client transport with JDBC Uri: jdbc:hive2://hcmaster:10000/default
01-20
在使用Hive的Beeline客户端进行连接时，可能会遇到错误提示"Could not open client transport with JDBC Uri: jdbc:hive2://hcmaster:10000/default"。这个错误通常意味着Beeline无法通过JDBC URL成功建立到Hive ...
数据仓库中的 SQL 性能优化（Hive篇）
weixin_34198583的博客
08-31 646
2019独角兽企业重金招聘Python工程师标准>>> ...
[hive]return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. GC overhead limit exceeded
胖胖的博客
11-10 2483
2、group by 数据倾斜。1、调大reduce个数。
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask
weixin_46705696的博客
11-12 2810
xxx has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 52.0
Hadoop v2(Yarn), HBase, Hive 异常处理
那闯
03-09 3293
1. Unhealthy Node     症状：执行MR在submit Job后停止， 不出现Map的进度，一直处在Pending状态，     ResourceManager和NodeManager的Log中无其他异常，在RM开启时的Log中显示节点Unhealthy。     方法：1). yarn-site.xml配置yarn.nodemanager.local-dirs
Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. PermGen space
挣扎技术
06-04 9485
Hive 执行group by 出现错误 FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. PermGen space 参考 数据倾斜以及本地模式的介绍 解决方法：改为本地模式 1、开启本地mr set hive.exec.mode.local.auto=tr...
（hive） Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask解决方案
WJWFighting的博客
08-14 2万+
解决bug思路： 出现这个问题说明你的mapreduce.input.fileinputformat.split.maxsize设置的太小了 解决方案：hive (default)&gt; set mapreduce.input.fileinputformat.split.maxsize=。。。。;设置最大切片值为。。。个字节 这个数取决于你的文件大小，假设下：你的文件50M，你就把set ...
hive on tez 常见报错问题收集
01-26
以上就是针对Hive on Tez常见报错的分析及解决方案。在实际操作中，应根据具体环境和需求调整相关配置，确保Hive作业的稳定性和效率。同时，对于复杂的查询场景，理解Hive和Tez的工作原理以及它们之间的交互至关重要...
Could not resolve dependencies for project org.apache.hive:hive-exec:jar:2.3.0:
09-10
hive 开发UDF 使用maven工程 引发jar包缺失 hive 开发UDF 使用maven工程 引发jar包缺失
hive_labs:Hive，Sqoop相关实验室
05-10
在大数据处理领域，Hive和Sqoop是两个非常重要的工具，它们在数据仓库和数据迁移方面发挥着关键作用。在这个“Hive，Sqoop相关实验室”中，我们将深入探讨这两个工具的功能、用法以及如何结合使用它们来实现高效的...
Hive:Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.FunctionTask Ask
张小丑
04-28 9452
I wrote atest.sqlfile and ran it with hive. I got the error when hive was parsing to the code(show below):[CODE]: CREATE TEMPORARY FUNCTION foo AS "com.mytest.udaf.HelloWorldFoo"; [ERROR]: This que...
hive报错：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
最新发布
m0_65097405的博客
07-12 3916
For more details see: http://wiki.apache.org/hadoop/ConnectionRefused)原因： 之前学习hive的时候创建的数据库lyw，那时已经在hdfs中创建了相对应的文件夹，后来把这些文件夹删除了，导致lyw数据库在hdfs中找不到其对应的文件夹lyw，无法创建lyw的子文件夹，因此无法在数据库lyw下创建表。解决方法：重新建一个数据库，然后把数据加载到新数据库。
已解决：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
qq_43408367的博客
01-08 8276
## 问题描述 1. 今天在跑hive sql的时候，报错;FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask【从hive log找到的错误提示】，蛮无语的，也看不出啥原因导致的。。。 2. 在网上查了蛮久的，找到一个解决方案 ## 解决方案 【[参考方案](https://blog.csdn.net/decsjdz/article/details/94561573)】 1. 参考错
Hive中运行任务报错：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
热门推荐
IT Baicn的博客
01-14 6万+
一 错误 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 二 错误原因 第一种，yarn资源不足 原因： 该错误是YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用...
hive执行任务报错Execution failed with exit status: 3
weixin_30443075的博客
07-31 288
2014-12-18 03:21:03 Starting to launch local task to process map join; maximum memory = 1065484288 2014-12-18 03:21:08 Processing rows: 200000 Hashtable size: 199999 Memory usage: 112049704 percentag...
Hive执行 报错：FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
goTsHgo的博客
12-08 7924
项目场景： HiveOnSpark环境下执行hive脚本，报错。 问题描述： 提示：这里描述项目中遇到的问题： 例如：数据传输过程中数据不时出现丢失的情况，偶尔会丢失一部分数据 APP 中接收数据代码： @Override public void run() { bytes = mmInStream.read(buffer); mHandler.obtainMessage(READ_DATA, bytes, -1, buffer).sendToTarget(); } 原因分析： ..
hive使用tez执行引擎insert报错
weixin_44870066的博客
04-08 3222
hive中配置号tez可以正常建表，但是insert报错 【错误提示】 FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider and com.sun.jersey.json.impl.provider.entity.JSONRootElementProv
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask错误的解决办法（图文详解）
ReyzeLamp的博客
11-23 7127
hbase迁移数据到hive中 hive创建表关联hbase表时报错 CREATE EXTERNAL TABLE test01 (key string, name string, age string, sex string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIE...
Hive报错：Failed with exception Unable to rename
06-13
这个报错一般是因为Hive在执行`ALTER TABLE`等操作时，需要将原表重命名为临时表，然后将新表重命名为原表，但是重命名操作失败了。 可能的原因有： 1. 操作系统权限不足或磁盘空间不足，导致无法进行重命名操作。...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值