Total MapReduce jobs = 1
14/08/24 20:29:11 WARN conf.Configuration: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/08/24 20:29:11 WARN conf.Configuration: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/08/24 20:29:11 WARN conf.Configuration: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/08/24 20:29:11 WARN conf.Configuration: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/08/24 20:29:11 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/08/24 20:29:11 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/08/24 20:29:12 WARN conf.HiveConf: DEPRECATED: Configuration property Hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Execution log at: /tmp/admin/.log
2014-08-24 08:29:12 Starting to launch local task to process map join; maximum memory = 503447552
2014-08-24 08:29:15 Processing rows: 200000 Hashtable size: 199999 Memory usage: 192705744 percentage: 0.383
2014-08-24 08:29:15 Processing rows: 300000 Hashtable size: 299999 Memory usage: 310744000 percentage: 0.617
2014-08-24 08:29:16 Processing rows: 400000 Hashtable size: 399999 Memory usage: 398019256 percentage: 0.791
2014-08-24 08:29:31 Processing rows: 500000 Hashtable size: 499999 Memory usage: 456595024 percentage: 0.907
Execution failed with exit status: 3
Obtaining error information
Task failed!
Task ID:
Stage-5
Logs:
/export/Logs/ode/admin/hive.log
FAILED: Execution Error, return code 3 from org.apache.Hadoop.hive.ql.exec.mr.MapredLocalTask
解决方法:
set hive.mapjoin.localtask.max.memory.usage=0.99
hive.mapjoin.localtask.max.memory.usage
•说明:本地任务可以使用内存的百分比 •默认值: 0.90,如果你的localtask mapjoin 表很小可以试试,但彻底解决需要
set hive.auto.convert.join=false;关闭自动mapjoin 但这个参数用的时候一定要注意,
如果你的sql 很长join会常多,关闭mapjoin任务数会成10倍激增,contener满了任务同样会非常之慢,set hive.auto.convert.join=false;一定要用在localtask级别这种超轻量及的job上。