hive hive.optimize.ppd=false导致map数量很大

hive官方https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView

Prior to Hive 0.6.0, lateral view did not support the predicate push-down optimization. In Hive 0.5.0 and earlier, if you used a WHERE clause your query may not have compiled. A workaround was to add set hive.optimize.ppd=false; before your query. The fix was made in Hive 0.6.0; see https://issues.apache.org/jira/browse/HIVE-1056: Predicate push down does not work with UDTF's.
在0.5.0版本前使用hive.optimize.ppd=false是能解决where条件不识别的。但我在使用0.9.0版本时使用了该参数,导致where不识别。
不使用参数日志:

Starting Job = job_201312261829_7430, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201312261829_7430
Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=master:8021 -kill job_201312261829_7430
Hadoop job information for Stage-1: number of mappers: 90; number of reducers: 287

使用该参数日志:

Starting Job = job_201312261829_7433, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201312261829_7433
Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=master:8021 -kill job_201312261829_7433
Hadoop job information for Stage-1: number of mappers: 4363; number of reducers: 999


所以在新版本的hive中使用hive.optimize.ppd=true;默认值即可
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值