Hive tuning tips

1. limit
Hive has a configuration property to enable sampling of source data for use with LIMIT:
hive.limit.optimize.enable, set this parameter to true to optimize limit operation.
2. PARALLEL
if your job was designed to some stages, if these stages has no dependencies, you can execute them parallel by
setting the parameter : set hive.exec.parallel=true;
3. adjust the mapper and reducer task count
The default value of  hive.exec.reducers.bytes.per.reduceris 1 GB. Changing this
value to 750 MB causes Hive to estimate four reducers for this job:
hive> set hive.exec.reducers.bytes.per.reducer=750000000;

--combinehiveinputformat
set hive.input.format;
set mapred.child.java.opts =  -Xmx524m;
set hive.exec.reducers.bytes.per.reducer=100000000;
set hive.merge.size.per.task=10010001000;

转载于:https://www.cnblogs.com/huaxiaoyao/p/4364610.html