hive 优化点

1,表分区

2,表分桶

3,index

4,尽量采用orc格式的表

5,合并小文件

6,Local mode running:

hive.exec.mode.local.auto=true  

hive.exec.mode.local.auto.inputbytes.max=50000000  

hive.exec.mode.local.auto.input.files.max=5 

7,jvm reuse:

SET mapred.job.reuse.jvm.num.tasks=5

8,Parallel execution :
hive.exec.parallel=true
hive.exec.parallel.thread.number=16

9,Common join
The common join is also called reduce side join. It is a basic join in Hive and works
for most of the time. For common joins, we need to make sure the big table is on the
right-most side or specifed by hit, as follows:
/*+ STREAMTABLE(stream_table_name) */ 

10,Map join :

hive.auto.convert.join=true

hive.mapjoin.smalltable.filesize=600000000

hive.auto.convert.join.noconditionaltask=true

hive.auto.convert.join.noconditionaltask.size=10000000;