-
复制所有优化命令
--分区 set hive.exec.dynamic.partition=true; set hive.exec.dynamic.partition.mode=nonstrict; set hive.exec.max.dynamic.partitions.pernode=10000; set hive.exec.max.dynamic.partitions=100000; set hive.exec.max.created.files=150000; --hive压缩 set hive.exec.compress.intermediate=true; set hive.exec.compress.output=true; --写入时压缩生效 set hive.exec.orc.compression.strategy=COMPRESSION; --分桶 set hive.enforce.bucketing=true; set hive.enforce.sorting=true; set hive.optimize.bucketmapjoin = true; set hive.auto.convert.sortmerge.join=true; set hive.auto.convert.sortmerge.join.noconditionaltask=true; --并行执行 set hive.exec.parallel=true; set hive.exec.parallel.thread.number=8; --小文件合并 -- set mapred.max.split.size=2147483648; -- set mapred.min.split.size.per.node=1000000000; -- set mapred.min.split.size.per.rack=1000000000; --矢量化查询 set hive.vectorized.execution.enabled=true
Hive表插入数据前的优化,分区优化、分桶优化、数据倾斜优化、并行优化等等...
于 2022-02-17 14:32:44 首次发布
![](https://img-home.csdnimg.cn/images/20240711042549.png)