SparkSQL/hivesql参数调优

最新推荐文章于 2024-07-24 17:51:02 发布

liuwei063608

最新推荐文章于 2024-07-24 17:51:02 发布

阅读量1.7k

点赞数

分类专栏： spark

本文链接：https://blog.csdn.net/liuwei063608/article/details/90211047

版权

spark 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

set spark.sql.autoBroadcastJoinThreshold=-1;
set spark.sql.shuffle.partitions=1000;
set spark.sql.files.maxPartitionBytes=536870912;
set spark.sql.files.openCostInBytes=4194304;
set spark.sql.adaptive.shuffle.targetPostShuffleInputSize=134217728;

set hive.hadoop.supports.splittable.combineinputformat=true;
set hive.new.job.grouping.set.cardinality=128;
set mapreduce.input.fileinputformat.split.maxsize=536870912;
set mapreduce.input.fileinputformat.split.minsize=536870912;
set mapreduce.input.fileinputformat.split.minsize.per.node=536870912;
set mapreduce.input.fileinputformat.split.minsize.per.rack=536870912;