–conf spark.sql.adaptive.enabled=true\ 开启shfffer解决小文件问题
–conf spark.sql.adaptive.minNumPostShufflePartitions=10
–conf spark.sql.adaptive.maxNumPostShufflePartitions=2000
–conf spark.sql.adaptive.shuffle.targetPostShuffleInputSize 134217728
–conf spark.sql.adaptive.shuffle.targetPostShuffleRowCount 10000000\
spark sql任务提交参数相关
最新推荐文章于 2023-11-27 23:47:23 发布