Spark默认并行度

 并行度设置

spark.default.parallelism

Default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set by user.

默认值

For distributed shuffle operations like reduceByKey and join, the largest number of partitions in a parent RDD. For operations like parallelize with no parent RDDs, it depends on the cluster manager:

• Local mode: number of cores on the local machine
• Mesos fine grained mode: 8
• Others: total number of cores on all executor nodes or 2, whichever is larger

参考官方文档 ：

Configuration - Spark 3.5.0 Documentation