Spark Configuration

Spark 使用本地文件系统

./bin/spark-shell --conf spark.hadoop.fs.defaultFS=file:///

USE_V1_SOURCE_LIST

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

  val USE_V1_SOURCE_LIST = buildConf("spark.sql.sources.useV1SourceList")
    .internal()
    .doc("A comma-separated list of data source short names or fully qualified data source " +
      "implementation class names for which Data Source V2 code path is disabled. These data " +
      "sources will fallback to Data Source V1 code path.")
    .version("3.0.0")
    .stringConf
    .createWithDefault("avro,csv,json,kafka,orc,parquet,text")

Shuffle 相关配置

spark.default.parallelism

For distributed shuffle operations like reduceByKey and join, the largest number of partitions in a parent RDD. For operations like parallelize with no parent RDDs, it depends on the cluster manager:
Local mode: number of cores on the local machine
Mesos fine grained mode: 8
Others: total number of cores on all executor nodes or 2, whichever is larger

spark.sql.shuffle.partitions

默认值为： 200	
The default number of partitions to use when shuffling data for joins or aggregations. 
Note: For structured streaming, this configuration cannot be changed between query restarts from the same checkpoint location.