Spark 使用本地文件系统
./bin/spark-shell --conf spark.hadoop.fs.defaultFS=file:///
USE_V1_SOURCE_LIST
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
val USE_V1_SOURCE_LIST = buildConf("spark.sql.sources.useV1SourceList")
.internal()
.doc("A comma-separated list of data source short names or fully qualified data source " +
"implementation class names for which Data Source V2 code path is disabled. These data " +
"sources will fallback to Data Source V1 code path.")
.version("3.0.0")
.stringConf
.createWithDefault("avro,csv,json,kafka,orc,parquet,text")
Shuffle 相关配置
spark.default.parallelism
For distributed shuffle operations like reduceByKey and join, the largest number of partitions in a parent RDD. For operations like parallelize with no parent RDDs, it depends on the cluster manager:
Local mode: number of cores on the local machine
Mesos fine grained mode: 8
Others: total number of cores on all executor nodes or 2, whichever is larger
spark.sql.shuffle.partitions
默认值为: 200
The default number of partitions to use when shuffling data for joins or aggregations.
Note: For structured streaming, this configuration cannot be changed between query restarts from the same checkpoint location.