Spark 外部数据源

External Data Sources

• rdbms，need JDBC jars
• Parquet、Phoenix、CSV、avro etc

已使用avro外部数据源为例：

参考【https://spark-packages.org/】这data Sources部分

With spark-shell or spark-submit

$ bin/spark-shell --packages com.databricks:spark-avro_2.11:4.0.0

使用Maven添加依赖

<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>spark-avro_2.11</artifactId>
    <version>4.0.0</version>
</dependency>

Scala API

// import needed for the .avro method to be added
import com.databricks.spark.avro._
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().master("local").getOrCreate()

// The Avro records get converted to Spark types, filtered, and
// then written back out as Avro records
val df = spark.read.avro("src/test/resources/episodes.avro")
df.filter("doctor > 5").write.avro("/tmp/output")

Alternatively you can specify the format to use instead:

val spark = SparkSession.builder().master("local").getOrCreate()
val df = spark.read
    .format("com.databricks.spark.avro")
    .load("src/test/resources/episodes.avro")

df.filter("doctor > 5").write.format("com.databricks.spark.avro").save("/tmp/output")

You can specify a custom Avro schema:

import org.apache.avro.Schema
import org.apache.spark.sql.SparkSession

val schema = new Schema.Parser().parse(new File("user.avsc"))
val spark = SparkSession.builder().master("local").getOrCreate()
spark
  .read
  .format("com.databricks.spark.avro")
  .option("avroSchema", schema.toString)
  .load("src/test/resources/episodes.avro").show()

参考【https://github.com/databricks/spark-avro#scala-api】