spark读取不存在文件会报错,当文件不存在时,能返回个空的自定义shema的dataframe是不是更好,思路如下
自定义schema,重写baseRelationToDataFrame逻辑,向resolveRelation传入false表示,当文件不存在时返回空的dataframe
代码如下
import org.apache.spark.sql.execution.datasources.DataSource
import org.apache.spark.sql.types._
scala> val priceSchema = new StructType().add("model",StringType).add("brand",StringType).add("release_date",StringType).add("price_group",StringType)
scala> var df = spark.baseRelationToDataFrame(DataSource.apply(spark,"csv",Array("/user/sdev/liujichao/all_devices.csv").toSeq,Option(priceSchema),options= Map("header" -> "true")).resolveRelation(false))
df: org.apache.spark.sql.DataFrame = [model: string, brand: string ... 2 more fields]
scala> df.show(false)
+--------+--------+---------