个人收录自己常用的集中数据保存和读取方式
1、从外部读取数据之textFile
读取
var departmentData = getLoadDepartment(sqlCon,"hdfs://cnsz17pl1443:8020/spark/ddt/ASRUA_TM_DEPARTMENT")
def getLoadDepartment(spark: SparkSession, path: String): DataFrame = {
import spark.implicits._
val departmentDF=spark.read.textFile(path).map(_.split("\u0001")).map(row =>departmentType(getString(row(0)), getString(row(1)), getString(row(2)), getString(row(3)))).persist(MEMORY_ONLY_SER)
val departmentData = departmentDF.toDF("dept_id","division_code","area_code","hq_code")
departmentData
}
保存
textFile.saveAsTextFile("file:///home/common/coding/coding/Scala/word-count/writeback")
2、hdfs的parquet类型文件读取和写入
读取 read().format()来源(json,jsonRDD,parquet)