Spark文件读取

1.SparkSession获取文件内信息

   val sparkConf = new SparkConf
    sparkConf.setMaster("local").setAppName("TextFileTest")
    sparkConf.set("spark.executor.instances", "2")
    sparkConf.set("spark.executor.cores", "4")
    val ss = SparkSession.builder.config(sparkConf).getOrCreate
    val dada=ss.read.textFile("/Users/zora/Documents/customer/scala/demo/src/main/README.md")
    dada.show()

文件内容

test erw
ee

打印结果

+--------+
|   value|
+--------+
|test erw|
|      ee|
+--------+

text返回 org.apache.spark.sql.DataFrame
textFile返回 org.apache.spark.sql.Dataset[scala.Predef.String]
这种方式每行是一个value

2.SparkContext读取文件信息

   val conf = new SparkConf()
    conf.set("spark.master", "local")
    conf.set("spark.app.name", "spark demo")
    val sc = new SparkContext(conf);
    val textFile = sc.textFile("/Users/zora/Documents/customer/scala/demo/src/main/README.md")
    val result = textFile.map { line => {
      println("line", line)
      line.split(" ").map(word => {
        println(word)
        word
      })
      line
    }
    }
    result.foreach(item => println("result" + item))

输出结果

(line,test erw)
test
erw
resulttest erw
(line,ee)
ee
resultee

留意一下执行顺序，如果最后一行代码注释掉，则这段代码将不会有任何打印输出