1.SparkSession获取文件内信息
val sparkConf = new SparkConf
sparkConf.setMaster("local").setAppName("TextFileTest")
sparkConf.set("spark.executor.instances", "2")
sparkConf.set("spark.executor.cores", "4")
val ss = SparkSession.builder.config(sparkConf).getOrCreate
val dada=ss.read.textFile("/Users/zora/Documents/customer/scala/demo/src/main/README.md")
dada.show()
文件内容
test erw
ee
打印结果
+--------+
| value|
+--------+
|test erw|
| ee|
+--------+
text返回 org.apache.spark.sql.DataFrame
textFile返回 org.apache.spark.sql.Dataset[scala.Predef.String]
这种方式每行是一个value
2.SparkContext读取文件信息
val conf = new SparkConf()
conf.set("spark.master", "local")
conf.set("spark.app.name", "spark demo")
val sc = new SparkContext(conf);
val textFile = sc.textFile("/Users/zora/Documents/customer/scala/demo/src/main/README.md")
val result = textFile.map { line => {
println("line", line)
line.split(" ").map(word => {
println(word)
word
})
line
}
}
result.foreach(item => println("result" + item))
输出结果
(line,test erw)
test
erw
resulttest erw
(line,ee)
ee
resultee
留意一下执行顺序,如果最后一行代码注释掉,则这段代码将不会有任何打印输出