把RDD保存成文件

最新推荐文章于 2023-07-04 18:12:43 发布

書盡

最新推荐文章于 2023-07-04 18:12:43 发布

阅读量1.7k

点赞数

分类专栏：大数据学习文章标签： spark

本文链接：https://blog.csdn.net/weixin_51309151/article/details/122179683

版权

大数据学习专栏收录该内容

21 篇文章 2 订阅

订阅专栏

第1种保存方法：

scala> val p= spark.read.format("json").load("file:///usr/local/spark/examples/src/main/resources/people.json")
p: org.apache.spark.sql.DataFrame = [age: bigint, name: string]
 
scala> p.select("name", "age").write.format("csv").save("file:///usr/local/spark/mycode/newpeople.csv")

//这里使用select(“name”, “age”)确定要把哪些列进行保存，然后调用write.format(“csv”).save ()保存成csv文件

write.format()支持输出 json,parquet, jdbc, orc, libsvm, csv, text等格式文件，如果要输出文本文件，可以采用write.format(“text”)

查看信息

scala> val t = sc.textFile("file:///usr/local/spark/mycode/newpeople.csv")
t: org.apache.spark.rdd.RDD[String] = file:///usr/local/spark/mycode/newpeople.csv MapPartitionsRDD[1] at textFile at <console>:24
scala> t.foreach(println)
Justin,19
Michael,
Andy,30

第2种保存方法：

scala> val p = spark.read.format("json").load("file:///usr/local/spark/examples/src/main/resources/people.json")
p: org.apache.spark.sql.DataFrame = [age: bigint, name: string]
 
scala> df.rdd.saveAsTextFile("file:///usr/local/spark/mycode/newpeople.txt")

查看

scala> val t = sc.textFile("file:///usr/local/spark/mycode/newpeople.txt")
t: org.apache.spark.rdd.RDD[String] = file:///usr/local/spark/mycode/newpeople.txt MapPartitionsRDD[11] at textFile at <console>:28
 
scala> t.foreach(println)
[null,Michael]
[30,Andy]
[19,Justin]