DataFrame与DataSet的互操作

最新推荐文章于 2024-01-18 08:15:00 发布

2401号行者

最新推荐文章于 2024-01-18 08:15:00 发布

阅读量139

点赞数

分类专栏：大数据 spark 文章标签： spark

本文链接：https://blog.csdn.net/weixin_41731982/article/details/106365552

版权

10 篇文章 0 订阅

订阅专栏

5 篇文章 0 订阅

订阅专栏

1）创建一个DateFrame

scala> val df = spark.read.json("examples/src/main/resources/people.json")
df: org.apache.spark.sql.DataFrame = [age: bigint, name: string]

2）创建一个样例类

scala> case class Person(name: String, age: Long)
defined class Person

3）将DateFrame转化为DataSet

scala> df.as[Person]
res14: org.apache.spark.sql.Dataset[Person] = [age: bigint, name: string]

scala> case class Person(name: String, age: Long)
defined class Person

2）创建DataSet

scala> val ds = Seq(Person("Andy", 32)).toDS()
ds: org.apache.spark.sql.Dataset[Person] = [name: string, age: bigint]

3）将DataSet转化为DataFrame

scala> val df = ds.toDF
df: org.apache.spark.sql.DataFrame = [name: string, age: bigint]

4）展示

scala> df.show
+----+---+
|name|age|
+----+---+
|Andy| 32|
+----+---+

这个很简单，因为只是把case class封装成Row
（1）导入隐式转换
import spark.implicits._
（2）转换
val testDF = testDS.toDF

关注