Spark 将dataFrame 转为 DataSet

一、dataFrame转为DataSet

第一种方式：通过class

val s = spark.sql(
            """
              |select "1" id,1.1 f2,1.2 f3
              |union all
              |select "2" id,1.1 f2,1.1 f3
            """.stripMargin).as[Bean](Encoders.bean(classOf[Bean])).show()

class Bean extends Serializable{
    @BeanProperty var id:String = _
    @BeanProperty var f2:Double = _
    @BeanProperty var f3:Double = _
}
如果不加@BeanProperty 会报错：
    Exception in thread "main" java.lang.UnsupportedOperationException: Cannot infer type for class com.***.***.Bean because it is not bean-compliant
可见scala的bean与javaBean

 第二种：通过cass class

这种方式只需

import spark.implicits._

supported by importing spark.implicits._

 

二、dataFrame转为DataSet

DataSet.toDF即可