Spark DSL各场景解决方案 - 汇总

1、DSL 字段数据类型转换。

val session = SparkSession.builder()
			.appName(this.getClass.getSimpleName).master("local[2]")
			.getOrCreate()
val nameRDD: RDD[String] = session.sparkContext.makeRDD(Array(
    """
      |{"name":"zhangsan","age":"18"}
      """.stripMargin
    ,
    """
      |{"name":"lisi","age":"11"}
      """.stripMargin
  ))
  val nameDF: DataFrame = session.read.json(nameRDD)
  nameDF.select(col("name"),col("age") cast(DecimalType(2,2)))
  		.select('name,'age cast(DoubleType))

2、DSL 字段聚合类型操作 , 字段别名

nameDF.agg(avg('age),sum('age),sum("age") cast(DecimalType(10,2)) alias("nc"))