Spark自定义UDAF函数（强类型-DSL语法）

最新推荐文章于 2024-01-28 11:34:08 发布

CyAuroras

最新推荐文章于 2024-01-28 11:34:08 发布

阅读量247

点赞数

分类专栏： # Spark 文章标签：大数据 spark

本文链接：https://blog.csdn.net/CyAurora/article/details/119790290

版权

Spark 专栏收录该内容

20 篇文章 0 订阅

订阅专栏

该博客演示了如何在Scala中创建一个自定义的Spark SQL聚合函数`CustomerAvg`来计算用户年龄的平均值。通过继承`Aggregator`，实现了`reduce`、`merge`和`finish`等方法，并将结果转换为DataFrame列进行展示。在运行过程中遇到了类型转换异常，因为尝试将`BigInt`转换为`Int`可能导致数据丢失。

摘要由CSDN通过智能技术生成

1.继承Aggregator
2.实现方法
3.注册函数
4.通过DataSet数据集获取结果

package com.wxx.bigdata.sql03
 
import org.apache.spark.sql.{Encoders, SparkSession}
import org.apache.spark.sql.expressions.Aggregator
 
object CustomerUDAFClassAPP {
  def main(args: Array[String]): Unit = {
 
    val spark = SparkSession.builder().master("local[2]").appName("CustomerUDFClassAPP").getOrCreate()
 
    val df = spark.read.json("data/test/user.json")
    import spark.implicits._
    //转换为DateSet
    val ds = df.as[Users]
 
    //将聚合函数转换为查询的列
    val avgage = CustomerAvg.toColumn.name("avgAge")
    //应用函数
    ds.select(avgage).show()
 
    spark.stop()
  }
 
}
//Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot up cast `age` from bigint to int as it may truncate
//The type path of the target object is:
//- field (class: "scala.Int", name: "age")
//- root class: "com.wxx.bigdata.sql03.Users"
case class Users(name : String, age : BigInt)
case class AvgBuffer(var sum : BigInt, var count : Int)
 
object CustomerAvg extends Aggregator[Users, AvgBuffer, Double]{
  override def zero = {
    AvgBuffer(0, 0)
  }
  // exector中的合并
  override def reduce(b: AvgBuffer, a: Users) = {
    b.sum =  b.sum + a.age
    b.count = b.count + 1
    b
  }
  //缓冲区的合并操作
  override def merge(b1: AvgBuffer, b2: AvgBuffer) = {
    b1.sum = b1.sum + b2.sum
    b1.count = b1.count + b2.count
    b1
  }
 
  //完成计算
  override def finish(reduction: AvgBuffer) = {
    reduction.sum.toDouble / reduction.count
  }
 
  override def bufferEncoder = Encoders.product
 
  override def outputEncoder = Encoders.scalaDouble
}