Spark中的UDAF简介及其Stage
udaf操作会分为两个stage:
-
- partial_merge: 本地进行merge,是一种窄依赖。tasks数量取决于上一步的partitions。
-
- merge:不同partition的数据进行merge,是一种宽依赖,需要shuffle,因此tasks数量取决于设置的值spark.default.parallelism
class MyAvg extends UserDefinedAggregateFunction {
// Input type
def inputSchema: org.apache.spark.sql.types.StructType = StructType(StructField("value", DoubleType) :: Nil)
// This is the internal fields you keep for computing your aggregate. 计算缓存
def bufferSchema: StructType = StructType( StructField("count", LongType) :: StructField("sum", DoubleType) :: Nil
)
// Return type
def dataType: DataType = DoubleType
//幂等性
def deterministic: Boolean = true
//初始值
def initialize(buffer: MutableAggregationBuffer): Unit &#