Saprk aggregateByKey操作示例

最新推荐文章于 2022-10-24 16:30:25 发布

u013063153

最新推荐文章于 2022-10-24 16:30:25 发布

阅读量1.5k

点赞数

分类专栏： Spark

本文链接：https://blog.csdn.net/u013063153/article/details/73865381

版权

Spark 专栏收录该内容

138 篇文章 6 订阅

订阅专栏

aggregateByKey(zeroValue)(seqOp, combOp, [numTasks]) When called on a dataset of (K, V) pairs, returns a dataset of (K, U) pairs where the values for each key are aggregated using the given combine functions and a neutral "zero" value. Allows an aggregated value type that is different than the input value type, while avoiding unnecessary allocations. Like in groupByKey, the number of reduce tasks is configurable through an optional second argument.

aggreateByKey(zeroValue: U)(seqOp: (U, T)=> U, combOp: (U, U) =>U) 和reduceByKey的不同在于，

reduceByKey输入输出都是(K, V)，

aggreateByKey输出是(K,U)，可以不同于输入(K, V) ，

aggreateByKey的三个参数：
zeroValue: U，初始值，比如空列表{} ；
seqOp: (U,T)=> U，seq操作符，描述如何将T合并入U，比如如何将item合并到列表；
combOp: (U,U) =>U，comb操作符，描述如果合并两个U，比如合并两个列表；
所以aggreateByKey可以看成更高抽象的，更灵活的reduce或group 。
val z = sc.parallelize(List(1,2,3,4,5,6), 2)
z.aggreate(0)(math.max(_, _), _ + _)
res0: Int = 9
val z = sc.parallelize(List((1, 3), (1, 2), (1, 4), (2, 3)))
z.aggregateByKey(0)(math.max(_, _), _ + _)
res1: Array[(Int, Int)] = Array((2,3), (1,9))