RDD之aggregate

最新推荐文章于 2021-11-29 20:19:33 发布

huanbia

最新推荐文章于 2021-11-29 20:19:33 发布

阅读量2k

点赞数 1

分类专栏： Spark 文章标签： RDD aggregate spark zeroValue

本文链接：https://blog.csdn.net/huanbia/article/details/51436822

版权

Spark 专栏收录该内容

14 篇文章 2 订阅

订阅专栏

定义

定义可参考RDD的API

aggregate[U](zeroValue: U)(seqOp: (U, T) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: ClassTag[U]): U
Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral “zero value”. This function can return a different result type, U, than the type of this RDD, T. Thus, we need one operation for merging a T into an U and one operation for merging two U’s, as in scala.TraversableOnce. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.
zeroValue
the initial value for the accumulated result of each partition for the seqOp operator, and also the initial value for the combine results from different partitions for the combOp operator - this will typically be the neutral element (e.g. Nil for list concatenation or 0 for summation)
seqOp
an operator used to accumulate results within a partition
combOp
an associative operator used to combine results from different partitions

实验1-熟悉使用

api讲的比较清楚了，该函数用来聚集每个分区的元素，并用合并函数和zeroValue来聚集分区结果。并给予我们两个函数，seqOp和CombOp

实验程序

打开spark-shell，我们执行实验1（当复制并粘贴以下代码实验时请将注释去掉）

//该函数用来将每个分区的index展示出来
def myfunc[T](index:Int,iter:Iterator[T]):Iterator[(Int,T)]={
var res = List[(Int,T)]()
for(x<-iter)
res.::=(index,x)
res.iterator
}
val data = sc.parallelize(1 to 10,3)
data.mapPartitionsWithIndex(myfunc).collect
data.aggregate(0)((a,b)=>if(a>b) a else b ,_+_)

实验结果

实验1结果

结果分析

实验2-zeroValue

api讲解如下：zeroValue值为seqOp函数的初始值，同时也是combOp函数的初始值。

实验程序

打开spark-shell，我们执行实验2（当复制并粘贴以下代码实验时请将注释去掉）

//seqOp函数
def seqOp(arg1:Int,arg2:Int):Int={
var res:Int=arg2
if(arg1>arg2)
res=arg1
println("seqOp:"+arg1+","+arg2+"=>"+res)
res
}
//combOp函数
def combOp(arg1:Int,arg2:Int):Int={
println("combOp:"+arg1+","+arg2+"=>"+(arg1+arg2))
arg1+arg2
}
//将每个分区index显示出来
def myfunc[T](index:Int,iter:Iterator[T]):Iterator[(Int,T)]={
var res = List[(Int,T)]()
for(x<-iter)
res.::=(index,x)
res.iterator
}
val data = sc.parallelize(1 to 10,3)
data.mapPartitionsWithIndex(myfunc).collect
data.aggregate(11)(seqOp,combOp)