spark中shuffle算子汇总

版本信息spark version 2.3.3jdk 1.8idea 2019MacBook Pro

版本信息 spark version 2.3.3 jdk 1.8 idea 2019 MacBook Pro

ShuffleDependency

我们先在idea中搜素一下ShuffleDependency可以看到,生成的依赖是ShuffleDependency的RDD有

CoGroupedRDDShuffledRDDSubtractedRDD

然后我们分别看下什么算子产生了这些RDD

ShuffledRDD

我们看到有4个算子

org.apache.spark.rdd.OrderedRDDFunctions#sortByKeyorg.apache.spark.rdd.OrderedRDDFunctions#repartitionAndSortWithinPartitionsorg.apache.spark.rdd.PairRDDFunctions#combineByKeyWithClassTagorg.apache.spark.rdd.PairRDDFunctions#partitionByorg.apache.spark.rdd.RDD#coalesce

其中combineByKeyWithClassTag 被下面的方法调用

org.apache.spark.rdd.PairRDDFunctions#combineByKeyorg.apache.spark.rdd.PairRDDFunctions#aggregateByKeyorg.apache.spark.rdd.PairRDDFunctions#foldByKeyorg.apache.spark.rdd.PairRDDFunctions#reduceByKeyorg.apache.spark.rdd.PairRDDFunctions#countApproxDistinctByKeyorg.apache.spark.rdd.PairRDDFunctions#groupByKey

所以,可知产生ShuffledRDD的算子有10个左右(有其他几个特别冷门的没写)

org.apache.spark.rdd.OrderedRDDFunctions#sortByKeyorg.apache.spark.rdd.OrderedRDDFunctions#repartitionAndSortWithinPartitionsorg.apache.spark.rdd.PairRDDFunctions#partitionByorg.apache.spark.rdd.RDD#coalesceorg.apache.spark.rdd.PairRDDFunctions#combineByKeyorg.apache.spark.rdd.PairRDDFunctions#aggregateByKeyorg.apache.spark.rdd.PairRDDFunctions#foldByKeyorg.apache.spark.rdd.PairRDDFunctions#reduceByKeyorg.apache.spark.rdd.PairRDDFunctions#countApproxDistinctByKeyorg.apache.spark.rdd.PairRDDFunctions#groupByKey

CoGroupedRDD

这三个是重载方法,所以只有1个

org.apache.spark.rdd.PairRDDFunctions#cogroup

SubtractedRDD

只有1个

org.apache.spark.rdd.PairRDDFunctions#subtractByKey

综上所述

rdd上的shuffle算子一共12个

org.apache.spark.rdd.OrderedRDDFunctions#sortByKeyorg.apache.spark.rdd.OrderedRDDFunctions#repartitionAndSortWithinPartitionsorg.apache.spark.rdd.PairRDDFunctions#partitionByorg.apache.spark.rdd.RDD#coalesceorg.apache.spark.rdd.PairRDDFunctions#combineByKeyorg.apache.spark.rdd.PairRDDFunctions#aggregateByKeyorg.apache.spark.rdd.PairRDDFunctions#foldByKeyorg.apache.spark.rdd.PairRDDFunctions#reduceByKeyorg.apache.spark.rdd.PairRDDFunctions#countApproxDistinctByKeyorg.apache.spark.rdd.PairRDDFunctions#groupByKeyorg.apache.spark.rdd.PairRDDFunctions#cogrouporg.apache.spark.rdd.PairRDDFunctions#subtractByKey

阅读全文: http://gitbook.cn/gitchat/activity/5d63f376caaaf91b3b1f2744

您还可以下载 CSDN 旗下精品原创内容社区 GitChat App ，阅读更多 GitChat 专享技术内容哦。