Spark中的Shuffle算子

官网的话

什么是Shuffle

In Spark, data is generally not distributed across partitions to be in the necessary place for a specific operation.
During computations, a single task will operate on a single partition - thus, 
to organize all the data for a single reduceByKey reduce task to execute, 
Spark needs to perform an all-to-all operation. It must read from all partitions to find all the values for all keys, 
and then bring together values across partitions to compute the final result for each key - this is called the shuffle.

我直接复制了整段话，其实用概括起来就是：

把不同节点的数据拉取到同一个节点的过程就叫做Shuffle

有哪些Shuffle算子

Operations which can cause a shuffle include 
repartition operations like repartition and coalesce, 
‘ByKey operations (except for counting) like groupByKey and reduceByKey, 
and join operations like cogroup and join.

这一句话完美总结了Spark中Shuffle算子的分类：

• 重分区算子
  （repartition ，coalesce）
• ByKey算子
  （groupByKey ，reduceByKey）
• Join算子
  （cogroup ，join）

详细总结三类Shuffle算子

其实官网写那几个就是最常用的了

• 重分区算子

1. repartition
2. coalesce

• ByKey算子

1. groupByKey
2. reduceByKey
3. aggregateByKey
4. combineByKey
5. sortByKey
6. sortBy

• Join算子

1. cogroup
2. join
3. leftOuterJoin
4. intersection
5. subtract
6. subtractByKey
   （姑且把后面三个也放到Join类算子）

后记

官网说了三类，这里再加一类：

• 去重算子
  distinct