4.Spark Core 应用解析之RDD常用转换操作

        RDD中的所有转换都是延迟加载的，也就是说，它们并不会直接计算结果。相反的，它们只是记住这些应用到基础数据集（例如一个文件）上的转换动作。只有当发生一个要求返回结果给Driver的动作时，这些转换才会真正运行。这种设计让Spark更加有效率地运行

1.map(func)

返回一个新的RDD，该RDD由每一个输入元素经过func函数转换后组成

将原来 RDD 的每个数据项通过 map 中的用户自定义函数 f 映射转变为一个新的元素。源码中的map算子相当于初始化一个RDD，新RDD叫做 MappedRDD(this, sc.clean(f))

scala> var source = sc.parallelize(1 to 10)
source: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> source.collect()
res0: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> val maprdd = source.map(_ * 2)
maprdd: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[1] at map at <console>:26

scala> maprdd.collect()
res1: Array[Int] = Array(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)

2.filter(func)

返回一个新的RDD，该RDD由经过func函数计算后返回值为 true 的输入元素组成

scala> var sourceFilter = sc.parallelize(Array("Java", "Scala", "Python"))
sourceFilter: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[2] at parallelize at <console>:24

scala> val filter = sourceFilter.filter(_.contains("Scala"))
filter: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[3] at filter at <console>:26

scala> sourceFilter.collect()
res2: Array[String] = Array(Java, Scala, Python)

scala> filter.collect()
res3: Array[String] = Array(Scala)

3.flatMap(func)

类似于map，但是每一个输入元素可以被映射为0或多个输出元素（所以func应该返回一个序列，而不是单一的元素）

将原来 RDD 中的每个元素通过函数 f 转换为新的元素，并将生成的 RDD 的每个集合中元素合并为一个集合。内部创建FlatMapedRDD(this, sc.clean(f))

如下，外部大方框可以认为是一个RDD分区，小方框代表一个集合，V1、V2、V3在一个集合作为RDD的一个数据项，转换为V’ 1、V‘ 2、V’ 3 后，将集合拆散，形成为RDD的数据项，U1、U2及M1、M2也是一样

scala> var sourceFlat = sc.parallelize(1 to 5)
sourceFlat: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[4] at parallelize at <console>:24

scala> sourceFlat.collect()
res4: Array[Int] = Array(1, 2, 3, 4, 5)

scala> val flatMap1 = sourceFlat.flatMap(x => (1 to x))
flatMap1: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[5] at flatMap at <console>:26

scala> val flatMap2 = sourceFlat.flatMap(1 to _)
flatMap2: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[6] at flatMap at <console>:26

scala> flatMap1.collect()
res5: Array[Int] = Array(1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5)

scala> flatMap2.collect()
res6: Array[Int] = Array(1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5)

4.mapPartitions(func)

和map函数类似，只不过映射函数的参数由RDD中的每一个元素变成了RDD中每一个分区的迭代器。如果在映射的过程中需要频繁创建额外的对象，使用mapPartitions要比map高效的多

def mapPartitions[U](f: (Iterator[T]) => Iterator[U], preservesPartitioning: Boolean = false)(implicit arg0: ClassTag[U]): RDD[U]

比如，将RDD中的所有数据通过JDBC连接写入数据库，如果使用map函数，可能要为每一个元素都创建一个connection，这样开销很大，如果使用mapPartitions，那么只需要针对每一个分区建立一个connection
参数preservesPartitioning表示是否保留父RDD的partitioner分区信息

scala> var rdd1 = sc.makeRDD(1 to 5, 2)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at makeRDD at <console>:24

scala> var rdd2 = rdd1.mapPartitions{ x => {
     | var result = List[Int]()
     |     var i = 0
     |     while(x.hasNext) {
     |         i += x.next()
     |     }
     |     result.::(i).iterator
     | }}
rdd2: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[2] at mapPartitions at <console>:26

scala> rdd2.collect
res0: Array[Int] = Array(3, 12)                                                 

scala> rdd2.partitions.size
res1: Int = 2

5.mapPartitionsWithIndex(func)

类似于mapPartitions，但是提供了两个参数，第一个参数为分区的索引

def mapPartitionsWithIndex[U](f: (Int, Iterator[T]) => Iterator[U], preservesPartitioning: Boolean = false)(implicit arg0: ClassTag[U]): RDD[U]

scala> var rdd1 = sc.makeRDD(1 to 5, 2)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[6] at makeRDD at <console>:24

scala> var rdd2 = rdd1.mapPartitionsWithIndex{
     |     (x, iter) => {
     |         var result = List[String]()
     |             var i = 0
     |             while(iter.hasNext){
     |                 i += iter.next()
     |             }
     |             result.::(x + "|" + i).iterator
     |      }
     | }
rdd2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[7] at mapPartitionsWithIndex at <console>:26

scala> rdd2.collect
res2: Array[String] = Array(0|3, 1|12)

scala> rdd2.partitions.size
res3: Int = 2

6.sample(withReplacement, fraction, seed)

以指定的随机种子随机抽样除数量为fraction的数据

withReplacement表示是抽出的数据是否放回

• true：元素放回，返回的子集会有重复，可以被多次抽样
• false：元素不放回，返回的子集没有重复

fraction：期望样本的大小作为RDD大小的一部分
seed：用于指定随机数生成器种子。如下图例子从RDD中随机且有放回的抽出50%的数据，随机种子值为3（即可能以1、2、3 的其中一个起始值）

scala> val rdd = sc.parallelize(1 to 10)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> rdd.collect()
res0: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> var sample1 = rdd.sample(true, 0.4, 2)
sample1: org.apache.spark.rdd.RDD[Int] = PartitionwiseSampledRDD[1] at sample at <console>:26

scala> sample1.collect()
res1: Array[Int] = Array(1, 2, 2, 6, 6, 10)

scala> var sample2 = rdd.sample(false, 0.2, 3)
sample2: org.apache.spark.rdd.RDD[Int] = PartitionwiseSampledRDD[2] at sample at <console>:26

scala> sample2.collect()
res2: Array[Int] = Array(1)

7.takeSample()

和Sample区别是，takeSample返回的是最终的结果集合

scala> val rdd = sc.parallelize(1 to 10)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> rdd.collect()
res0: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> var takeSample1 = rdd.takeSample(true, 4, 2)
takeSample1: Array[Int] = Array(9, 4, 10, 4)

scala> for(x <- takeSample1) println(x)
9
4
10
4

scala> var takeSample2 = rdd.takeSample(false, 3, 3)
takeSample2: Array[Int] = Array(6, 4, 2)

scala> for(x <- takeSample2) println(x)
6
4
2

8.union(otherDataset)

对源RDD和参数RDD求并集后返回一个新的RDD

scala> val rdd1 = sc.parallelize(1 to 5)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> rdd1.collect()
res0: Array[Int] = Array(1, 2, 3, 4, 5)

scala> val rdd2 = sc.parallelize(5 to 10)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:24

scala> rdd2.collect()
res1: Array[Int] = Array(5, 6, 7, 8, 9, 10)

scala> val rdd3 = rdd1.union(rdd2)
rdd3: org.apache.spark.rdd.RDD[Int] = UnionRDD[2] at union at <console>:28

scala> rdd3.collect()
res2: Array[Int] = Array(1, 2, 3, 4, 5, 5, 6, 7, 8, 9, 10)

9.intersection(otherDataset)

对源RDD和参数RDD求交集后返回一个新的RDD

scala> val rdd1 = sc.parallelize(1 to 7)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[3] at parallelize at <console>:24

scala> val rdd2 = sc.parallelize(5 to 10)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[4] at parallelize at <console>:24

scala> val rdd3 = rdd1.intersection(rdd2)
rdd3: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[10] at intersection at <console>:28

scala> rdd3.collect()
res3: Array[Int] = Array(6, 7, 5)

10.distinct([numTasks])

对源RDD进行去重后返回一个新的RDD，默认情况下，只有8个并行任务来操作，可以传入一个可选的numTasks参数改变它

scala> val rdd = sc.parallelize(List(1, 2, 2, 4, 6, 7, 8, 8))
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[11] at parallelize at <console>:24

scala> val distinctRDD = rdd.distinct()
distinctRDD: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[14] at distinct at <console>:26

scala> distinctRDD.collect()
res4: Array[Int] = Array(4, 1, 6, 7, 8, 2)

scala> val distinctRDD2 = rdd.distinct(2)
distinctRDD2: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[17] at distinct at <console>:26

scala> distinctRDD2.collect()
res5: Array[Int] = Array(4, 6, 8, 2, 1, 7)

11.partitionBy()

对RDD进行分区操作，如果原有的partitionRDD和现有的partitionRDD是一致的话，就不进行分区，否则会生成ShuffleRDD

scala> val rdd = sc.parallelize(Array((1, "a"), (2, "b"), (3, "c"), (4, "d")), 4)
rdd: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[18] at parallelize at <console>:24

scala> rdd.partitions.size
res6: Int = 4

scala> val rdd2 = rdd.partitionBy(new org.apache.spark.HashPartitioner(2))
rdd2: org.apache.spark.rdd.RDD[(Int, String)] = ShuffledRDD[20] at partitionBy at <console>:26

scala> rdd2.partitions.size
res9: Int = 2

12.reduceByKey(func, [numTasks])

在一个(K, V)的RDD上调用，返回一个(K, V)的RDD，使用指定的reduce函数，将相同的key聚合在一起，reduce任务的个数可以通过第二个可选的参数来设置

scala> val rdd = sc.parallelize(List(("Apple", 1), ("Apple", 4), ("Pear", 4), ("Pear", 3)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[21] at parallelize at <console>:24

scala> val result = rdd.reduceByKey((x, y) => x + y)
result: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[22] at reduceByKey at <console>:26

scala> result.collect()
res10: Array[(String, Int)] = Array((Apple,5), (Pear,7))

13.groupByKey()

groupByKey也是对每个Key进行操作，但只生成一个Sequence

scala> val words = Array("one", "two", "two", "three", "three", "three")
words: Array[String] = Array(one, two, two, three, three, three)

scala> val wordPairsRDD = sc.parallelize(words).map(word => (word, 1))
wordPairsRDD: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[24] at map at <console>:26

scala> val group = wordPairsRDD.groupByKey()
group: org.apache.spark.rdd.RDD[(String, Iterable[Int])] = ShuffledRDD[25] at groupByKey at <console>:28

scala> group.collect()
res11: Array[(String, Iterable[Int])] = Array((two,CompactBuffer(1, 1)), (one,CompactBuffer(1)), (three,CompactBuffer(1, 1, 1)))

scala> val res = group.map(t => (t._1, t._2.sum))
res: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[26] at map at <console>:30

scala> res.collect()
res12: Array[(String, Int)] = Array((two,2), (one,1), (three,3))

14.combineByKey[C](createCombiner: V => C, mergeValue:(C, V) => C, mergeCombiners:(C, C) => C)

聚合函数，对相同K，把V合并成一个集合

• createCombiner: V => C ，这个函数把当前的值作为参数，此时我们可以对其做些附加操作(类型转换)并把它返回
  (这一步类似于初始化操作)
• mergeValue: (C, V) => C，该函数把元素V合并到之前的元素C(createCombiner)上
  (这个操作在每个分区内进行)
• mergeCombiners: (C, C) => C，该函数把2个元素C合并 (这个操作在不同分区间进行)

示例：求平均数

scala> val scores = Array(("Fred", 88.0), ("Fred", 95.0), ("Fred", 91.0), ("Wilma", 93.0), ("Wilma", 95.0), ("Wilma", 98.0))
scores: Array[(String, Double)] = Array((Fred,88.0), (Fred,95.0), (Fred,91.0), (Wilma,93.0), (Wilma,95.0), (Wilma,98.0))

scala> val rdd = sc.parallelize(scores)
rdd: org.apache.spark.rdd.RDD[(String, Double)] = ParallelCollectionRDD[8] at parallelize at <console>:26

scala> type MVType = (Int, Double)
defined type alias MVType

scala> rdd.combineByKey(
     | score => (1, score),
     | (c1: MVType, newScore) => (c1._1 + 1, c1._2 + newScore),
     | (c1: MVType, c2: MVType) => (c1._1 + c2._1, c1._2 + c2._2)
     | ).map { case (name, (num, score)) => (name, score / num) }.collect
res4: Array[(String, Double)] = Array((Wilma,95.33333333333333), (Fred,91.33333333333333))

参数含义的解释：
a、score => (1, score)，我们把分数作为参数,并返回了附加的元组类型。 以"Fred"为列，当前其分数为88.0 =>(1,88.0) 1表示当前科目的计数器，此时只有一个科目

b、(c1: MVType, newScore) => (c1._1 + 1, c1._2 + newScore)，注意这里的c1就是createCombiner初始化得到的(1,88.0)。在一个分区内，我们又碰到了"Fred"的一个新的分数91.0。当然我们要把之前的科目分数和当前的分数加起来即c1._2 + newScore,然后把科目计算器加1即c1._1 + 1

c、 (c1: MVType, c2: MVType) => (c1._1 + c2._1, c1._2 + c2._2)，注意"Fred"可能是个学霸,他选修的科目可能过多而分散在不同的分区中。所有的分区都进行mergeValue后,接下来就是对分区间进行合并了,分区间科目数和科目数相加分数和分数相加就得到了总分和总科目数

15.aggregateByKey(zeroValue:U, [partitioner: Partitioner]) (seqOp: (U, V) => U,combOp: (U, U) => U)

在kv对的RDD中，按key将value进行分组合并，合并时，将每个value和初始值作为seq函数的参数，进行计算，返回的结果作为一个新的kv对，然后再将结果按照key进行合并，最后将每个分组的value传递给combine函数进行计算（先将前两个value进行计算，将返回结果和下一个value传给combine函数，以此类推），将key与计算结果作为一个新的kv对输出。

seqOp函数用于在每一个分区中用初始值逐步迭代value，combOp函数用于合并每个分区中的结果

例如：分一个分区，以key为1的分区为例，0先和3比较得3，3在和2比较得3，3在和4比较得4，所以整个key为1的组最终结果为（1，4），同理，key为2的最终结果为（2，3），key为3的为（3，8）如果分三个分区，前两个是一个分区，中间两个是一个分区，最后两个是一个分区，第一个分区的最终结果为（1，3），第二个分区为（1，4）（2，3），最后一个分区为（3，8），combine后为 (3,8), (1,7), (2,3)

scala> val rdd = sc.parallelize(List((1, 3), (1, 2), (1, 4), (2, 3), (3, 6), (3, 8)))
rdd: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[30] at parallelize at <console>:24

scala> val agg = rdd.aggregateByKey(0)(math.max(_, _), _+_)
agg: org.apache.spark.rdd.RDD[(Int, Int)] = ShuffledRDD[31] at aggregateByKey at <console>:26

scala> agg.collect()
res14: Array[(Int, Int)] = Array((1,4), (3,8), (2,3))

16.foldByKey(zeroValue: V)(func: (V, V) => V): RDD[(K, V)]

aggregateByKey的简化操作，seqop和combop相同

scala> val rdd = sc.parallelize(List((1, 3), (1, 2), (1, 4), (2, 3), (3, 6), (3, 8)), 3)
rdd: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[32] at parallelize at <console>:24

scala> val agg = rdd.foldByKey(0)(_+_)
agg: org.apache.spark.rdd.RDD[(Int, Int)] = ShuffledRDD[33] at foldByKey at <console>:26

scala> agg.collect()
res16: Array[(Int, Int)] = Array((3,14), (1,9), (2,3))

17.sortByKey([ascending], [numTasks])

在一个(K,V)的RDD上调用，K必须实现Ordered接口，返回一个按照key进行排序的(K,V)的RDD

scala> val rdd = sc.parallelize(Array((3, "a"), (6, "c"), (2, "b"), (1, "d")))
rdd: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[34] at parallelize at <console>:24

scala> rdd.sortByKey(true).collect()
res17: Array[(Int, String)] = Array((1,d), (2,b), (3,a), (6,c))

scala> rdd.sortByKey(false).collect()
res18: Array[(Int, String)] = Array((6,c), (3,a), (2,b), (1,d))

18.sortBy(func,[ascending], [numTasks])

与sortByKey类似，但是更灵活,可以用func先对数据进行处理，按照处理后的数据比较结果排序

scala> val rdd = sc.parallelize(List(1, 2, 3, 4))
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[37] at parallelize at <console>:24

scala> rdd.sortBy(x => x).collect()
res19: Array[Int] = Array(1, 2, 3, 4)

scala> rdd.sortBy(x => x%3).collect()
res20: Array[Int] = Array(3, 1, 4, 2)

scala> rdd.sortBy(x => x, false).collect()
res22: Array[Int] = Array(4, 3, 2, 1)

19.join(otherDataset, [numTasks])

在类型为(K,V)和(K,W)的RDD上调用，返回一个相同key对应的所有元素对在一起的(K,(V,W))的RDD

scala> val rdd1 = sc.parallelize(Array((1, "a"), (2, "b"), (3, "c")))
rdd1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[50] at parallelize at <console>:24

scala> val rdd2 = sc.parallelize(Array((1, 4), (2, 5), (3, 6)))
rdd2: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[51] at parallelize at <console>:24

scala> rdd1.join(rdd2).collect()
res23: Array[(Int, (String, Int))] = Array((1,(a,4)), (3,(c,6)), (2,(b,5)))

20.cogroup(otherDataset, [numTasks])

在类型为(K,V)和(K,W)的RDD上调用，返回一个(K,(Iterable,Iterable))类型的RDD

scala> val rdd1 = sc.parallelize(Array((1, "a"), (2, "b"), (3, "c")))
rdd1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[50] at parallelize at <console>:24

scala> val rdd2 = sc.parallelize(Array((1, 4), (2, 5), (3, 6)))
rdd2: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[51] at parallelize at <console>:24

scala> rdd1.cogroup(rdd2).collect()
res24: Array[(Int, (Iterable[String], Iterable[Int]))] = Array((1,(CompactBuffer(a),CompactBuffer(4))), (3,(CompactBuffer(c),CompactBuffer(6))), (2,(CompactBuffer(b),CompactBuffer(5))))

scala> val rdd3 = sc.parallelize(Array((1,"a"),(1,"d"),(2,"b"),(3,"c")))
rdd3: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[57] at parallelize at <console>:24

scala> rdd3.cogroup(rdd2).collect()
res26: Array[(Int, (Iterable[String], Iterable[Int]))] = Array((1,(CompactBuffer(a, d),CompactBuffer(4))), (3,(CompactBuffer(c),CompactBuffer(6))), (2,(CompactBuffer(b),CompactBuffer(5))))

21.cartesian(otherDataset)

笛卡尔积

scala> val rdd1 = sc.parallelize(1 to 3)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[62] at parallelize at <console>:24

scala> val rdd2 = sc.parallelize(2 to 5)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[63] at parallelize at <console>:24

scala> rdd1.cartesian(rdd2).collect()
res27: Array[(Int, Int)] = Array((1,2), (1,3), (1,4), (1,5), (2,2), (2,3), (2,4), (2,5), (3,2), (3,3), (3,4), (3,5))

22.pipe(command, [envVars])

对于每个分区，都执行一个perl或者shell脚本，返回输出的RDD
Spark 中，有种特殊的RDD，即pipeRDD，可以调用外部程序使其能够更快的计算

• 调用linux shell 命令

scala> val rdd = sc.parallelize(1 to 10, 5)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[65] at parallelize at <console>:24

scala> rdd.partitions.size
res28: Int = 5

scala> rdd.pipe("head -n 1").collect  # 提取每一个分区中的第一个元素构成新的RDD
res29: Array[String] = Array(1, 3, 5, 7, 9)

• 调用linux shell 脚本
  编写shell脚本

#!/bin/sh
echo "A"
while read LINE; do
   echo ${LINE}
done

注意：shell脚本需要集群中的所有节点都能访问到

scala> val data = sc.parallelize(List("tom", "jack", "harvey"), 3)
data: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[72] at parallelize at <console>:24

scala> val scriptPath = "/home/hadoop/shell/echo.sh"
scriptPath: String = /home/hadoop/shell/echo.sh

scala> val pipeRDD = data.pipe(scriptPath)
pipeRDD: org.apache.spark.rdd.RDD[String] = PipedRDD[73] at pipe at <console>:28

scala> pipeRDD.collect()
res33: Array[String] = Array(hello, tom!, hello, jack!, hello, harvey!)

scala> val data2 = sc.parallelize(List("tom", "jack", "harvey"), 2)
data2: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[86] at parallelize at <console>:24

scala> data2.pipe(scriptPath).collect()
res40: Array[String] = Array(hello, tom!, hello, jack!, harvey!)

23.coalesce(numPartitions)

缩减分区数，用于大数据集过滤后，提高小数据集的执行效率

scala> val rdd = sc.parallelize(1 to 16, 4)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[88] at parallelize at <console>:24

scala> rdd.partitions.size
res41: Int = 4

scala> val coalesceRDD = rdd.coalesce(3)
coalesceRDD: org.apache.spark.rdd.RDD[Int] = CoalescedRDD[89] at coalesce at <console>:26

scala> coalesceRDD.partitions.size
res42: Int = 3

24.repartition(numPartitions)

根据分区数，增加或减少此RDD中的并行度。在内部，这使用shuffle来重新分配数据。

scala> val rdd = sc.parallelize(1 to 16, 4)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[90] at parallelize at <console>:24

scala> rdd.partitions.size
res43: Int = 4

scala> val rerdd = rdd.repartition(2)
rerdd: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[94] at repartition at <console>:26

scala> rerdd.partitions.size
res44: Int = 2

scala> val rerdd = rdd.repartition(4)
rerdd: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[98] at repartition at <console>:26

scala> rerdd.partitions.size
res45: Int = 4

25.glom()

将每一个分区形成一个数组，形成新的RDD类型时RDD[Array[T]]

scala> val rdd = sc.parallelize(1 to 16, 4)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala> rdd.glom.collect()
res0: Array[Array[Int]] = Array(Array(1, 2, 3, 4), Array(5, 6, 7, 8), Array(9, 10, 11, 12), Array(13, 14, 15, 16))

26.mapValues()

针对于(K,V)形式的类型只对V进行操作

scala> val rdd = sc.parallelize(Array((1,"a"),(1,"d"),(2,"b"),(3,"c")))
rdd: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[2] at parallelize at <console>:24

scala> rdd.mapValues(_+"|||").collect()
res1: Array[(Int, String)] = Array((1,a|||), (1,d|||), (2,b|||), (3,c|||))

27.subtract()

计算差的一种函数去除两个RDD中相同的元素，不同的RDD将保留下来

scala> val rdd1 = sc.parallelize(3 to 8)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[4] at parallelize at <console>:24

scala> rdd1.collect
res3: Array[Int] = Array(3, 4, 5, 6, 7, 8)

scala> val rdd2 = sc.parallelize(1 to 5)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[5] at parallelize at <console>:24

scala> rdd2.collect
res4: Array[Int] = Array(1, 2, 3, 4, 5)

scala> rdd1.subtract(rdd2).collect()  # rdd1中取出rdd1和rdd2相同的元素，即3、4、5
res2: Array[Int] = Array(6, 7, 8)