1.1 map(func)
返回一个新的RDD，该RDD由每一个输入元素经过func函数转换后组成
scala>valrdd=sc.parallelize(1to10)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[123] atparallelizeat<console>:27
scala>rdd.collect
res8: Array[Int] =Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
scala>rdd.map(_*2)
res9: org.apache.spark.rdd.RDD[Int] =MapPartitionsRDD[124] atmapat<console>:29
scala>rdd.map(_*2).collect
res10: Array[Int] =Array(2, 4, 6, 8, 10, 12, 14, 16, 18, 20)
1.2 mapPartitions(func)
类似于map，但独立地在RDD的每一个分片上运行，因此在类型为T的RDD上运行时，func的函数类型必须是Iterator[T] =>Iterator[U]。假设有N个元素，有M个分区，那么map的函数的将被调用N次,而mapPartitions被调用M次,一个函数一次处理所有分区
scala>valpartitionsFun= (iter : Iterator[(String,String)]) =>{
varwoman=List[String]()
while (iter.hasNext){
valnext=iter.next()
nextmatch {
case (_,"female") =>woman=next._1 :: woman
case_=>
}
}
woman.iterator
}
partitionsFun: Iterator[(String, String)] =>Iterator[String] =<function1>
scala>rdd.mapPartitions(partitionsFun)
res11: org.apache.spark.rdd.RDD[String] =MapPartitionsRDD[127] atmapPartitionsat<console>:31
scala>rdd.mapPartitions(partitionsFun).collect
res12: Array[String] =Array(kpop, lucy)
1.3 glom
将每一个分区形成一个数组，形成新的RDD类型时RDD[Array[T]]
scala>valrdd=sc.parallelize(1to16,4)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[129] atparallelizeat<console>:27
scala>rdd.glom().collect()
res13: Array[Array[Int]] =Array(Array(1, 2, 3, 4), Array(5, 6, 7, 8), Array(9, 10, 11, 12), Array(13, 14, 15, 16))
1.4 flatMap(func)
map后再扁平化
类似于map，但是每一个输入元素可以被映射为0或多个输出元素（所以func应该返回一个序列，而不是单一元素）
scala>valrdd=sc.parallelize(1to5)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[133] atparallelizeat<console>:27
scala>rdd.collect
res14: Array[Int] =Array(1, 2, 3, 4, 5)
scala>rdd.flatMap(1to_)
res15: org.apache.spark.rdd.RDD[Int] =MapPartitionsRDD[134] atflatMapat<console>:29
scala>rdd.flatMap(1to_).collect
res16: Array[Int] =Array(1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5)
1.5 filter(func)
返回一个新的RDD，该RDD由经过func函数计算后返回值为true的输入元素组成
scala>valrdd=sc.parallelize(Array("xiaoming","xiaojiang","xiaohe","dazhi"))
rdd: org.apache.spark.rdd.RDD[String] =ParallelCollectionRDD[136] atparallelizeat<console>:27
scala>rdd.filter(_.contains("xiao"))
res17: org.apache.spark.rdd.RDD[String] =MapPartitionsRDD[137] atfilterat<console>:29
scala>rdd.filter(_.contains("xiao")).collect
res18: Array[String] =Array(xiaoming, xiaojiang, xiaohe)
1.6 mapPartitionsWithIndex(func)
类似于mapPartitions，但func带有一个整数参数表示分片的索引值，因此在类型为T的RDD上运行时，func的函数类型必须是(Int, Interator[T]) => Iterator[U]
scala>valrdd=sc.parallelize(List(("kpop","female"),("zorro","male"),("mobin","male"),("lucy","female")))
rdd: org.apache.spark.rdd.RDD[(String, String)] =ParallelCollectionRDD[139] atparallelizeat<console>:27
valfunc= (index: Int, iter: Iterator[(String,String)]) => {
iter.map(x=>"[partID:"+index+", val: "+x+"]")
}
scala>rdd.mapPartitionsWithIndex(func)
res20: org.apache.spark.rdd.RDD[String] =MapPartitionsRDD[140] atmapPartitionsWithIndexat<console>:31
scala>rdd.mapPartitionsWithIndex(func).collect
res21: Array[String] =Array([partID:0, val: (kpop,female)], [partID:1, val: (zorro,male)], [partID:2, val: (mobin,male)], [partID:3, val: (lucy,female)])
1.7 sample(withReplacement, fraction, seed)
以指定的随机种子随机抽样出数量为fraction的数据，withReplacement表示是抽出的数据是否放回，true为有放回的抽样，false为无放回的抽样，seed用于指定随机数生成器种子。例子从RDD中随机且有放回的抽出50%的数据，随机种子值为3（即可能以1 2 3的其中一个起始值）
scala>valrdd=sc.parallelize(1to10)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[142] atparallelizeat<console>:27
scala>rdd.collect()
res22: Array[Int] =Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
scala>varsample1=rdd.sample(true,0.4,2)
sample1: org.apache.spark.rdd.RDD[Int] =PartitionwiseSampledRDD[143] atsampleat<console>:28
scala>sample1.collect()
res23: Array[Int] =Array(1, 2, 2, 7, 7, 8, 9)
scala>varsample2=rdd.sample(false,0.2,3)
sample2: org.apache.spark.rdd.RDD[Int] =PartitionwiseSampledRDD[144] atsampleat<console>:28
scala>sample2.collect()
res24: Array[Int] =Array(1, 9)
1.8 distinct([numTasks]))
对源RDD进行去重后返回一个新的RDD. 默认情况下，只有8个并行任务来操作，但是可以传入一个可选的numTasks参数改变它。
scala>valrdd=sc.parallelize(List(1,2,1,5,2,9,6,1))
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[145] atparallelizeat<console>:27
scala>rdd.distinct().collect
res25: Array[Int] =Array(1, 9, 5, 6, 2)
scala>rdd.distinct(2).collect
res26: Array[Int] =Array(6, 2, 1, 9, 5)
1.9 partitionBy
对RDD进行分区操作，如果原有的partionRDD和现有的partionRDD是一致的话就不进行分区， 否则会生成ShuffleRDD。(针对pairRDD)
scala>valrdd=sc.parallelize(Array((1,"aaa"),(2,"bbb"),(3,"ccc"),(4,"ddd")),4)
rdd: org.apache.spark.rdd.RDD[(Int, String)] =ParallelCollectionRDD[152] atparallelizeat<console>:27
scala>rdd.partitions.size
res27: Int=4
scala>varrdd2=rdd.partitionBy(neworg.apache.spark.HashPartitioner(2))
rdd2: org.apache.spark.rdd.RDD[(Int, String)] =ShuffledRDD[153] atpartitionByat<console>:28
scala>rdd2.partitions.size
res28: Int=2
1.10 coalesce(numPartitions)
与repartition的区别: repartition(numPartitions:Int):RDD[T]和coalesce(numPartitions:Int，shuffle:Boolean=false):RDD[T] repartition只是coalesce接口中shuffle为true的实现.
缩减分区数，用于大数据集过滤后，提高小数据集的执行效率。
scala>valrdd=sc.parallelize(1to16,4)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[154] atparallelizeat<console>:27
scala>rdd.partitions.size
res29: Int=4
scala>valcoalesceRDD=rdd.coalesce(3)
coalesceRDD: org.apache.spark.rdd.RDD[Int] =CoalescedRDD[155] atcoalesceat<console>:28
scala>coalesceRDD.partitions.size
res30: Int=3
1.11 repartition(numPartitions)
根据分区数，从新通过网络随机洗牌所有数据。
scala>valrdd=sc.parallelize(1to16,4)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[156] atparallelizeat<console>:27
scala>rdd.partitions.size
res31: Int=4
scala>valrerdd=rdd.repartition(2)
rerdd: org.apache.spark.rdd.RDD[Int] =MapPartitionsRDD[160] atrepartitionat<console>:28
scala>rerdd.partitions.size
res32: Int=2
scala>valrerdd=rdd.repartition(4)
rerdd: org.apache.spark.rdd.RDD[Int] =MapPartitionsRDD[164] atrepartitionat<console>:28
scala>rerdd.partitions.size
res33: Int=4
1.12 repartitionAndSortWithinPartitions(partitioner)
repartitionAndSortWithinPartitions函数是repartition函数的变种，与repartition函数不同的是，repartitionAndSortWithinPartitions在给定的partitioner内部进行排序，性能比repartition要高。
scala>valrdd=sc.parallelize(1to16,4)
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[165] atparallelizeat<console>:27
scala>valnewrdd=rdd.map((_, 1)).repartitionAndSortWithinPartitions(neworg.apache.spark.HashPartitioner(2))
newrdd: org.apache.spark.rdd.RDD[(Int, Int)] =ShuffledRDD[167] atrepartitionAndSortWithinPartitionsat<console>:28
scala>newrdd.partitions.size
res34: Int=2
scala>newrdd.collect
res35: Array[(Int, Int)] =Array((2,1), (4,1), (6,1), (8,1), (10,1), (12,1), (14,1), (16,1), (1,1), (3,1), (5,1), (7,1), (9,1), (11,1), (13,1), (15,1))
scala>valfunc= (index: Int, iter: Iterator[(Int, Int)]) => {
iter.map(x=>"[partID:"+index+", val: "+x+"]")
}
func: (Int, Iterator[(Int, Int)]) =>Iterator[String] =<function2>
scala>newrdd.mapPartitionsWithIndex(func).collect
res36: Array[String] =Array([partID:0, val: (2,1)], [partID:0, val: (4,1)], [partID:0, val: (6,1)], [partID:0, val: (8,1)], [partID:0, val: (10,1)], [partID:0, val: (12,1)], [partID:0, val: (14,1)], [partID:0, val: (16,1)], [partID:1, val: (1,1)], [partID:1, val: (3,1)], [partID:1, val: (5,1)], [partID:1, val: (7,1)], [partID:1, val: (9,1)], [partID:1, val: (11,1)], [partID:1, val: (13,1)], [partID:1, val: (15,1)])
1.13 sortBy(func,[ascending], [numTasks])
用func先对数据进行处理，按照处理后的数据比较结果排序。
scala>valrdd=sc.parallelize(List(1,2,3,4))
rdd: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[169] atparallelizeat<console>:27
scala>rdd.sortBy(x=>x).collect()
res37: Array[Int] =Array(1, 2, 3, 4)
scala>rdd.sortBy(x=>x%3).collect()
res38: Array[Int] =Array(3, 1, 4, 2)
1.14 union(otherDataset)
对源RDD和参数RDD求并集后返回一个新的RDD 不去重
scala>valrdd1=sc.parallelize(1to5)
rdd1: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[180] atparallelizeat<console>:27
scala>valrdd2=sc.parallelize(5to10)
rdd2: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[181] atparallelizeat<console>:27
scala>valrdd3=rdd1.union(rdd2)
rdd3: org.apache.spark.rdd.RDD[Int] =UnionRDD[182] atunionat<console>:30
scala>rdd3.collect()
res39: Array[Int] =Array(1, 2, 3, 4, 5, 5, 6, 7, 8, 9, 10)
1.15 subtract (otherDataset)
计算rdd与rdd2的差集(即rdd1中不包含与rdd2相同的元素部分)
scala>valrdd1=sc.parallelize(3to8)
rdd1: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[195] atparallelizeat<console>:27
scala>rdd1.collect
res42: Array[Int] =Array(3, 4, 5, 6, 7, 8)
scala>valrdd2=sc.parallelize(1to5)
rdd2: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[196] atparallelizeat<console>:27
scala>rdd2.collect
res43: Array[Int] =Array(1, 2, 3, 4, 5)
scala>rdd1.subtract(rdd2).collect()
res44: Array[Int] =Array(8, 6, 7)
1.16 intersection(otherDataset)
对源RDD和参数RDD求交集后返回一个新的RDD
scala>valrdd1=sc.parallelize(1to7)
rdd1: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[201] atparallelizeat<console>:27
scala>rdd1.collect
res45: Array[Int] =Array(1, 2, 3, 4, 5, 6, 7)
scala>valrdd2=sc.parallelize(5to10)
rdd2: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[202] atparallelizeat<console>:27
scala>rdd2.collect
res46: Array[Int] =Array(5, 6, 7, 8, 9, 10)
scala>valrdd3=rdd1.intersection(rdd2)
rdd3: org.apache.spark.rdd.RDD[Int] =MapPartitionsRDD[208] atintersectionat<console>:30
scala>rdd3.collect()
res47: Array[Int] =Array(5, 6, 7)
1.17 cartesian(otherDataset)
笛卡尔积
scala>valrdd1=sc.parallelize(1to3)
rdd1: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[209] atparallelizeat<console>:27
scala>rdd1.collect
res48: Array[Int] =Array(1, 2, 3)
scala>valrdd2=sc.parallelize(2to5)
rdd2: org.apache.spark.rdd.RDD[Int] =ParallelCollectionRDD[210] atparallelizeat<console>:27
scala>rdd2.collect
res49: Array[Int] =Array(2, 3, 4, 5)
scala>rdd1.cartesian(rdd2).collect()
res50: Array[(Int, Int)] =Array((1,2), (1,3), (1,4), (1,5), (2,2), (2,3), (2,4), (2,5), (3,2), (3,3), (3,4), (3,5))
1.18 pipe(command, [envVars])
管道，对于每个分区，都执行一个perl或者shell脚本，返回输出的RDD
Shell脚本
#!/bin/sh
echo"AA"
whilereadLINE; do
echo">>>"${LINE}
done
scala>valrdd=sc.parallelize(List("hi","Hello","how","are","you"),1)
rdd: org.apache.spark.rdd.RDD[String] =ParallelCollectionRDD[50] atparallelizeat<console>:24
scala>rdd.pipe("/home/bigdata/pipe.sh").collect()
res18: Array[String] =Array(AA, >>>hi, >>>Hello, >>>how, >>>are, >>>you)
scala>valrdd=sc.parallelize(List("hi","Hello","how","are","you"),2)
rdd: org.apache.spark.rdd.RDD[String] =ParallelCollectionRDD[52] atparallelizeat<console>:24
scala>rdd.pipe("/home/bigdata/pipe.sh").collect()
res19: Array[String] =Array(AA, >>>hi, >>>Hello, AA, >>>how, >>>are, >>>you)
pipe.sh:
#!/bin/sh
echo"AA"
whilereadLINE; do
echo">>>"${LINE}
done
1.19 join(otherDataset, [numTasks])
在类型为(K,V)和(K,W)的RDD上调用，返回一个相同key对应的所有元素对在一起的(K,(V,W))的RDD
scala>valrdd1=sc.parallelize(Array((1,"a"),(2,"b"),(3,"c")))
rdd1: org.apache.spark.rdd.RDD[(Int, String)] =ParallelCollectionRDD[212] atparallelizeat<console>:27
scala>valrdd2=sc.parallelize(Array((1,4),(2,5),(3,6)))
rdd2: org.apache.spark.rdd.RDD[(Int, Int)] =ParallelCollectionRDD[213] atparallelizeat<console>:27
scala>rdd1.join(rdd2).collect
res51: Array[(Int, (String, Int))] =Array((1,(a,4)), (2,(b,5)), (3,(c,6)))
1.20 cogroup(otherDataset, [numTasks])
在类型为(K,V)和(K,W)的RDD上调用，返回一个(K,(Iterable<V>,Iterable<W>))类型的RDD
scala>valrdd1=sc.parallelize(Array((1,"a"),(2,"b"),(3,"c")))
rdd1: org.apache.spark.rdd.RDD[(Int, String)] =ParallelCollectionRDD[217] atparallelizeat<console>:27
scala>valrdd2=sc.parallelize(Array((1,4),(2,5),(3,6)))
rdd2: org.apache.spark.rdd.RDD[(Int, Int)] =ParallelCollectionRDD[218] atparallelizeat<console>:27
scala>rdd1.cogroup(rdd2).collect
res52: Array[(Int, (Iterable[String], Iterable[Int]))] =Array((1,(CompactBuffer(a),CompactBuffer(4))), (2,(CompactBuffer(b),CompactBuffer(5))), (3,(CompactBuffer(c),CompactBuffer(6))))
scala>valrdd3=sc.parallelize(Array((4,4),(2,5),(3,6)))
rdd3: org.apache.spark.rdd.RDD[(Int, Int)] =ParallelCollectionRDD[221] atparallelizeat<console>:27
scala>rdd1.cogroup(rdd3).collect
res54: Array[(Int, (Iterable[String], Iterable[Int]))] =Array((4,(CompactBuffer(),CompactBuffer(4))), (1,(CompactBuffer(a),CompactBuffer())), (2,(CompactBuffer(b),CompactBuffer(5))), (3,(CompactBuffer(c),CompactBuffer(6))))
scala>valrdd4=sc.parallelize(Array((1,"a"),(1,"d"),(2,"b"),(3,"c")))
rdd4: org.apache.spark.rdd.RDD[(Int, String)] =ParallelCollectionRDD[226] atparallelizeat<console>:27
scala>rdd4.cogroup(rdd3).collect
res55: Array[(Int, (Iterable[String], Iterable[Int]))] =Array((4,(CompactBuffer(),CompactBuffer(4))), (1,(CompactBuffer(a, d),CompactBuffer())), (2,(CompactBuffer(b),CompactBuffer(5))), (3,(CompactBuffer(c),CompactBuffer(6))))
1.21 reduceByKey(func, [numTasks])
在一个(K,V)的RDD上调用，返回一个(K,V)的RDD，使用指定的reduce函数，将相同key的值聚合到一起，reduce任务的个数可以通过第二个可选的参数来设置。
scala>valrdd=sc.parallelize(List(("female",1),("male",5),("female",5),("male",2)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] =ParallelCollectionRDD[229] atparallelizeat<console>:27
scala>valreduce=rdd.reduceByKey((x,y) =>x+y)
reduce: org.apache.spark.rdd.RDD[(String, Int)] =ShuffledRDD[230] atreduceByKeyat<console>:28
scala>reduce.collect()
res56: Array[(String, Int)] =Array((female,6), (male,7))
1.22 groupByKey
groupByKey也是对每个key进行操作，但只生成一个sequence。
scala>valwords=Array("one", "two", "two", "three", "three", "three")
words: Array[String] =Array(one, two, two, three, three, three)
scala>valwordPairsRDD=sc.parallelize(words).map(word=> (word, 1))
wordPairsRDD: org.apache.spark.rdd.RDD[(String, Int)] =MapPartitionsRDD[232] atmapat<console>:29
scala>valgroup=wordPairsRDD.groupByKey()
group: org.apache.spark.rdd.RDD[(String, Iterable[Int])] =ShuffledRDD[233] atgroupByKeyat<console>:28
scala>group.collect()
res57: Array[(String, Iterable[Int])] =Array((two,CompactBuffer(1, 1)), (one,CompactBuffer(1)), (three,CompactBuffer(1, 1, 1)))
scala>group.map(t=> (t._1, t._2.sum))
res58: org.apache.spark.rdd.RDD[(String, Int)] =MapPartitionsRDD[234] atmapat<console>:29
scala>group.map(t=> (t._1, t._2.sum)).collect
res59: Array[(String, Int)] =Array((two,2), (one,1), (three,3))
1.23 combineByKey[C]
( createCombiner: V => C, mergeValue: (C, V) => C, mergeCombiners: (C, C) => C)
对相同K，把V合并成一个集合。
createCombiner: combineByKey() 会遍历分区中的所有元素，因此每个元素的键要么还没有遇到过，要么就 和之前的某个元素的键相同。如果这是一个新的元素,combineByKey() 会使用一个叫作 createCombiner() 的函数来创建 那个键对应的累加器的初始值
mergeValue: 如果这是一个在处理当前分区之前已经遇到的键， 它会使用 mergeValue() 方法将该键的累加器对应的当前值与这个新的值进行合并
mergeCombiners: 由于每个分区都是独立处理的， 因此对于同一个键可以有多个累加器。如果有两个或者更多的分区都有对应同一个键的累加器， 就需要使用用户提供的 mergeCombiners() 方法将各个分区的结果进行合并。
scala>valscores=Array(("Fred", 88), ("Fred", 95), ("Fred", 91), ("Wilma", 93), ("Wilma", 95), ("Wilma", 98))
scores: Array[(String, Int)] =Array((Fred,88), (Fred,95), (Fred,91), (Wilma,93), (Wilma,95), (Wilma,98))
scala>valinput=sc.parallelize(scores)
input: org.apache.spark.rdd.RDD[(String, Int)] =ParallelCollectionRDD[236] atparallelizeat<console>:29
scala>valcombine=input.combineByKey(
| (v)=>(v,1),
| (acc:(Int,Int),v)=>(acc._1+v,acc._2+1),
| (acc1:(Int,Int),acc2:(Int,Int))=>(acc1._1+acc2._1,acc1._2+acc2._2))
combine: org.apache.spark.rdd.RDD[(String, (Int, Int))] =ShuffledRDD[237] atcombineByKeyat<console>:28
scala>valresult=combine.map{
| case (key,value) => (key,value._1/value._2.toDouble)}
result: org.apache.spark.rdd.RDD[(String, Double)] =MapPartitionsRDD[238] atmapat<console>:28
scala>result.collect()
res60: Array[(String, Double)] =Array((Wilma,95.33333333333333), (Fred,91.33333333333333))
1.24 aggregateByKey
(zeroValue:U,[partitioner: Partitioner]) (seqOp: (U, V) => U,combOp: (U, U) => U)
在kv对的RDD中，，按key将value进行分组合并，合并时，将每个value和初始值作为seq函数的参数，进行计算，返回的结果作为一个新的kv对，然后再将结果按照key进行合并，最后将每个分组的value传递给combine函数进行计算（先将前两个value进行计算，将返回结果和下一个value传给combine函数，以此类推），将key与计算结果作为一个新的kv对输出。
seqOp函数用于在每一个分区中用初始值逐步迭代value，combOp函数用于合并每个分区中的结果。
scala>valrdd=sc.parallelize(List((1,3),(1,2),(1,4),(2,3),(3,6),(3,8)),3)
rdd: org.apache.spark.rdd.RDD[(Int, Int)] =ParallelCollectionRDD[239] atparallelizeat<console>:27
scala>valagg=rdd.aggregateByKey(0)(math.max(_,_),_+_)
agg: org.apache.spark.rdd.RDD[(Int, Int)] =ShuffledRDD[240] ataggregateByKeyat<console>:28
scala>agg.collect()
res61: Array[(Int, Int)] =Array((3,8), (1,7), (2,3))
scala>agg.partitions.size
res62: Int=3
scala>valrdd=sc.parallelize(List((1,3),(1,2),(1,4),(2,3),(3,6),(3,8)),1)
rdd: org.apache.spark.rdd.RDD[(Int, Int)] =ParallelCollectionRDD[241] atparallelizeat<console>:27
scala>valagg=rdd.aggregateByKey(0)(math.max(_,_),_+_).collect()
agg: Array[(Int, Int)] =Array((1,4), (3,8), (2,3))
1.25 foldByKey
(zeroValue: V)(func: (V, V) => V): RDD[(K, V)]
aggregateByKey的简化操作，seqop和combop相同
scala>valrdd=sc.parallelize(List((1,3),(1,2),(1,4),(2,3),(3,6),(3,8)),3)
rdd: org.apache.spark.rdd.RDD[(Int, Int)] =ParallelCollectionRDD[0] atparallelizeat<console>:24
scala>valagg=rdd.foldByKey(0)(_+_)
agg: org.apache.spark.rdd.RDD[(Int, Int)] =ShuffledRDD[1] atfoldByKeyat<console>:25
scala>agg.collect()
res1: Array[(Int, Int)] =Array((3,14), (1,9), (2,3))
1.26 sortByKey([ascending], [numTasks])
在一个(K,V)的RDD上调用，K必须实现Ordered接口，返回一个按照key进行排序的(K,V)的RDD
scala>valrdd=sc.parallelize(Array((3,"aa"),(6,"cc"),(2,"bb"),(1,"dd")))
rdd: org.apache.spark.rdd.RDD[(Int, String)] =ParallelCollectionRDD[2] atparallelizeat<console>:24
scala>rdd.sortByKey(true).collect()
res2: Array[(Int, String)] =Array((1,dd), (2,bb), (3,aa), (6,cc))
scala>rdd.sortByKey(false).collect()
res3: Array[(Int, String)] =Array((6,cc), (3,aa), (2,bb), (1,dd))
1.27 mapValues
针对于(K,V)形式的类型只对V进行操作
scala>valrdd=sc.parallelize(Array((1,"a"),(1,"d"),(2,"b"),(3,"c")))
rdd: org.apache.spark.rdd.RDD[(Int, String)] =ParallelCollectionRDD[11] atparallelizeat<console>:24
scala>rdd.mapValues(_+"|||").collect()
res4: Array[(Int, String)] =Array((1,a|||), (1,d|||), (2,b|||), (3,c|||))
464
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
