RDD算子总结——转换型算子

最新推荐文章于 2022-02-11 20:49:35 发布

辜智强 -buaa

最新推荐文章于 2022-02-11 20:49:35 发布

阅读量757

点赞数

本文链接：https://blog.csdn.net/m0_37719047/article/details/89843144

版权

本文详细介绍了Spark RDD中的转换型算子，包括map、flatMap、distinct、coalesce、repartition、randomSplit、glom、union、intersection、subtract、mapPartitions、mapPartitionsWithIndex、zip、zipPartitions、mapValues、flatMapValues、partitionBy、combineByKey、foldByKey、groupByKey、reduceByKey和cogroup等，涵盖了这些算子的功能、参数和使用场景。

摘要由CSDN通过智能技术生成

map（func）：
将一个RDD中的每个数据项，通过map中的函数映射变为一个新的元素。输入分区与输出分区一对一，即：有多少个输入分区，就有多少个输出分区
flatMap(func):
功能和map大致相同，但是将所有的输出分区合并在一起
note： flatMap只会将String扁平化成字符数组，并不会把Array[String]也扁平化成字符数组。

scala> var  data=sc.textFile("/test/datas")
data: org.apache.spark.rdd.RDD[String] = /test/datas MapPartitionsRDD[7] at textFile at <console>:24

scala> data.flatMap(x=>x.split(" ")).collect
res9: Array[String] = Array(hello, word, hello, spark, hello, hadoop)

scala> data.map(x=>x.split(" ")).collect
res10: Array[Array[String]] = Array(Array(hello, word), Array(hello, spark), Array(hello, hadoop))
.
.
.
.
.
scala> data.map(_.toUpperCase).collect
res32: Array[String] = Array(HELLO WORLD, HELLO SPARK, HELLO HIVE, HI SPARK)
scala> data.flatMap(_.toUpperCase).collect
res33: Array[Char] = Array(H, E, L, L, O,  , W, O, R, L, D, H, E, L, L, O,  , S, P, A, R, K, H, E, L, L, O,  , H, I, V, E, H, I,  , S, P, A, R, K)

distinct:
对Rdd中的元素进行去重操作

scala> var  data=sc.textFile("/test/datas")
data: org.apache.spark.rdd.RDD[String] = /test/datas MapPartitionsRDD[1] at textFile at <console>:24

scala> data.collect
res2: Array[String] = Array("hello word ", hello spark, hello hadoop)

scala> data.distinct.collect
res3: Array[String] = Array("hello word ", hello spark, hello hadoop)

scala> data.flatMap(_.split(" ")).distinct.collect
res4: Array[String] = Array(word, hello, spark, hadoop)

parallelize

coalesce：

def coalesce(numPartitions: Int, shuffle: Boolean = false)(implicit ord: Ordering[T] = null): RDD[T]

该函数用于将RDD进行重分区，使用HashPartitioner。

第一个参数为重分区的数目，第二个为是否进行shuffle，默认为false;
如果指定分区数目大于原来分区数目，则第二个参数应该设置为true,否则参数不变

scala> var data = sc.textFile("/tmp/lxw1234/1.txt")
data: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[53] at textFile at :21
 
scala> data.collect
res37: Array[String] = Array(hello world, hello spark, hello hive, hi spark)
 
scala> data.partitions.size
res38: Int = 2  //RDD data默认有两个分区
 
scala> var rdd1 = data.coalesce(1)
rdd1: org.apache.spark.rdd.RDD[String] = CoalescedRDD[2] at coalesce at :23
 
scala> rdd1.partitions.size
res1: Int = 1   //rdd1的分区数为1
 
 
scala> var rdd1 = data.coalesce(4)
rdd1: org.apache.spark.rdd.RDD[String] = CoalescedRDD[3] at coalesce at :23
 
scala> rdd1.partitions.size
res2: Int = 2   //如果重分区的数目大于原来的分区数，那么必须指定shuffle参数为true，//否则，分区数不便
 
scala> var rdd1 = data.coalesce(4,true)
rdd1: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[7] at coalesce at :23
 
scala> rdd1.partitions.size
res3: Int = 4

repartition

def repartition(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T]

该函数其实就是coalesce函数第二个参数为true的实现

    scala> var rdd2 = data.repartition(1)
    rdd2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[11] at repartition at :23
     
    scala> rdd2.partitions.size
    res4: Int = 1
     
    scala> var rdd2 = data.repartition(4)
    rdd2: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[15] at repartition at :23
     
    scala> rdd2.partitions.size
    res5: Int = 4

randomSplit

def randomSplit(weights: Array[Double], seed: Long = Utils.random.nextLong): Array[RDD[T]]

该函数根据weights权重，将一个RDD切分成多个RDD。

该权重参数为一个Double数组

第二个参数为random的种子，基本可忽略。

scala> var rdd = sc.makeRDD(1 to 10,10)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[16] at makeRDD at :21
 
scala> rdd.collect
res6: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)  
 
scala> var splitRDD = rdd.randomSplit(Array(1.0,2.0,3.0,4.0))
splitRDD: Array[org.apache.spark.rdd.RDD[Int]] = Array(MapPartitionsRDD[17] at randomSplit at :23, 
MapPartitionsRDD[18] at randomSplit at :23, 
MapPartitionsRDD[19] at randomSplit at :23, 
MapPartitionsRDD[20] at randomSplit at :23)
 
//这里注意：randomSplit的结果是一个RDD数组
scala> splitRDD.size
res8: Int = 4
//由于randomSplit的第一个参数weights中传入的值有4个，因此，就会切分成4个RDD,
//把原来的rdd按照权重1.0,2.0,3.0,4.0，随机划分到这4个RDD中，权重高的RDD，划分到//的几率就大一些。
//注意，权重的总和加起来为1，否则会不正常
 
scala> splitRDD(0).collect
res10: Array[Int] = Array(1, 4)
 
scala> splitRDD(1).collect
res11: Array[Int] = Array(3)                                                    
 
scala> splitRDD(2).collect
res12: Array[Int] = Array(5, 9)
 
scala> splitRDD(3).collect
res13: Array[Int] = Array(2, 6, 7, 8, 10)
 glom

def glom(): RDD[Array[T]]

该函数是将RDD中每一个分区中类型为T的元素转换成Array[T]，这样每一个分区就只有一个数组元素。

scala> var rdd = sc.makeRDD(1 to 10,3)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[38] at makeRDD at :21
scala> rdd.partitions.size
res33: Int = 3  //该RDD有3个分区
scala> rdd.glom().collect
res35: Array[Array[Int]] = Array(Array(1, 2, 3), Array(4, 5, 6), Array(7, 8, 9, 10))
//glom将每个分区中的元素放到一个数组中，这样，结果就变成了3个数组

union

def union(other: RDD[T]): RDD[T]

该函数比较简单，就是将两个RDD进行合并，不去重。

scala> var rdd1 = sc.makeRDD(1 to 2,1)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[45] at makeRDD at :21
 
scala> rdd1.collect
res42: Array[Int] = Array(1, 2)
 
scala> var rdd2 = sc.makeRDD(2 to 3,1)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[46] at makeRDD at :21
 
scala> rdd2.collect
res43: Array[Int] = Array(2, 3)
 
scala> rdd1.union(rdd2).collect
res44: Array[Int] = Array(1, 2, 2, 3
union

union
def union(other: RDD[T]): RDD[T]

该函数比较简单，就是将两个RDD进行合并，不去重。

scala> var rdd1 = sc.makeRDD(1 to 2,1)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[45] at makeRDD at :21
 
scala> rdd1.collect
res42: Array[Int] = Array(1, 2)
 
scala> var rdd2 = sc.makeRDD(2 to 3,1)
rdd2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[46] at makeRDD at :21
 
scala> rdd2.collect
res43: Array[Int] = Array(2, 3)
 
sca