RDD操作详解（二）键值转换

最新推荐文章于 2022-07-02 18:31:45 发布

Running_Tiger

最新推荐文章于 2022-07-02 18:31:45 发布

阅读量1.6k

点赞数 1

分类专栏： spark 文章标签： spark scala 函数

本文链接：https://blog.csdn.net/qq_41455420/article/details/79462801

版权

本文详细介绍了Spark中的键值转换操作，包括partitionBy、mapValues、flatMapValues、combineByKey、foldByKey、reduceByKeyLocally、cogroup与groupByKey的区别、join、leftOuterJoin和rightOuterJoin。通过这些操作，可以对RDD进行有效的数据处理和组合，适用于键值对数据的聚合和关联。

摘要由CSDN通过智能技术生成

RDD操作详解（二）键值转换

启动spark-shell

spark-shell --master spark://node-1:7077

1. partitionBy

def partitionBy(partitioner: Partitioner): RDD[(K, V)]
该函数根据partitioner函数生成新的ShuffleRDD，将原RDD重新分区。

scala> var rdd1 = sc.makeRDD(Array((1,"A"),(2,"B"),(3,"C"),(4,"D")),2)
rdd1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[23] at makeRDD at :21
scala> rdd1.partitions.size
res20: Int = 2

//查看rdd1中每个分区的元素
scala> rdd1.mapPartitionsWithIndex{
     |         (partIdx,iter) => {
     |           var part_map = scala.collection.mutable.Map[String,List[(Int,String)]]()
     |             while(iter.hasNext){
     |               var part_name = "part_" + partIdx;
     |               var elem = iter.next()
     |               if(part_map.contains(part_name)) {
     |                 var elems = part_map(part_name)
     |                 elems ::= elem
     |                 part_map(part_name) = elems
     |               } else {
     |                 part_map(part_name) = List[(Int,String)]{elem}
     |               }
     |             }
     |             part_map.iterator
     |            
     |         }
     |       }.collect
res22: Array[(String, List[(Int, String)])] = Array((part_0,List((2,B), (1,A))), (part_1,List((4,D), (3,C))))
//(2,B),(1,A)在part_0中，(4,D),(3,C)在part_1中

//使用partitionBy重分区
scala> var rdd2 = rdd1.partitionBy(new org.apache.spark.HashPartitioner(2))
rdd2: org.apache.spark.rdd.RDD[(Int, String)] = ShuffledRDD[25] at partitionBy at :23

scala> rdd2.partitions.size
res23: Int = 2

//查看rdd2中每个分区的元素
scala> rdd2.mapPartitionsWithIndex{
     |         (partIdx,iter) => {
     |           var part_map = scala.collection.mutable.Map[String,List[(Int,String)]]()
     |             while(iter.hasNext){
     |               var part_name = "part_" + partIdx;
     |               var elem = iter.next()
     |               if(part_map.contains(part_name)) {
     |                 var elems = part_map(part_name)
     |                 elems ::= elem
     |                 part_map(part_name) = elems
     |               } else {
     |                 part_map(part_name) = List[(Int,String)]{elem}
     |               }
     |             }
     |             part_map.iterator
     |         }