Spark common join vs map join

最新推荐文章于 2024-06-12 00:46:20 发布

潇洒-人生

最新推荐文章于 2024-06-12 00:46:20 发布

阅读量599

点赞数

分类专栏：大数据 spark spark

本文链接：https://blog.csdn.net/qq_35744460/article/details/90207935

版权

大数据同时被 3 个专栏收录

45 篇文章 0 订阅

订阅专栏

spark

17 篇文章 0 订阅

订阅专栏

spark

12 篇文章 0 订阅

订阅专栏

Spark common join vs map join

common join

demo


scala>  val rdd1=sc.parallelize(List((1,"a"),(2,"b"),(1,"c"),(2,"d"),(3,"e")))
rdd1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala>     val data=sc.parallelize(List((1,"AA"),(2,"BB")))
data: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[1] at parallelize at <console>:24

scala> rdd1.join(data).foreach(println(_))
(1,(a,AA))
(1,(c,AA))
(2,(b,BB))
(2,(d,BB))

ui
普通join会产生shuffle
common join

map join

Spark 使用广播变量小表进行广播

scala>  val rdd1=sc.parallelize(List((1,"a"),(2,"b"),(1,"c"),(2,"d"),(3,"e")))
rdd1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[0] at parallelize at <console>:24

scala>     val data=sc.parallelize(List((1,"AA"),(2,"BB")))
data: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[1] at parallelize at <console>:24

scala> val broadcastdata=sc.broadcast(data.collectAsMap())
broadcastdata: org.apache.spark.broadcast.Broadcast[scala.collection.Map[Int,String]] = Broadcast(1)

scala> rdd1.mapPartitions(x=>{
    |       val mapdata=broadcastdata.value
    |       for((key,value) <- x if(mapdata.contains(key)))
    |         yield (key,mapdata.get(key),value)
    |     }).foreach(println(_))
(1,Some(AA),a)
(2,Some(BB),b)
(1,Some(AA),c)
(2,Some(BB),d)

ui
map join 没有产生shuffle