Spark ShuffleDependency Shuffle依赖关系

最新推荐文章于 2022-06-19 11:19:10 发布

thinktothings

最新推荐文章于 2022-06-19 11:19:10 发布

阅读量544

点赞数 1

分类专栏： Spark 文章标签： Spark ShuffleDependency Shuffle依赖关系

本文链接：https://blog.csdn.net/thinktothings/article/details/84831954

版权

Spark 专栏收录该内容

28 篇文章 1 订阅

订阅专栏

Spark ShuffleDependency Shuffle依赖关系

Represents a dependency on the output of a shuffle stage. Note that in the case of shuffle,the RDD is transient since we don’t need it on the executor side.

youtub视频演示

https://youtu.be/8T6PyHuf_wQ (youtube视频)
https://www.bilibili.com/video/av37442139/?p=5 (bilibili视频)
github: https://github.com/opensourceteams/spark-scala-maven

width="800" height="500" src="//player.bilibili.com/player.html?aid=37442139&cid=65822917&page=5" scrolling="no" border="0" allowfullscreen="true">

输入数据

List(('c',1),('b',1),('a',1),('a',1)

处理程序scala


package com.opensource.bigdata.spark.local.rdd.operation.dependency.shuffle.n_01_ShuffleDependency

import com.opensource.bigdata.spark.local.rdd.operation.base.BaseScalaSparkContext

object Run  extends BaseScalaSparkContext{

  def main(args: Array[String]): Unit = {

    val sc = pre()
    val rdd1 = sc.parallelize(List(('c',1),('b',1),('a',1),('a',1)),2)
    val rdd2 =rdd1.reduceByKey((a,b) => a + b)



    println("rdd2\n" + rdd2.collect().mkString("\n"))

    sc.stop()
  }

}