spark2.3源码分析之ResultTask读取并处理shuffle file的流程

ResultTask

概述

ResultTask 执行当前分区的计算，首先从ShuffleMapTask拿到当前partition的数据，会从所有的ShuffleMapTask都拿一遍当前的partition数据。最后合并所有的ResultTask输出结果，返回给driver application。

reducer task有几个特点:

1. reducer的数量等于partition的数量. 每个reducer task只处理一个partition的数据量.
2. 如果shuffleMapTask的数量为N,则每个reducer task的网络请求数为N. 因为每个reducer task需要从所有的ShuffleMapTask都拿一遍当前的partition数据
3. reducer task的计算逻辑(对数据进行什么样的操作)在shuffledRDD中,因此运行reducer task首先就是反序列化RDD,调用shuffledRDD的compute方法.

从本质思维上讲, reducer task的本质是:向所有的shuffleMapTask所在的节点请求当前partition的数据,并进行数据处理

从流程拆解思维上看, 我们可以拆解为几个点: 所有的shuffleMapTask、当前的partition

接下来的问题会围绕几个问题展开:

如何证明他只请求当前partition的数据,而不是请求多个连续partition的数据

如何证明他是向所有的shuffleMapTask所在的节点请求数据,而不是向部分节点请求数据?

成员变量

private[spark] class ResultTask[T, U](
    stageId: Int,
    stageAttemptId: Int,
    taskBinary: Broadcast[Array[Byte]],
    partition: Partition,
    locs: Seq[TaskLocation],
    val outputId: Int,
    localProperties: Properties,
    serializedTaskMetrics: Array[Byte],
    jobId: Option[Int] = None,
    appId: Option[String] = None,
    appAttemptId: Option[String] = None,
    isBarrier: Boolean = false)
  extends Task[U](stageId, stageAttemptId, partition.index, localProperties, serializedTaskMetrics,
    jobId, appId, appAttemptId, isBarrier)

runTask方法

  override def runTask(context: TaskContext): U = {
    // Deserialize the RDD and the func using the broadcast variables.
    val threadMXBean = ManagementFactory.getThreadMXBean
    val deserializeStartTime = System.currentTimeMillis()
    val deserializeStartCpuTime = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
      threadMXBean.getCurrentThreadCpuTime
    } else 0L
    val ser = SparkEnv.get.closureSerializer.newInstance()
//反序列化ResultTask，结果为rdd,和func函数
//taskBinary的值是在DAGScheduler.submitMissingTasks()方法中进行序列化的
    val (rdd, func) = ser.deserialize[(RDD[T], (TaskContext, Iterator[T]) => U)](
      ByteBuffer.wrap(taskBinary.value), Thread.currentThread.getContextClassLoader)
    _executorDeserializeTime = System.currentTimeMillis() - deserializeStartTime
    _executorDeserializeCpuTime = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
      threadMXBean.getCurrentThreadCpuTime - deserializeStartCpuTime
    } else 0L
//整个ResultTask计算在 rdd.iterator(partition, context) 中完成
//此时的RDD为:ShuffleRDD,所以rdd.iterator()方法调用的是ShuffleRDD.iterator()方法，会调用ShuffleRDD.compute()方法
//func 函数将Iterator转换为数组: RDD.collect()方法中的 (iter: Iterator[T]) => iter.toArray
    func(context, rdd.iterator(partition, context))
  }

 ShuffleRDD.compute()方法

调用 BlockStoreShuffleReader.read()方法去各个节点拉取ShuffleMapTask输出的当前partition的磁盘文件。

override def compute(split: Partition, context: TaskContext): Iterator[(K, C)] = {
    val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, C]]
    SparkEnv.get.shuffleManager.getReader(dep.shuffleHandle, split.index, split.index + 1, context)
      .read()
      .asInstanceOf[Iterator[(K, C)]]
  }

BlockStoreShuffleReader的构造函数为(BaseShuffleHandle shuffleHandle, int startPartition, int endPartition, TaskContext context). 用于从ShuffleMapTask的输出文件中,读取指定的[startPartition, endPartition)范围内的数据.

在 ShuffleRDD.compute()方法传入的[startPartition, endPartition)为[当前partition的index, 当前partition的index + 1). 说明每个reducer task只读取并处理一个partition的数据.

private[spark] class BlockStoreShuffleReader[K, C](
    handle: BaseShuffleHandle[K, _, C],  //shuffleHandle
    startPartition: Int,  //指定的startPartition
    endPartition: Int, //指定的endPartition
    context: TaskContext, 
    serializerManager: SerializerManager = SparkEnv.get.serializerManager,
    blockManager: BlockManager = SparkEnv.get.blockManager, 
    mapOutputTracker: MapOutputTracker = SparkEnv.get.mapOutputTracker)

 

spark shuffle过程中典型的fetch shuffle file堆栈如下：

org.apache.spark.shuffle.IndexShuffleBlockResolver.getBlockData(IndexShuffleBlockResolver.scala:191)
	at org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:298)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks(ShuffleBlockFetcherIterator.scala:238)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:269)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:112)
	at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:43)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:98)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)

参考：Spark源码分析之ResultTask处理