ResultTask
概述
ResultTask 执行当前分区的计算,首先从ShuffleMapTask拿到当前partition的数据,会从所有的ShuffleMapTask都拿一遍当前的partition数据。最后合并所有的ResultTask输出结果,返回给driver application。
reducer task有几个特点:
- reducer的数量等于partition的数量. 每个reducer task只处理一个partition的数据量.
- 如果shuffleMapTask的数量为N,则每个reducer task的网络请求数为N. 因为每个reducer task需要从所有的ShuffleMapTask都拿一遍当前的partition数据
- reducer task的计算逻辑(对数据进行什么样的操作)在shuffledRDD中,因此运行reducer task首先就是反序列化RDD,调用shuffledRDD的compute方法.
从本质思维上讲, reducer task的本质是:向所有的shuffleMapTask所在的节点请求当前partition的数据,并进行数据处理
从流程拆解思维上看, 我们可以拆解为几个点: 所有的shuffleMapTask、当前的partition
接下来的问题会围绕几个问题展开:
如何证明他只请求当前partition的数据,而不是请求多个连续partition的数据
如何证明他是向所有的shuffleMapTask所在的节点请求数据,而不是向部分节点请求数据?
成员变量
private[spark] class ResultTask[T, U](
stageId: Int,
stageAttemptId: Int,
taskBinary: Broadcast[Array[Byte]],
partition: Partition,
locs: Seq[TaskLocation],
val outputId: Int,
localProperties: Properties,
serializedTaskMetrics: Array[Byte],
jobId: Option[Int] = None,
appId: Option[String] = None,
appAttemptId: Option[String] = None,
isBarrier: Boolean = false)
extends Task[U](stageId, stageAttemptId, partition.index, localProperties, serializedTaskMetrics,
jobId, appId, appAttemptId, isBarrier)
runTask方法
override def runTask(context: TaskContext): U = {
// Deserialize the RDD and the func using the broadcast variables.
val threadMXBean = ManagementFactory.getThreadMXBean
val deserializeStartTime = System.currentTimeMillis()
val deserializeStartCpuTime = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
threadMXBean.getCurrentThreadCpuTime
} else 0L
val ser = SparkEnv.get.closureSerializer.newInstance()
//反序列化ResultTask,结果为rdd,和func函数
//taskBinary的值是在DAGScheduler.submitMissingTasks()方法中进行序列化的
val (rdd, func) = ser.deserialize[(RDD[T], (TaskContext, Iterator[T]) => U)](
ByteBuffer.wrap(taskBinary.value), Thread.currentThread.getContextClassLoader)
_executorDeserializeTime = System.currentTimeMillis() - deserializeStartTime
_executorDeserializeCpuTime = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
threadMXBean.getCurrentThreadCpuTime - deserializeStartCpuTime
} else 0L
//整个ResultTask计算在 rdd.iterator(partition, context) 中完成
//此时的RDD为:ShuffleRDD,所以rdd.iterator()方法调用的是ShuffleRDD.iterator()方法,会调用ShuffleRDD.compute()方法
//func 函数将Iterator转换为数组: RDD.collect()方法中的 (iter: Iterator[T]) => iter.toArray
func(context, rdd.iterator(partition, context))
}
ShuffleRDD.compute()方法
调用 BlockStoreShuffleReader.read()方法去各个节点拉取ShuffleMapTask输出的当前partition的磁盘文件。
override def compute(split: Partition, context: TaskContext): Iterator[(K, C)] = {
val dep = dependencies.head.asInstanceOf[ShuffleDependency[K, V, C]]
SparkEnv.get.shuffleManager.getReader(dep.shuffleHandle, split.index, split.index + 1, context)
.read()
.asInstanceOf[Iterator[(K, C)]]
}
BlockStoreShuffleReader的构造函数为(BaseShuffleHandle shuffleHandle, int startPartition, int endPartition, TaskContext context). 用于从ShuffleMapTask的输出文件中,读取指定的[startPartition, endPartition)范围内的数据.
在 ShuffleRDD.compute()方法传入的[startPartition, endPartition)为[当前partition的index, 当前partition的index + 1). 说明每个reducer task只读取并处理一个partition的数据.
private[spark] class BlockStoreShuffleReader[K, C](
handle: BaseShuffleHandle[K, _, C], //shuffleHandle
startPartition: Int, //指定的startPartition
endPartition: Int, //指定的endPartition
context: TaskContext,
serializerManager: SerializerManager = SparkEnv.get.serializerManager,
blockManager: BlockManager = SparkEnv.get.blockManager,
mapOutputTracker: MapOutputTracker = SparkEnv.get.mapOutputTracker)
spark shuffle过程中典型的fetch shuffle file堆栈如下:
org.apache.spark.shuffle.IndexShuffleBlockResolver.getBlockData(IndexShuffleBlockResolver.scala:191)
at org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:298)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.fetchLocalBlocks(ShuffleBlockFetcherIterator.scala:238)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.initialize(ShuffleBlockFetcherIterator.scala:269)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.<init>(ShuffleBlockFetcherIterator.scala:112)
at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:43)
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:98)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)