spark work ResultTask 源码分析

spark work task 源码分析

rdd在各个 CoarseGrainedExecutorBackend 进程中，经典的业务执行过程

private[spark] class ResultTask[T, U](
stageId: Int,
stageAttemptId: Int,
taskBinary: Broadcast[Array[Byte]],
partition: Partition,
locs: Seq[TaskLocation],
val outputId: Int,
internalAccumulators: Seq[Accumulator[Long]])
  extends Task[U](stageId, stageAttemptId, partition.index, internalAccumulators)
  with Serializable {

  @transient private[this] val preferredLocs: Seq[TaskLocation] = {
if (locs == null) Nil else locs.toSet.toSeq
  }

  override def runTask(context: TaskContext): U = {
// Deserialize the RDD and the func using the broadcast variables.
val deserializeStartTime = System.currentTimeMillis()
val ser = SparkEnv.get.closureSerializer.newInstance()
// 数据的反序列化
val (rdd, func) = ser.deserialize[(RDD[T], (TaskContext, Iterator[T]) => U)](
  ByteBuffer.wrap(taskBinary.value), Thread.currentThread.getContextClassLoader)
_executorDeserializeTime = System.currentTimeMillis() - deserializeStartTime
// 一个经典的rdd对一个分区的数据的执行过程，fun传业务执行函数，partition执行的数据分区，rdd就是那个对分片数据执行拆分的过程
metrics = Some(context.taskMetrics)
func(context, rdd.iterator(partition, context))
  }

  // This is only callable on the driver side.
  override def preferredLocations: Seq[TaskLocation] = preferredLocs

  override def toString: String = "ResultTask(" + stageId + ", " + partitionId + ")"
}

从上面的代码可以看到，从driver端序列化 fun和rdd等数据到各个work node进程中，然后单独执行这么一个业务fun函数，然后把结果返回，
这样就实现了spark rdd的分布式数据分片的计算过程