Spark Task原理与源码分析

最新推荐文章于 2024-06-27 19:23:04 发布

发布了一场Chat

最新推荐文章于 2024-06-27 19:23:04 发布

阅读量791

点赞数

分类专栏： spark深入学习文章标签： spark task原理 spark task源码解析

本文链接：https://blog.csdn.net/u013174239/article/details/80403883

版权

本文深入探讨了Spark Task的工作原理，通过分析Executor.scala、Task.scala等关键源码文件，揭示了ShuffleMapTask和ResultTask的执行流程，以及RDD和MapPartitionRDD在任务调度中的作用。此外，还介绍了CoarseGrainedExecutorBackend和CoarseGrainedSchedulerBackend在任务分配和管理中的核心功能。

摘要由CSDN通过智能技术生成

① task的原理示意图

②task源码分析

Executor.scala

  /**
    * 这里就是task运行的工作原理
    */
  class TaskRunner(
      execBackend: ExecutorBackend,
      val taskId: Long,
      val attemptNumber: Int,
      taskName: String,
      serializedTask: ByteBuffer)
    extends Runnable {
          ...
    override def run(): Unit = {
      ...

      try {
        // 对task数据，反序列化
        val (taskFiles, taskJars, taskProps, taskBytes) =
          Task.deserializeWithDependencies(serializedTask)

        // Must be set before updateDependencies() is called, in case fetching dependencies
        // requires access to properties contained within (e.g. for access control).
        Executor.taskDeserializationProps.set(taskProps)

        // 将依赖的文件资源、jar拷贝到到task读取文件的对应目录
        updateDependencies(taskFiles, taskJars)

        // 反序列化task的数据集
        task = ser.deserialize[Task[Any]](taskBytes, Thread.currentThread.getContextClassLoader)
        task.localProperties = taskProps
        task.setTaskMemoryManager(taskMemoryManager)

        // If this task has been killed before we deserialized it, let's quit now. Otherwise,
        // continue executing the task.
        if (killed) {
          // Throw an exception rather than returning, because returning within a try{} block
          // causes a NonLocalReturnControl exception to be thrown. The NonLocalReturnControl
          // exception will be caught by the catch block, leading to an incorrect ExceptionFailure
          // for the task.
          throw new TaskKilledException
        }

        logDebug("Task " + taskId + "'s epoch is " + task.epoch)
        env.mapOutputTracker.updateEpoch(task.epoch)

        // Run the actual task and measure its runtime.
        // task执行开始时间
        taskStart = System.currentTimeMillis()
        taskStartCpu = if (threadMXBean.isCurrentThreadCpuTimeSupported) {
          threadMXBean.getCurrentThreadCpuTime
        } else 0L
        var threwException = true

        val value = try {

          // 这里的res就是MapStatus
          // 如果后面执行的还是一个ShuffleMapTask，就会联系MaoOutputTracker
          // 获取上一个ShuffleMapTask的输出结果。 ResultTask也是一样的。
          val res = task.run(
            taskAttemptId = taskId,
            attemptNumber = attemptNumber,
            metricsSystem = env.metricsSystem)
          threwException = false
          res
        } finally {
          val releasedLocks = env.blockManager.releaseAllLocksForTask(taskId)
          val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()

          if (freedMemory > 0 && !threwException) {
            val errMsg = s"Managed memory leak detected; size = $freedMemory bytes, TID = $taskId"
            if (conf.getBoolean("spark.unsafe.exceptionOnMemoryLeak", false)) {
              throw new SparkException(errMsg)
            } else {
              logWarning(errMsg)
            }
          }

          if (releasedLocks.nonEmpty && !threwException) {
            val errMsg =
              s"${releasedLocks.size} block locks were not released by TID = $taskId:\n" +
                releasedLocks.mkString("[", ", ", "]")
            if (conf.getBoolean("spark.storage.exceptionOnPinLeak", false)) {
              throw new SparkException(errMsg)
            } else {
              logWarning(errMsg)
            }