深入Spark内核：任务调度(3)-TaskScheduler

本文链接：https://blog.csdn.net/qiuhaomaodan/article/details/44996631

本文深入探讨了Spark的TaskScheduler，它作为低级任务调度器，主要负责任务的发送和监控。TaskScheduler接收到DAGScheduler的TaskSet后，将任务分配到Worker的Executor中执行。根据不同运行模式，如Standalone、Yarn-Cluster等，TaskScheduler会选择相应的SchedulerBackend。以Standalone模式为例，TaskSchedulerImpl启动SparkDeploySchedulerBackend，后者负责Executor的部署和资源管理。TaskSchedulerImpl还实现了任务调度策略，包括FIFO和FAIR。TaskSchedulerImpl的start方法启动SchedulerBackend，创建DriverActor并启动ExecutorBackend。

摘要由CSDN通过智能技术生成

TaskScheduler相较于DAGScheduler来说是一个低级的任务调度器(low-level task scheduler)，更加准确的来说它更像是一个任务调度器接口。TaskScheduler的实现类只有一个TaskSchedulerImpl。TaskScheduler核心工作负责任务的发送和任务执行过程的监控，而一个TaskScheduler只为一个SparkContext实例服务，并接受DAGScheduler发送过来的TaskSet，TaskScheduler收到任务后负责把任务分发到集群中Worker的Excutor中去运行。因为DAGScheduler给TaskScheduler发送任务时是以Stage为单位来提交的，所以每个Stage内部都是TaskScheduler管理的一个单位就是TaskSet，TaskSet里面会有若干个任务(Task)，而任任务(Task)是以数据分片为单位划分的。如果某个任务(Task)运行失败，TaskScheduler就要负责重试；如果TaskScheduler发现某个任务(Task)一直未运行完或者运行过慢则有可能会在另一个机器开辟同一个任务(Task)，哪个任务(Task)先运行完就用哪个任务(Task)的结果。

/**
 * Low-level task scheduler interface, currently implemented exclusively by TaskSchedulerImpl.
 * This interface allows plugging in different task schedulers. Each TaskScheduler schedulers tasks
 * for a single SparkContext. These schedulers get sets of tasks submitted to them from the
 * DAGScheduler for each stage, and are responsible for sending the tasks to the cluster, running
 * them, retrying if there are failures, and mitigating stragglers. They return events to the
 * DAGScheduler.
 */
private[spark] trait TaskScheduler {

  def rootPool: Pool

  def schedulingMode: SchedulingMode

  def start(): Unit

  // Invoked after system has successfully initialized (typically in spark context).
  // Yarn uses this to bootstrap allocation of resources based on preferred locations,
  // wait for slave registerations, etc.
  def postStartHook() { }

  // Disconnect from the cluster.
  def stop(): Unit

  // Submit a sequence of tasks to run.
  def submitTasks(taskSet: TaskSet): Unit

  // Cancel a stage.
  def cancelTasks(stageId: Int, interruptThread: Boolean)

  // Set the DAG scheduler for upcalls. This is guaranteed to be set before submitTasks is called.
  def setDAGScheduler(dagScheduler: DAGScheduler): Unit

  // Get the default level of parallelism to use in the cluster, as a hint for sizing jobs.
  def defaultParallelism(): Int

  /**
   * Update metrics for in-progress tasks and let the master know that the BlockManager is still
   * alive. Return true if the driver knows about the given block manager. Otherwise, return false,
   * indicating that the block manager should re-register.
   */
  def executorHeartbeatReceived(execId: String, taskMetrics: Array[(Long, TaskMetrics)],
    blockManagerId: BlockManagerId): Boolean
}

在SparkContext中TaskScheduler实例对象创建的源码如下：

  // Create and start the scheduler
  private[spark] var taskScheduler = SparkContext.createTaskScheduler(this, master)

接下来让我们进入createTaskScheduler方法，通过模式匹配可以看出创建任务时存在多种运行模式，不同的运行模式都有与之相对应的SchedulerBackend：

Local模式：TaskSchedulerImpl + LocalBackend

　　Standalone模式：TaskSchedulerImpl + SparkDepolySchedulerBackend

　　Yarn-Cluster模式：YarnClusterScheduler + CoarseGrainedSchedulerBackend

　　Yarn-Client模式：YarnClientClusterScheduler + YarnClientSchedulerBackend

Mesos模式：TaskSchedulerImpl + CoarseMesosSchedulerBackend

 /** Creates a task scheduler based on a given master URL. Extracted for testing. */
  private def createTaskScheduler(sc: SparkContext, master: String): TaskScheduler = {
    // Regular expression used for local[N] and local[*] master formats
    val LOCAL_N_REGEX = """local\[([0-9]+|\*)\]""".r
    // Regular expression for local[N, maxRetries], used in tests with failing tasks
    val LOCAL_N_FAILURES_REGEX = """local\[([0-9]+|\*)\s*,\s*([0-9]+)\]""".r
    // Regular expression for simulating a Spark cluster of [N, cores, memory] locally
    val LOCAL_CLUSTER_REGEX = """local-cluster\[\s*([0-9]+)\s*,\s*([0-9]+)\s*,\s*([0-9]+)\s*]""".r
    // Regular express