深入Spark内核：任务调度(3)-TaskScheduler

       TaskScheduler相较于DAGScheduler来说是一个低级的任务调度器(low-level task scheduler)，更加准确的来说它更像是一个任务调度器接口。TaskScheduler的实现类只有一个TaskSchedulerImpl。TaskScheduler核心工作负责任务的发送和任务执行过程的监控，而一个TaskScheduler只为一个SparkContext实例服务，并接受DAGScheduler发送过来的TaskSet，TaskScheduler收到任务后负责把任务分发到集群中Worker的Excutor中去运行。因为DAGScheduler给TaskScheduler发送任务时是以Stage为单位来提交的，所以每个Stage内部都是TaskScheduler管理的一个单位就是TaskSet，TaskSet里面会有若干个任务(Task)，而任任务(Task)是以数据分片为单位划分的。如果某个任务(Task)运行失败，TaskScheduler就要负责重试；如果TaskScheduler发现某个任务(Task)一直未运行完或者运行过慢则有可能会在另一个机器开辟同一个任务(Task)，哪个任务(Task)先运行完就用哪个任务(Task)的结果。

/**
 * Low-level task scheduler interface, currently implemented exclusively by TaskSchedulerImpl.
 * This interface allows plugging in different task schedulers. Each TaskScheduler schedulers tasks
 * for a single SparkContext. These schedulers get sets of tasks submitted to them from the
 * DAGScheduler for each stage, and are responsible for sending the tasks to the cluster, running
 * them, retrying if there are failures, and mitigating stragglers. They return events to the
 * DAGScheduler.
 */
private[spark] trait TaskScheduler {

  def rootPool: Pool

  def schedulingMode: SchedulingMode

  def start(): Unit

  // Invoked after system has successfully initialized (typically in spark context).
  // Yarn uses this to bootstrap allocation of resources based on preferred locations,
  // wait for slave registerations, etc.
  def postStartHook() { }

  // Disconnect from the cluster.
  def stop(): Unit

  // Submit a sequence of tasks to run.
  def submitTasks(taskSet: TaskSet): Unit

  // Cancel a stage.
  def cancelTasks(stageId: Int, interruptThread: Boolean)

  // Set the DAG scheduler for upcalls. This is guaranteed to be set before submitTasks is called.
  def setDAGScheduler(dagScheduler: DAGScheduler): Unit

  // Get the default level of parallelism to use in the cluster, as a hint for sizing jobs.
  def defaultParallelism(): Int

  /**
   * Update metrics for in-progress tasks and let the master know that the BlockManager is still
   * alive. Return true if the driver knows about the given block manager. Otherwise, return false,
   * indicating that the block manager should re-register.
   */
  def executorHeartbeatReceived(execId: String, taskMetrics: Array[(Long, TaskMetrics)],
    blockManagerId: BlockManagerId): Boolean
}

           在SparkContext中TaskScheduler实例对象创建的源码如下：

  // Create and start the scheduler
  private[spark] var taskScheduler = SparkContext.createTaskScheduler(this, master)

       接下来让我们进入createTaskScheduler方法，通过模式匹配可以看出创建任务时存在多种运行模式，不同的运行模式都有与之相对应的SchedulerBackend：

       Local模式：TaskSchedulerImpl + LocalBackend

　　Standalone模式：TaskSchedulerImpl + SparkDepolySchedulerBackend

　　Yarn-Cluster模式：YarnClusterScheduler + CoarseGrainedSchedulerBackend

　　Yarn-Client模式：YarnClientClusterScheduler + YarnClientSchedulerBackend

       Mesos模式：TaskSchedulerImpl + CoarseMesosSchedulerBackend

 /** Creates a task scheduler based on a given master URL. Extracted for testing. */
  private def createTaskScheduler(sc: SparkContext, master: String): TaskScheduler = {
    // Regular expression used for local[N] and local[*] master formats
    val LOCAL_N_REGEX = """local\[([0-9]+|\*)\]""".r
    // Regular expression for local[N, maxRetries], used in tests with failing tasks
    val LOCAL_N_FAILURES_REGEX = """local\[([0-9]+|\*)\s*,\s*([0-9]+)\]""".r
    // Regular expression for simulating a Spark cluster of [N, cores, memory] locally
    val LOCAL_CLUSTER_REGEX = """local-cluster\[\s*([0-9]+)\s*,\s*([0-9]+)\s*,\s*([0-9]+)\s*]""".r
    // Regular express