TaskSchedulerImpl
概述
不同类型的集群对应于不同的SchedulerBackend:YarnSchedulerBackend、StandaloneSchedulerBackend、LocalSchedulerBackend等。TaskSchedulerImpl为不同的SchedulerBackend处理相同的逻辑,例如决定任务之间的调度顺序等。
client端必须先调用TaskSchedulerImpl的initialize()和start()方法,才能再调用TaskSchedulerImpl的submitTasks()方法提交TaskSet。
TaskSchedulerImpl#submitTasks()方法
提交一个TaskSet(stage)内的所有task去运行。
流程如下:
1、首先创建TaskSetManager。TaskSetManager负责管理TaskSchedulerImpl中一个单独TaskSet,跟踪每一个task,如果task失败,负责重试task直到达到task重试次数的最多次数。并且通过延迟调度来执行task的位置感知调度。
2、将TaskSetManger加入schedulableBuilder。schedulableBuilder的类型是 SchedulerBuilder,SchedulerBuilder是一个trait,有两个实现FIFOSchedulerBuilder和 FairSchedulerBuilder,并且默认采用的是FIFO方式。schedulableBuilder是TaskScheduler中一个重要成员,他根据调度策略决定了TaskSetManager的调度顺序。
3、调用SchedulerBackend的riviveOffers方法对Task进行调度,决定task具体运行在哪个Executor中。
override def submitTasks(taskSet: TaskSet) {
//获取TaskSet中Task数组
val tasks = taskSet.tasks
logInfo("Adding task set " + taskSet.id + " with " + tasks.length + " tasks")
this.synchronized {
//创建TaskSetManager,TaskSetManager将会管理一个TaskSet内所有Task的生命周期
val manager = createTaskSetManager(taskSet, maxTaskFailures)
//获取TaskSet的stageId
val stage = taskSet.stageId
val stageTaskSets =
taskSetsByStageIdAndAttempt.getOrElseUpdate(stage, new HashMap[Int, TaskSetManager])
// Mark all the existing TaskSetManagers of this stage as zombie, as we are adding a new one.
// This is necessary to handle a corner case. Let's say a stage has 10 partitions and has 2
// TaskSetManagers: TSM1(zombie) and TSM2(active). TSM1 has a running task for partition 10
// and it completes. TSM2 finishes tasks for partition 1-9, and thinks he is still active
// because partition 10 is not completed yet. However, DAGScheduler gets task completion
// events for all the 10 partitions and thinks the stage is finished. If it's a shuffle stage
// and somehow it has missing map outputs, then DAGScheduler will resubmit it and create a
// TSM3 for it. As a stage can't have more than one active task set managers, we must mark
// TSM2 as zombie (it actually is).
stageTaskSets.foreach { case (_, ts) =>
ts.isZombie = true
}
stageTaskSets(taskSet.stageAttemptId) = manager
//将TaskSetManger加入schedulableBuilder
schedulableBuilder.addTaskSetManager(manager, manager.taskSet.properties)
if (!isLocal && !hasReceivedTask) {
starvationTimer.scheduleAtFixedRate(new TimerTask() {
override def run() {
if (!hasLaunchedTask) {
logWarning("Initial job has not accepted any resources; " +
"check your cluster UI to ensure that workers are registered " +
"and have sufficient resources")
} else {
this.cancel()
}
}
}, STARVATION_TIMEOUT_MS, STARVATION_TIMEOUT_MS)
}
hasReceivedTask = true
}
//schedulerBackEnd#reviveOffers()方法
backend.reviveOffers()
}
运行spark(on yarn)任务时典型的日志如下:
[dag-scheduler-event-loop] INFO cluster.YarnScheduler:58: Adding task set 0.0 with 6 tasks
[dispatcher-event-loop-11] INFO scheduler.TaskSetManager:58: Starting task 0.0 in stage 0.0 (TID 0, hadoop5, partition 0,PROCESS_LOCAL, 9549 bytes)
[dispatcher-event-loop-11] INFO scheduler.TaskSetManager:58: Starting task 1.0 in stage 0.0 (TID 1, hadoop4, partition 1,PROCESS_LOCAL, 5974 bytes)
[dispatcher-event-loop-11] INFO scheduler.TaskSetManager:58: Starting task 2.0 in stage 0.0 (TID 2, hadoop9, partition 2,PROCESS_LOCAL, 4826 bytes)
[dispatcher-event-loop-11] INFO scheduler.TaskSetManager:58: Starting task 3.0 in stage 0.0 (TID 3, hadoop5, partition 3,PROCESS_LOCAL, 6937 bytes)
[dispatcher-event-loop-11] INFO scheduler.TaskSetManager:58: Starting task 4.0 in stage 0.0 (TID 4, hadoop4, partition 4,PROCESS_LOCAL, 5587 bytes)
[dispatcher-event-loop-11] INFO scheduler.TaskSetManager:58: Starting task 5.0 in stage 0.0 (TID 5, hadoop9, partition 5,PROCESS_LOCAL, 6734 bytes)
CoarseGrainedExecutorBackEnd#reviveOffers()方法
该方法给driverEndpoint发送ReviveOffer消息
override def reviveOffers() {
driverEndpoint.send(ReviveOffers)
}
driverEndpoint收到ReviveOffer消息后调用makeOffers方法
case ReviveOffers =>
makeOffers()
DriverEndpoint#makeOffers()方法
上面代码中的executorDataMap,在客户端向Master注册Application的时候,Master已经为Application分配并启动好Executor,然后注册给CoarseGrainedSchedulerBackend,注册信息就是存储在executorDataMap数据结构中。
// Make fake resource offers on all executors
private def makeOffers() {
// Make sure no executor is killed while some task is launching on it
val taskDescs = withLock {
// Filter out executors under killing
//过滤掉处于被killing状态的Executor
val activeExecutors = executorDataMap.filterKeys(executorIsAlive)
val workOffers = activeExecutors.map {
case (id, executorData) =>
将Executor封装成 WorkerOffer 对象
new WorkerOffer(id, executorData.executorHost, executorData.freeCores,
Some(executorData.executorAddress.hostPort))
}.toIndexedSeq
scheduler.resourceOffers(workOffers)
}
if (!taskDescs.isEmpty) {
launchTasks(taskDescs)
}
}