基于spark1.3.1的源码解读
不得不佩服spark源码的精炼,standalone模式中,不到100行就搞定了资源调度,spark提供两种资源调度方式,尽量打散:即我们的executor会尽量的分配更多的worker上;尽量集中:即我们的executor会尽量的分配更少的worker上;这其中是通过spreadOutApps变量来控制的,true为尽量分散。
private def schedule() {
#当master不是alive时,直接reuturn
#也就是说Standby是不参与资源调度的
if (state != RecoveryState.ALIVE) { return }
//Random.shuffle作用就是把集合随机打乱
//取出workers中所有之前注册的worker,进行过滤,必须 状态 是Alive的worker
//把worker随机的打乱
val shuffledAliveWorkers = Random.shuffle(workers.toSeq.filter(
_.state == WorkerState.ALIVE))
val numWorkersAlive = shuffledAliveWorkers.size。
只有在模式是yarn-cluster提交后,才会注册driver,因为standalone与yarn-client
都会在本地启动dirver,而不会来注册driver,就更不可能被master来调度
所以说下面的这个for只会运行在yarn-cluster模式下提交下
for (driver <- waitingDrivers.toList) { // iterate over a copy of waitingDrivers
// We assign workers to each waiting driver in a round-robin fashion. For each driver, we
// start from the last worker that was assigned a driver, and continue onwards until we have
// explored all alive workers.
var launched = false
var numWorkersVisited = 0
/**while中的条件,当还有活着的worker没有被遍历到,就