spark调度系列----1. spark stanalone模式下Master对worker上各个executor资源的分配

spark Application的注册时在Master类中对RegisterApplication消息的处理。

Master类里面对

RegisterApplication

消息处理时， 首先将Application信息注册到Master，然后调用了Master.schedule方法

相关代码如下：

case  RegisterApplication(description , driver) => {
  //  TODO Prevent repeated registrations from some driver
  if ( state == RecoveryState. STANDBY) {
    // ignore, don't send response
  }  else {
    logInfo( "Registering app " + description.name)
    val app = createApplication(description , driver)
    registerApplication(app)//注册application
    logInfo( "Registered app " + description.name +  " with ID " + app.id)
    persistenceEngine.addApplication(app)
    driver.send( RegisteredApplication(app.id , self))
    schedule()//启动driver和executor
  }
}

private def  schedule():  Unit = {
  if ( state != RecoveryState. ALIVE) {  return }
  val conf =  new Configuration()
  val fs = FileSystem. get(URI. create( "hdfs://ddos12:9000/log/workshuff.log") , conf)
  val hdfsInStream = fs.open( new Path( "hdfs://ddos12:9000/log/workshuff.log"))
  // Drivers take strict precedence over executors
  val shuffledWorkers = Random.shuffle( workers)  // Randomization helps balance drivers
  for (worker <- shuffledWorkers  if worker. state == WorkerState. ALIVE) {
    for (driver <-  waitingDrivers) {
      if (worker.memoryFree >= driver.desc.mem && worker.coresFree >= driver.desc.cores) {
        launchDriver(worker , driver)//先启动Driver
        waitingDrivers -= driver
      }
    }
  }
  startExecutorsOnWorkers()//选择executor在那些节点上运行，然后分配资源，启动executor
}

private def startExecutorsOnWorkers(): Unit = {
    // Right now this is a very simple FIFO scheduler. We keep trying to fit in the first app
    // in the queue, then the second app, etc.
    if (spreadOutApps) {
      // Try to spread out each app among all the workers, until it has all its cores
      for (app <- waitingApps if app.coresLeft > 0) {
        val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)//活着的节点
          .filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB && //节点的剩余内存大于executor内存
            worker.coresFree >= app.desc.coresPerExecutor.getOrElse(1))//节点的空闲core个数大于一个executor需要的core个数,如果没有设置spark.executor.cores，则默认1个Executor有1个core
          .sortBy(_.coresFree).reverse
        val numUsable = usableWorkers.length
        val assigned = new Array[Int](numUsable) // Number of cores to give on each node
        var toAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
        var pos = 0
        while (toAssign > 0) {
          if (usableWorkers(pos).coresFree - assigned(pos) > 0) {//这个worker还有空闲core，则为app在这个worker分配一个core，轮询尽可能多的worker
            toAssign -= 1
            assigned(pos) += 1
          }
          pos = (pos + 1) % numUsable//选择下一个worker节点
        }
        // Now that we've decided how many cores to give on each node, let's actually give them
        for (pos <- 0 until numUsable if assigned(pos) > 0) {
          allocateWorkerResourceToExecutors(app, assigned(pos), usableWorkers(pos))//在选定的worker上分配executor,一个worker可能分配多个executor
        }
      }
    } else {
      // Pack each app into as few workers as possible until we've assigned all its cores
      for (worker <- workers if worker.coresFree > 0 && worker.state == WorkerState.ALIVE) {
        for (app <- waitingApps if app.coresLeft > 0) {
          allocateWorkerResourceToExecutors(app, app.coresLeft, worker)
        }
      }
    }
  }

在这里需要补充一下，当启动Spark Application的时候如果没有设置spark.executor.cores，那么在SparkDeploySchedulerBackend中创建ApplicationDescription时，coresPerExecutor的值为Option(None)，这个时候Executor的cores个数回被设置成1，也就是默认情况下Executor的cores个数是1

private def  allocateWorkerResourceToExecutors(
    app: ApplicationInfo ,
    coresToAllocate:  Int,
    worker: WorkerInfo):  Unit = {
  val memoryPerExecutor = app.desc.memoryPerExecutorMB
  val coresPerExecutor = app.desc.coresPerExecutor.getOrElse(coresToAllocate) //如果没有明确指定一个executor core的个数，则把这个worker上分配的所有core分配给一个executor
  var coresLeft = coresToAllocate
  while (coresLeft >= coresPerExecutor && worker.memoryFree >= memoryPerExecutor) { //每次为一个executor分配的core个数至少为明确指定的core个数
    val exec = app.addExecutor(worker , coresPerExecutor)
    coresLeft -= coresPerExecutor
    launchExecutor(worker , exec)
    app. state = ApplicationState. RUNNING
  }
}

结论1：spark standalone模式当设置的单个executor core个数和application总的core的个数不合理的时候，可能导致executor无法分配。一个worker上可能存在某个application的多个executor。

情况1：有3个节点，每个节点30个cores，

命令：spark-submit --executor-cores 2 --total-executor-cores 12

在Master.startExecutorsOnWorkers 分配资源的时候，一个worker分配了4个cores，在allocateWorkerResourceToExecutors 为每个executor分配资源的时候，一次分配2个cores，因此启动了2个executor

情况2: 30个节点，每个节点30个cores

命令: spark-submit --executor-cores 5 --total-executor-cores 30

在Master.startExecutorsOnWorkers 分配资源的时候，一个worker分配了1个cores，在allocateWorkerResourceToExecutors 为每个executor分配资源的时候，一次分配最少5个cores，少于这个节点上可用的core的个数，所以导致不能分配executor

结论2：从startExecutorsOnWorkers 方法可以看出，在分配core的时候，每次一个position只分配一个core，通过这种方式让尽可能多的worker上运行executor