Hadoop Yarn 3.1.0 源码分析 （02 作业调度）

在上一节中，我们详细分析了作业是如何上岸的，现在作业 已经到达了RM端，并且交给了 RMAppManager进行继续运转，我们继续跟踪作业是如何在YARN中如何运转。
Server端：
ApplicationClientProtocolPBSeriveImpl.submitApplication() -> ClientRMService.submitApplicaiton() -> RMAppManger.submitApplication():

 protected void submitApplication(
      ApplicationSubmissionContext submissionContext, long submitTime,
      String user) throws YarnException {
    ApplicationId applicationId = submissionContext.getApplicationId();
    //根据提交的，submissionContext创造一个RMAppImpl实例，并且创造一个applicationID到RMAppImpl的映射关系的缓存，方便之后查询
    RMAppImpl application = createAndPopulateNewRMApp(
        submissionContext, submitTime, user, false, -1);
    try {
      //是否开启安全机制，一些安全相关操作
      if (UserGroupInformation.isSecurityEnabled()) {
        this.rmContext.getDelegationTokenRenewer()
            .addApplicationAsync(applicationId,
                BuilderUtils.parseCredentials(submissionContext),
                submissionContext.getCancelTokensWhenComplete(),
                application.getUser(),
                BuilderUtils.parseTokensConf(submissionContext));
      } else {
        //未开启安全机制，就获取dispatcher然后交给对应的EventHandler,处理RMAppEventType.START事件，触发了RMAppImpl对象的状态机
        this.rmContext.getDispatcher().getEventHandler()
            .handle(new RMAppEvent(applicationId, RMAppEventType.START));
      }
    } catch (Exception e) {
      LOG.warn("Unable to parse credentials for " + applicationId, e);
      //异常情况触发，RMAppEventType.APP_REJECTED事件
      // Sending APP_REJECTED is fine, since we assume that the
      // RMApp is in NEW state and thus we haven't yet informed the
      // scheduler about the existence of the application
      this.rmContext.getDispatcher().getEventHandler()
          .handle(new RMAppEvent(applicationId,
              RMAppEventType.APP_REJECTED, e.getMessage()));
      throw RPCUtil.getRemoteException(e);
    }
  }

我们来看一下createAndPopulateNewRMApp这个函数的主要操作
ApplicationClientProtocolPBSeriveImpl.submitApplication() -> ClientRMService.submitApplicaiton() -> RMAppManger.submitApplication() -> RMAppManger.createAndPopulateNewRMApp():

  private RMAppImpl createAndPopulateNewRMApp(
      ApplicationSubmissionContext submissionContext, long submitTime,
      String user, boolean isRecovery, long startTime) throws YarnException {
      //得到当前作业的applicationId
       ApplicationId applicationId = submissionContext.getApplicationId();
      //检验资源请求是否合理
    List<ResourceRequest> amReqs = validateAndCreateResourceRequest(
        submissionContext, isRecovery);
      // 创建RMApp的实例即 RMAppImpl
    RMAppImpl application =
        new RMAppImpl(applicationId, rmContext, this.conf,
            submissionContext.getApplicationName(), user,
            submissionContext.getQueue(),
            submissionContext, this.scheduler, this.masterService,
            submitTime, submissionContext.getApplicationType(),
            submissionContext.getApplicationTags(), amReqs, placementContext,
            startTime);
     //如果rmContext中的缓存中没有该applicationId对应的applicationId，则将其缓存进去，需要的时候就可以直接查询
if (rmContext.getRMApps().putIfAbsent(applicationId, application) !=
        null) {
      String message = "Application with id " + applicationId
          + " is already present! Cannot add a duplicate!";
      LOG.warn(message);
      throw new YarnException(message);
    }

创建RMAppImpl ，是最主要的操作，这是一个作业在RM端的呈现形式。构造此类的参数中包括两部分的内容。

• 客户端提交过来的submissionContext相关的一些信息：
  作业的名字：getApplicationName（），作业的ID:applicationId，提交的队列：getQueue（），作业类型：getApplicationType（）， 作业的标志信息：getApplicationTags（）， 资源请求 amReqs ， 等等。

• RM端RMAppManager提供的的一些相关信息：
  RM的上下文信息：rmContext，一些相关配置信息：this.conf，采用的调度器是哪一个：this.scheduler, 管理AM的服务：this.masterService 等。

回到上面的 RMAppManger.submitApplication()过程：

        this.rmContext.getDispatcher().getEventHandler()
            .handle(new RMAppEvent(applicationId, RMAppEventType.START));

作业的提交在RM端触发了，作业RMAppEventType.START事件。通过RPC传到RM端的每一个作业，RMAppManager都会为其创建一个RMAppImpl对象，是RM中表示一个作业的对象，作业的运转是通过状态机来表示的，关于状态机，会在其他章节进行详细讲解，这里就是RMAppImpl的状态机。
this.rmContext.getDispatcher()得到的是一个异步调度器，是一个AsyncDispatcher类对象rmDispatcher ，然后通过注册到这个对象具体的rmDispatcher.getEventHandler()得到一个实现了不同界面的EventHandler事件类，进行handle()具体的事件，然后调用该事件对应的跳变函数，同时发生对应的状态机跳变。这里发送的事件是RMAppEventType.START事件。
RMAppManager中的rmContext是ResourceManager中的rmConext对象传进去的，我们首先来看一下rmConext在ResourceManager类中是如何创建的：

protected void serviceInit(Configuration conf) throws Exception {
    this.conf = conf;
    //先创造这个RMContextImpl这个类的实例
    this.rmContext = new RMContextImpl();
    //通过set的方式来填充一些变量，我们这里先只看Dispatcher相关的set操作 
    rmDispatcher = setupDispatcher();
    addIfService(rmDispatcher);
    rmContext.setDispatcher(rmDispatcher);
    super.serviceInit(this.conf);
  }

 private Dispatcher setupDispatcher() {
    Dispatcher dispatcher = createDispatcher();
    dispatcher.register(RMFatalEventType.class,
        new ResourceManager.RMFatalEventDispatcher());
    return dispatcher;
  }

  protected Dispatcher createDispatcher() {
    return new AsyncDispatcher("RM Event dispatcher");
  }

我们可以看到AsyncDispatcher类对象 rmDispatcher 是在ResourceManager serviceInit()过程中创建的。在ResourceManager的内部类RMActiveServices 中的serviceInit()过程中，还给这个rmDispatcher 注册了很多EventHandler事件类，供以后进行对不同的事件进行分发到不同的事件类中：

//RM中包含所有活跃服务的类
public class RMActiveServices extends CompositeService {
   
    //创建安全相关的Token的类 
    private DelegationTokenRenewer delegationTokenRenewer;
    //调度器相关的事件类
    private EventHandler<SchedulerEvent> schedulerDispatcher;
    //在RM端发起AM的对象
    private ApplicationMasterLauncher applicationMasterLauncher;
    //容器分配过期对应的类
    private ContainerAllocationExpirer containerAllocationExpirer;
    private ResourceManager rm;
    private boolean fromActive = false;
    private StandByTransitionRunnable standByTransitionRunnable;
 //RMActiveServices 中的serviceInit()过程，我们只关心rmDispatcher的注册过程
 protected void serviceInit(Configuration configuration) throws Exception {
    //为NodesListManager在rmDispatcher中注册事件处理器类
    rmDispatcher.register(NodesListManagerEventType.class, nodesListManager);
    //为调度器在rmDispatcher中注册事件处理器类
    rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);
    //为RM中的application呈现对象RMApp在rmDispatcher注册事件处理器类
    rmDispatcher.register(RMAppEventType.class, new ApplicationEventDispatcher(rmContext));
    //为RM中的RMApp对象的一次尝试对应的对象RMAppAttempt在rmDispatcher注册事件处理器类
    rmDispatcher.register(RMAppAttemptEventType.class, new ApplicationAttemptEventDispatcher(rmContext));
    //为RM中的节点呈现对象RMNode在rmDispatcher注册事件处理器类
    rmDispatcher.register(RMNodeEventType.class, new NodeEventDispatcher(rmContext));

注册的过程我们举一个例子看一下，就看为调度器在rmDispatcher中注册事件处理器类的过程吧：rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher)：
其中SchedulerEventType类，是事件的类型类，是一个enum，我们可以看到其中所有调度器相关的事件类型有哪些

public enum SchedulerEventType {

  // Node操作对应的调度器事件类型
  NODE_ADDED,
  NODE_REMOVED,
  NODE_UPDATE,
  NODE_RESOURCE_UPDATE,
  NODE_LABELS_UPDATE,

  // RMApp操作对应的调度器事件类型
  APP_ADDED,
  APP_REMOVED,

  // RMAppAttempt操作对应的调度器事件类型
  APP_ATTEMPT_ADDED,
  APP_ATTEMPT_REMOVED,

  //ContainerAllocationExpirer对应的事件类型
  CONTAINER_EXPIRED,

  // Source: SchedulerAppAttempt::pullNewlyUpdatedContainer.
  RELEASE_CONTAINER,

  /* Source: SchedulingEditPolicy */
  KILL_RESERVED_CONTAINER,

  // Mark a container for preemption
  MARK_CONTAINER_FOR_PREEMPTION,

  // Mark a for-preemption container killable
  MARK_CONTAINER_FOR_KILLABLE,

  // Cancel a killable container
  MARK_CONTAINER_FOR_NONKILLABLE,

  //Queue Management Change
  MANAGE_QUEUE
}

schedulerDispatcher就是一个调度器有关的事件对应的事件处理器类，调度器相关的事件处理器类较其他事件处理器类复杂很多，我们深入看一下，schedulerDispatcher = createSchedulerEventDispatcher() -> ResourceManager.createSchedulerEventDispatcher()

protected ResourceScheduler scheduler;
protected EventHandler<SchedulerEvent> createSchedulerEventDispatcher() {
    return new EventDispatcher(this.scheduler, "SchedulerEventDispatcher");
  }

public interface ResourceScheduler extends YarnScheduler, Recoverable {
    }

public interface YarnScheduler extends EventHandler<SchedulerEvent> {
   }

//传入的handler是一个ResourceScheduler ，继承了 YarnScheduler， 而YarnScheduler又继承了 EventHandler<SchedulerEvent>

大概了解了事件类型和，事件处理器后，再次回到上面的 RMAppManger.submitApplication()过程：

        this.rmContext.getDispatcher().getEventHandler()
            .handle(new RMAppEvent(applicationId, RMAppEventType.START));

this.rmContext.getDispatcher().getEventHandler()这个过程我们上面已经了解了，rmDispatcher把事件交给之前注册过的RMApp对应的事件处理器之后，handle（）函数就是对于具体事件类型，事件处理器的操作了：
ApplicationEventDispatcher.handle():

public void handle(RMAppEvent event) {
      ApplicationId appID = event.getApplicationId();
      RMApp rmApp = this.rmContext.getRMApps().get(appID);
      if (rmApp != null) {
        try {
          rmApp.handle(event);
        } catch (Throwable t) {
          LOG.error("Error in handling event type " + event.getType()
              + " for application " + appID, t);
        }
      }
    }

事件处理器，把对应的事件类型转交给RMApp，然后调用rmApp.handle(event)，事件上调用的是RMAppImpl.handle(event)，为什么可以转交给RMAppImpl处理呢，因为RMAppImpl也是实现了EventHandler，RMAppImpl的类图如下，实现的接口RMApp是继承自EventHandler的：

ApplicationEventDispatcher.handle() -> RMAppImpl.handle() :

public void handle(RMAppEvent event) {

    this.writeLock.lock();

    try {
      ApplicationId appID = event.getApplicationId();
      LOG.debug("Processing event for " + appID + " of type "
          + event.getType());
      final RMAppState oldState = getState();
      try {
        /* keep the master in sync with the state machine */
        //根据事件类型，完成相应的状态机跳变操作
        this.stateMachine.doTransition(event.getType(), event);
      } catch (InvalidStateTransitionException e) {
        LOG.error("App: " + appID
            + " can't handle this event at current state", e);
        onInvalidStateTransition(event.getType(), oldState);
      }

      // Log at INFO if we're not recovering or not in a terminal state.
      // Log at DEBUG otherwise.
      if ((oldState != getState()) &&
          (((recoveredFinalState == null)) ||
            (event.getType() != RMAppEventType.RECOVER))) {
        LOG.info(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
            getState(), event.getType()));
      } else if ((oldState != getState()) && LOG.isDebugEnabled()) {
        LOG.debug(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
            getState(), event.getType()));
      }
    } finally {
      this.writeLock.unlock();
    }
  }

现在传入的事件类型 event.getType()即为： RMAppEventType.START。RMAppImpl通过StateMachineFactory，初始化状态机跳变的函数，我们截取部分来看一下，初始化的RMApp的初始化状态是RMAppState.NEW状态，接下来是添加的从NEW状态转移到其他状态对应的跳变操作函数。当然StateMachineFactory不仅仅是从NEW状态跳变相关的部分，还保存了其他状态进行转移的跳变表。

private static final StateMachineFactory<RMAppImpl,
                                           RMAppState,
                                           RMAppEventType,
                                           RMAppEvent> stateMachineFactory
                               = new StateMachineFactory<RMAppImpl,
                                           RMAppState,
                                           RMAppEventType,
                                           RMAppEvent>(RMAppState.NEW)


     // Transitions from NEW state