yarn3.2源码分析之YarnClient与ResourceManager通信之submitApplication（三）

概述

submitApplication会经过一系列Event：

1. app_accepted类型的RMAppEvent  
2. start类型的RMAppAttemptEvent 
3. appAttemptAdd类型的SchedulerEvent 
4. AttemptAdded类型的RMAppAttemptEvent 
5. storeAppAttempt类型的RMStateStoreEvent 
6. AttemptNewSaved类型的RMAppAttemptEvent 
7. launch类型的AMLaucherEvent

在处理app_accepted类型的RMAppEvent的过程中，创建RMAppAttempt。

在处理start类型的RMAppAttemptEvent的过程中，记录RMAppAttempt id到ApplicationMasterService。

在处理 appAttemptAdd类型的SchedulerEvent的过程中，获取AbstractYarnScheduler中的applications集合，记录其相应appId的currentAppAttempt为此RMAppAttempt。

在处理storeAppAttempt类型的RMStateStoreEvent的过程中，将RMAppAttempt封装成ApplicationAttemptStateData，存储到RMStateStore。

在处理AttemptNewSaved类型的RMAppAttemptEvent的过程中，注册ClientTokenMasterKey。

 

 

 StartAppAttemptTransition处理APP_ACCEPTED类型的RMAppEvent

中央异步调度器AsyncDispatcher将RMAppEvent分发给ApplicationEventDispatcher处理。

ApplicationEventDispatcher获取RMAppEvent对应的RMAppImpl，委托给RMAppImpl处理RMAppEvent。

RMAppImpl注册处理app_accepted类型的RMAppEvent的Transition。

 .addTransition(RMAppState.SUBMITTED, RMAppState.ACCEPTED,
        RMAppEventType.APP_ACCEPTED, new StartAppAttemptTransition())

StartAppAttemptTransition创建RMAppAttemptImpl实例，然后创建start类型的RMAppAttemptEvent，并提交给AsyncDispatcher

private static final class StartAppAttemptTransition extends RMAppTransition {
    @Override
    public void transition(RMAppImpl app, RMAppEvent event) {
      app.createAndStartNewAttempt(false);
    };
  }

private void
      createAndStartNewAttempt(boolean transferStateFromPreviousAttempt) {
    createNewAttempt();
    handler.handle(new RMAppStartAttemptEvent(currentAttempt.getAppAttemptId(),
      transferStateFromPreviousAttempt));
  }

 public RMAppStartAttemptEvent(ApplicationAttemptId appAttemptId,
      boolean transferStateFromPreviousAttempt) {
    super(appAttemptId, RMAppAttemptEventType.START);
    this.transferStateFromPreviousAttempt = transferStateFromPreviousAttempt;
  }

ApplicationAttemptEventDispatcher处理start类型的RMAppAttemptEvent

ResourceManager的内部类RMActiveService为AsyncDispatcher注册RMAppAttemptEvent类型的处理器—— ApplicationAttemptEventDispatcher

 @Override
    protected void serviceInit(Configuration configuration) throws Exception {
 
      // Register event handler for RmAppAttemptEvents
      rmDispatcher.register(RMAppAttemptEventType.class,
          new ApplicationAttemptEventDispatcher(rmContext));
}

中央异步调度器AsyncDispatcher将RMAppAttemptEvent分发给 ApplicationAttemptEventDispatcher处理。

ApplicationAttemptEventDispatcher获取RMAppAttemptEvent对应的RMAppAttemptImpl，委托给RMAppAttemptImpl处理RMAppAttemptEvent。

@Private
  public static final class ApplicationAttemptEventDispatcher implements
      EventHandler<RMAppAttemptEvent> {

    private final RMContext rmContext;

    public ApplicationAttemptEventDispatcher(RMContext rmContext) {
      this.rmContext = rmContext;
    }

    @Override
    public void handle(RMAppAttemptEvent event) {
      ApplicationAttemptId appAttemptId = event.getApplicationAttemptId();
      ApplicationId appId = appAttemptId.getApplicationId();
      RMApp rmApp = this.rmContext.getRMApps().get(appId);
      if (rmApp != null) {
        RMAppAttempt rmAppAttempt = rmApp.getRMAppAttempt(appAttemptId);
        if (rmAppAttempt != null) {
          try {
            rmAppAttempt.handle(event);
          } catch (Throwable t) {
            LOG.error("Error in handling event type " + event.getType()
                + " for applicationAttempt " + appAttemptId, t);
          }
        } else if (rmApp.getApplicationSubmissionContext() != null
            && rmApp.getApplicationSubmissionContext()
            .getKeepContainersAcrossApplicationAttempts()
            && event.getType() == RMAppAttemptEventType.CONTAINER_FINISHED) {
          // For work-preserving AM restart, failed attempts are still
          // capturing CONTAINER_FINISHED events and record the finished
          // containers which will be used by current attempt.
          // We just keep 'yarn.resourcemanager.am.max-attempts' in
          // RMStateStore. If the finished container's attempt is deleted, we
          // use the first attempt in app.attempts to deal with these events.

          RMAppAttempt previousFailedAttempt =
              rmApp.getAppAttempts().values().iterator().next();
          if (previousFailedAttempt != null) {
            try {
              LOG.debug("Event " + event.getType() + " handled by "
                  + previousFailedAttempt);
              previousFailedAttempt.handle(event);
            } catch (Throwable t) {
              LOG.error("Error in handling event type " + event.getType()
                  + " for applicationAttempt " + appAttemptId
                  + " with " + previousFailedAttempt, t);
            }
          } else {
            LOG.error("Event " + event.getType()
                + " not handled, because previousFailedAttempt is null");
          }
        }
      }
    }
  }

RMAppAttemptImpl处理RMAppAttemptEvent

使用transition进行状态迁移

@Override
  public void handle(RMAppAttemptEvent event) {

    this.writeLock.lock();

    try {
  .........
      try {
        /* keep the master in sync with the state machine */
        this.stateMachine.doTransition(event.getType(), event);
      } catch (InvalidStateTransitionException e) {
        ................
      }
......................
    } finally {
      this.writeLock.unlock();
    }
  }

AttemptStartedTransition处理start类型的RMAppAttemptEvent

RMAppAttemptImpl的内部stateMachineFactory注册处理start类型的RMAppAttemptEvent的transition —— AttemptStartedTransition

.addTransition(RMAppAttemptState.NEW, RMAppAttemptState.SUBMITTED,
          RMAppAttemptEventType.START, new AttemptStartedTransition())

AttemptedStartedTransition创建appAttemptAdd类型的SchedulerEvent，并提交给AsyncDspatcher

  private static final class AttemptStartedTransition extends BaseTransition {
	@Override
    public void transition(RMAppAttemptImpl appAttempt,
        RMAppAttemptEvent event) {

	    boolean transferStateFromPreviousAttempt = false;
      if (event instanceof RMAppStartAttemptEvent) {
        transferStateFromPreviousAttempt =
            ((RMAppStartAttemptEvent) event)
              .getTransferStateFromPreviousAttempt();
      }
//赋值RMAppAttempt的startTime为当前时间
      appAttempt.startTime = System.currentTimeMillis();

      // Register with the ApplicationMasterService
//记录RMAppAttempt id到ApplicatonMasterService
      appAttempt.masterService
          .registerAppAttempt(appAttempt.applicationAttemptId);

      if (UserGroupInformation.isSecurityEnabled()) {
//赋值RMAppAttempt的clientTokenMasterKey
        appAttempt.clientTokenMasterKey =
            appAttempt.rmContext.getClientToAMTokenSecretManager()
              .createMasterKey(appAttempt.applicationAttemptId);
      }

      // Add the applicationAttempt to the scheduler and inform the scheduler
      // whether to transfer the state from previous attempt.
      appAttempt.eventHandler.handle(new AppAttemptAddedSchedulerEvent(
        appAttempt.applicationAttemptId, transferStateFromPreviousAttempt));
    }
  }

public AppAttemptAddedSchedulerEvent(
      ApplicationAttemptId applicationAttemptId,
      boolean transferStateFromPreviousAttempt,
      boolean isAttemptRecovering) {
    super(SchedulerEventType.APP_ATTEMPT_ADDED);
    this.applicationAttemptId = applicationAttemptId;
    this.transferStateFromPreviousAttempt = transferStateFromPreviousAttempt;
    this.isAttemptRecovering = isAttemptRecovering;
  }

AsyncDispatcher分发appAttemptAdded类型的SchedulerEvent给schedulerDispatcher

ResourceManager的内部类RMActiveService为AsyncDispatcher注册事件——SchedulerEventType类型由schedulerDispatcher处理。

protected void serviceInit(Configuration configuration) throws Exception {
      scheduler = createScheduler();
      scheduler.setRMContext(rmContext);
      addIfService(scheduler);
      rmContext.setScheduler(scheduler);
 
      schedulerDispatcher = createSchedulerEventDispatcher();
      addIfService(schedulerDispatcher);
      rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);
    
}
 
protected EventHandler<SchedulerEvent> createSchedulerEventDispatcher() {
    return new EventDispatcher(this.scheduler, "SchedulerEventDispatcher");
  }

FifoScheduler处理appAttemptAdded类型的SchedulerEvent

假设在ResourceManager中，createScheduler()方法创建的是FifoScheduler。

FifoScheduler创建attemptAdded类型的RMAppAttemptEvent，并提交给中央异步调度器AsyncDispatcher。

  @Override
  public void handle(SchedulerEvent event) {
    switch(event.getType()) {   
    .............. 
    case APP_ATTEMPT_ADDED:
    {
      AppAttemptAddedSchedulerEvent appAttemptAddedEvent =
          (AppAttemptAddedSchedulerEvent) event;
      addApplicationAttempt(appAttemptAddedEvent.getApplicationAttemptId(),
        appAttemptAddedEvent.getTransferStateFromPreviousAttempt(),
        appAttemptAddedEvent.getIsAttemptRecovering());
    }
    break;
    ..................
  }

public synchronized void
      addApplicationAttempt(ApplicationAttemptId appAttemptId,
          boolean transferStateFromPreviousAttempt,
          boolean isAttemptRecovering) {
//从AbstractYarnScheduler中的applications集合获取相应app
    SchedulerApplication<FifoAppAttempt> application =
        applications.get(appAttemptId.getApplicationId());
    String user = application.getUser();
    // TODO: Fix store
    FifoAppAttempt schedulerApp =
        new FifoAppAttempt(appAttemptId, user, DEFAULT_QUEUE,
          activeUsersManager, this.rmContext);

    if (transferStateFromPreviousAttempt) {
      schedulerApp.transferStateFromPreviousAttempt(application
        .getCurrentAppAttempt());
    }
//记录相应app的currentAppAttempt为此RMAppAttempt
    application.setCurrentAppAttempt(schedulerApp);

    metrics.submitAppAttempt(user);
    LOG.info("Added Application Attempt " + appAttemptId
        + " to scheduler from user " + application.getUser());
    if (isAttemptRecovering) {
      if (LOG.isDebugEnabled()) {
        LOG.debug(appAttemptId
            + " is recovering. Skipping notifying ATTEMPT_ADDED");
      }
    } else {
      rmContext.getDispatcher().getEventHandler().handle(
        new RMAppAttemptEvent(appAttemptId,
            RMAppAttemptEventType.ATTEMPT_ADDED));
    }
  }

ScheduleTransition处理attemptAdded类型的RMAppAttemptEvent

中央异步调度器AsyncDispatcher将RMAppAttemptEvent分发给 ApplicationAttemptEventDispatcher处理。

ApplicationAttemptEventDispatcher获取RMAppAttemptEvent对应的RMAppAttemptImpl，委托给RMAppAttemptImpl处理RMAppAttemptEvent。

RMAppAttemptImpl添加处理AttemptAdded类型的RMAppAttemptEvent的transition —— ScheduleTransition

// Transitions from SUBMITTED state
      .addTransition(RMAppAttemptState.SUBMITTED, 
          EnumSet.of(RMAppAttemptState.LAUNCHED_UNMANAGED_SAVING,
                     RMAppAttemptState.SCHEDULED),
          RMAppAttemptEventType.ATTEMPT_ADDED,
          new ScheduleTransition())

ScheduleTransition处理attemptAdded类型的RMAppAttemptEvent

  public static final class ScheduleTransition
      implements
      MultipleArcTransition<RMAppAttemptImpl, RMAppAttemptEvent, RMAppAttemptState> {
    @Override
    public RMAppAttemptState transition(RMAppAttemptImpl appAttempt,
        RMAppAttemptEvent event) {
      ApplicationSubmissionContext subCtx = appAttempt.submissionContext;
//判断RM是否应该管理AM的执行
//如果AM的执行应该被RM管理
      if (!subCtx.getUnmanagedAM()) {
        // Need reset #containers before create new attempt, because this request
        // will be passed to scheduler, and scheduler will deduct the number after
        // AM container allocated
        
        // Currently, following fields are all hard coded,
        // TODO: change these fields when we want to support
        // priority or multiple containers AM container allocation.
        for (ResourceRequest amReq : appAttempt.amReqs) {
          amReq.setNumContainers(1);
          amReq.setPriority(AM_CONTAINER_PRIORITY);
        }

        int numNodes =
            RMServerUtils.getApplicableNodeCountForAM(appAttempt.rmContext,
                appAttempt.conf, appAttempt.amReqs);
        if (LOG.isDebugEnabled()) {
          LOG.debug("Setting node count for blacklist to " + numNodes);
        }
        appAttempt.getAMBlacklistManager().refreshNodeHostCount(numNodes);

        ResourceBlacklistRequest amBlacklist =
            appAttempt.getAMBlacklistManager().getBlacklistUpdates();
        if (LOG.isDebugEnabled()) {
          LOG.debug("Using blacklist for AM: additions(" +
              amBlacklist.getBlacklistAdditions() + ") and removals(" +
              amBlacklist.getBlacklistRemovals() + ")");
        }

        QueueInfo queueInfo = null;
        for (ResourceRequest amReq : appAttempt.amReqs) {
          if (amReq.getNodeLabelExpression() == null && ResourceRequest.ANY
              .equals(amReq.getResourceName())) {
            String queue = appAttempt.rmApp.getQueue();

            //Load queue only once since queue will be same across attempts
            if (queueInfo == null) {
              try {
                queueInfo = appAttempt.scheduler.getQueueInfo(queue, false,
                    false);
              } catch (IOException e) {
                LOG.error("Could not find queue for application : ", e);
                // Set application status to REJECTED since we cant find the
                // queue
                appAttempt.rmContext.getDispatcher().getEventHandler().handle(
                    new RMAppAttemptEvent(appAttempt.getAppAttemptId(),
                        RMAppAttemptEventType.FAIL,
                        "Could not find queue for application : " +
                        appAttempt.rmApp.getQueue()));
                appAttempt.rmContext.getDispatcher().getEventHandler().handle(
                    new RMAppEvent(appAttempt.rmApp.getApplicationId(), RMAppEventType
                        .APP_REJECTED,
                        "Could not find queue for application : " +
                            appAttempt.rmApp.getQueue()));
                return RMAppAttemptState.FAILED;
              }
            }

            String labelExp = RMNodeLabelsManager.NO_LABEL;
            if (queueInfo != null) {
              if (LOG.isDebugEnabled()) {
                LOG.debug("Setting default node label expression : " + queueInfo
                    .getDefaultNodeLabelExpression());
              }
              labelExp = queueInfo.getDefaultNodeLabelExpression();
            }

            amReq.setNodeLabelExpression(labelExp);
          }
        }

        // AM resource has been checked when submission
        Allocation amContainerAllocation =
            appAttempt.scheduler.allocate(
                appAttempt.applicationAttemptId,
                appAttempt.amReqs, null, EMPTY_CONTAINER_RELEASE_LIST,
                amBlacklist.getBlacklistAdditions(),
                amBlacklist.getBlacklistRemovals(),
                new ContainerUpdates());
        if (amContainerAllocation != null
            && amContainerAllocation.getContainers() != null) {
          assert (amContainerAllocation.getContainers().size() == 0);
        }
        return RMAppAttemptState.SCHEDULED;
      } else {
//如果AM的执行不用被RM管理
        // save state and then go to LAUNCHED state
        appAttempt.storeAttempt();
        return RMAppAttemptState.LAUNCHED_UNMANAGED_SAVING;
      }
    }
  }

AppAttemptImpl存储Attempt

private void storeAttempt() {
    // store attempt data in a non-blocking manner to prevent dispatcher
    // thread starvation and wait for state to be saved
    LOG.info("Storing attempt: AppId: " + 
              getAppAttemptId().getApplicationId() 
              + " AttemptId: " + 
              getAppAttemptId()
              + " MasterContainer: " + masterContainer);
    rmContext.getStateStore().storeNewApplicationAttempt(this);
  }

RMStateStore将RMAppAttempt封装成ApplicationAttemptStateData，创建storeAppAttempt类型的storeEvent，提交给中央异步调度器AsyncDispatcher

/**
   * Non-blocking API
   * ResourceManager services call this to store state on an application attempt
   * This does not block the dispatcher threads
   * RMAppAttemptStoredEvent will be sent on completion to notify the RMAppAttempt
   */
  public void storeNewApplicationAttempt(RMAppAttempt appAttempt) {
    Credentials credentials = getCredentialsFromAppAttempt(appAttempt);
    RMAppAttemptMetrics attempMetrics = appAttempt.getRMAppAttemptMetrics();
    AggregateAppResourceUsage resUsage =
        attempMetrics.getAggregateAppResourceUsage();
    ApplicationAttemptStateData attemptState =
        ApplicationAttemptStateData.newInstance(
            appAttempt.getAppAttemptId(),
            appAttempt.getMasterContainer(),
            credentials, appAttempt.getStartTime(),
            resUsage.getResourceUsageSecondsMap(),
            attempMetrics.getPreemptedResourceSecondsMap());

    getRMStateStoreEventHandler().handle(
      new RMStateStoreAppAttemptEvent(attemptState));
  }

  public RMStateStoreAppAttemptEvent(ApplicationAttemptStateData attemptState) {
    super(RMStateStoreEventType.STORE_APP_ATTEMPT);
    this.attemptState = attemptState;
  }

AsyncDispatcher将RMStateStoreEvent分发给ForwardingEventHandler处理 

（略，见上一篇博客）

StoreAppAttemptTransition处理storeAppAttempt类型的RMStateStoreEvent

RMStateStore的内部StateMachineFactory添加处理storeAppAttempt类型的RMStateStoreEvent的transition。

 .addTransition(RMStateStoreState.ACTIVE,
          EnumSet.of(RMStateStoreState.ACTIVE, RMStateStoreState.FENCED),
          RMStateStoreEventType.STORE_APP_ATTEMPT,
          new StoreAppAttemptTransition())

StoreAppAttemptTransition创建AttemptNewSaved类型的RMAppAttemptEvent，并提交到AsyncDispatcher

private static class StoreAppAttemptTransition implements
      MultipleArcTransition<RMStateStore, RMStateStoreEvent,
          RMStateStoreState> {
    @Override
    public RMStateStoreState transition(RMStateStore store,
        RMStateStoreEvent event) {
      if (!(event instanceof RMStateStoreAppAttemptEvent)) {
        // should never happen
        LOG.error("Illegal event type: " + event.getClass());
        return RMStateStoreState.ACTIVE;
      }
      boolean isFenced = false;
      ApplicationAttemptStateData attemptState =
          ((RMStateStoreAppAttemptEvent) event).getAppAttemptState();
      try {
        if (LOG.isDebugEnabled()) {
          LOG.debug("Storing info for attempt: " + attemptState.getAttemptId());
        }
//将ApplicationAttemptStateData存储到RMStateStore
        store.storeApplicationAttemptStateInternal(attemptState.getAttemptId(),
            attemptState);
        store.notifyApplicationAttempt(new RMAppAttemptEvent
               (attemptState.getAttemptId(),
               RMAppAttemptEventType.ATTEMPT_NEW_SAVED));
      } catch (Exception e) {
        LOG.error("Error storing appAttempt: " + attemptState.getAttemptId(), e);
        isFenced = store.notifyStoreOperationFailedInternal(e);
      }
      return finalState(isFenced);
    };
  }

 /**
   * This method is called to notify the application attempt
   * that new attempt is stored or updated in state store
   * @param event App attempt event containing the app attempt
   * id and event type
   */
  private void notifyApplicationAttempt(RMAppAttemptEvent event) {
    rmDispatcher.getEventHandler().handle(event);
  }

AttemptStoredTransition处理AttemptNewSaved类型的RMAppAttemptEvent

RMAppAttemptImpl的内部StateMachineFactory添加处理AttemptNewSaved类型的RMAppAttemptEvent的transition  —— AttemptStoredTransition

// Transitions from ALLOCATED_SAVING State
      .addTransition(RMAppAttemptState.ALLOCATED_SAVING, 
          RMAppAttemptState.ALLOCATED,
          RMAppAttemptEventType.ATTEMPT_NEW_SAVED, new AttemptStoredTransition())

AttemptStoredTransition创建launch类型的AMLaucherEvent，提交给AsyncDispatcher

private static final class AttemptStoredTransition extends BaseTransition {
    @Override
    public void transition(RMAppAttemptImpl appAttempt,
                                                    RMAppAttemptEvent event) {
//在保存到RMStateStore后，注册ClientTokenMasterKey
      appAttempt.registerClientToken();
      appAttempt.launchAttempt();
    }
  }

private void launchAttempt(){
    launchAMStartTime = System.currentTimeMillis();
    // Send event to launch the AM Container
    eventHandler.handle(new AMLauncherEvent(AMLauncherEventType.LAUNCH, this));
  }