在上一节中,我们详细分析了作业是如何上岸的,现在作业 已经到达了RM端,并且交给了 RMAppManager进行继续运转,我们继续跟踪作业是如何在YARN中如何运转。
Server端:
ApplicationClientProtocolPBSeriveImpl.submitApplication() -> ClientRMService.submitApplicaiton() -> RMAppManger.submitApplication():
protected void submitApplication(
ApplicationSubmissionContext submissionContext, long submitTime,
String user) throws YarnException {
ApplicationId applicationId = submissionContext.getApplicationId();
//根据提交的,submissionContext创造一个RMAppImpl实例,并且创造一个applicationID到RMAppImpl的映射关系的缓存,方便之后查询
RMAppImpl application = createAndPopulateNewRMApp(
submissionContext, submitTime, user, false, -1);
try {
//是否开启安全机制,一些安全相关操作
if (UserGroupInformation.isSecurityEnabled()) {
this.rmContext.getDelegationTokenRenewer()
.addApplicationAsync(applicationId,
BuilderUtils.parseCredentials(submissionContext),
submissionContext.getCancelTokensWhenComplete(),
application.getUser(),
BuilderUtils.parseTokensConf(submissionContext));
} else {
//未开启安全机制,就获取dispatcher然后交给对应的EventHandler,处理RMAppEventType.START事件,触发了RMAppImpl对象的状态机
this.rmContext.getDispatcher().getEventHandler()
.handle(new RMAppEvent(applicationId, RMAppEventType.START));
}
} catch (Exception e) {
LOG.warn("Unable to parse credentials for " + applicationId, e);
//异常情况触发,RMAppEventType.APP_REJECTED事件
// Sending APP_REJECTED is fine, since we assume that the
// RMApp is in NEW state and thus we haven't yet informed the
// scheduler about the existence of the application
this.rmContext.getDispatcher().getEventHandler()
.handle(new RMAppEvent(applicationId,
RMAppEventType.APP_REJECTED, e.getMessage()));
throw RPCUtil.getRemoteException(e);
}
}
我们来看一下createAndPopulateNewRMApp这个函数的主要操作
ApplicationClientProtocolPBSeriveImpl.submitApplication() -> ClientRMService.submitApplicaiton() -> RMAppManger.submitApplication() -> RMAppManger.createAndPopulateNewRMApp():
private RMAppImpl createAndPopulateNewRMApp(
ApplicationSubmissionContext submissionContext, long submitTime,
String user, boolean isRecovery, long startTime) throws YarnException {
//得到当前作业的applicationId
ApplicationId applicationId = submissionContext.getApplicationId();
//检验资源请求是否合理
List<ResourceRequest> amReqs = validateAndCreateResourceRequest(
submissionContext, isRecovery);
// 创建RMApp的实例即 RMAppImpl
RMAppImpl application =
new RMAppImpl(applicationId, rmContext, this.conf,
submissionContext.getApplicationName(), user,
submissionContext.getQueue(),
submissionContext, this.scheduler, this.masterService,
submitTime, submissionContext.getApplicationType(),
submissionContext.getApplicationTags(), amReqs, placementContext,
startTime);
//如果rmContext中的缓存中没有该applicationId对应的applicationId,则将其缓存进去,需要的时候就可以直接查询
if (rmContext.getRMApps().putIfAbsent(applicationId, application) !=
null) {
String message = "Application with id " + applicationId
+ " is already present! Cannot add a duplicate!";
LOG.warn(message);
throw new YarnException(message);
}
创建RMAppImpl ,是最主要的操作,这是一个作业在RM端的呈现形式。构造此类的参数中包括两部分的内容。
客户端提交过来的submissionContext相关的一些信息:
作业的名字:getApplicationName(),作业的ID:applicationId,提交的队列:getQueue(),作业类型:getApplicationType(), 作业的标志信息:getApplicationTags(), 资源请求 amReqs , 等等。RM端RMAppManager提供的的一些相关信息:
RM的上下文信息:rmContext,一些相关配置信息:this.conf,采用的调度器是哪一个:this.scheduler, 管理AM的服务:this.masterService 等。
回到上面的 RMAppManger.submitApplication()过程:
this.rmContext.getDispatcher().getEventHandler()
.handle(new RMAppEvent(applicationId, RMAppEventType.START));
作业的提交在RM端触发了,作业RMAppEventType.START事件。通过RPC传到RM端的每一个作业,RMAppManager都会为其创建一个RMAppImpl对象,是RM中表示一个作业的对象,作业的运转是通过状态机来表示的,关于状态机,会在其他章节进行详细讲解,这里就是RMAppImpl的状态机。
this.rmContext.getDispatcher()得到的是一个异步调度器,是一个AsyncDispatcher类对象rmDispatcher ,然后通过注册到这个对象具体的rmDispatcher.getEventHandler()得到一个实现了不同界面的EventHandler事件类,进行handle()具体的事件,然后调用该事件对应的跳变函数,同时发生对应的状态机跳变。这里发送的事件是RMAppEventType.START事件。
RMAppManager中的rmContext是ResourceManager中的rmConext对象传进去的,我们首先来看一下rmConext在ResourceManager类中是如何创建的:
protected void serviceInit(Configuration conf) throws Exception {
this.conf = conf;
//先创造这个RMContextImpl这个类的实例
this.rmContext = new RMContextImpl();
//通过set的方式来填充一些变量,我们这里先只看Dispatcher相关的set操作
rmDispatcher = setupDispatcher();
addIfService(rmDispatcher);
rmContext.setDispatcher(rmDispatcher);
super.serviceInit(this.conf);
}
private Dispatcher setupDispatcher() {
Dispatcher dispatcher = createDispatcher();
dispatcher.register(RMFatalEventType.class,
new ResourceManager.RMFatalEventDispatcher());
return dispatcher;
}
protected Dispatcher createDispatcher() {
return new AsyncDispatcher("RM Event dispatcher");
}
我们可以看到AsyncDispatcher类对象 rmDispatcher 是在ResourceManager serviceInit()过程中创建的。在ResourceManager的内部类RMActiveServices 中的serviceInit()过程中,还给这个rmDispatcher 注册了很多EventHandler事件类,供以后进行对不同的事件进行分发到不同的事件类中:
//RM中包含所有活跃服务的类
public class RMActiveServices extends CompositeService {
//创建安全相关的Token的类
private DelegationTokenRenewer delegationTokenRenewer;
//调度器相关的事件类
private EventHandler<SchedulerEvent> schedulerDispatcher;
//在RM端发起AM的对象
private ApplicationMasterLauncher applicationMasterLauncher;
//容器分配过期对应的类
private ContainerAllocationExpirer containerAllocationExpirer;
private ResourceManager rm;
private boolean fromActive = false;
private StandByTransitionRunnable standByTransitionRunnable;
//RMActiveServices 中的serviceInit()过程,我们只关心rmDispatcher的注册过程
protected void serviceInit(Configuration configuration) throws Exception {
//为NodesListManager在rmDispatcher中注册事件处理器类
rmDispatcher.register(NodesListManagerEventType.class, nodesListManager);
//为调度器在rmDispatcher中注册事件处理器类
rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher);
//为RM中的application呈现对象RMApp在rmDispatcher注册事件处理器类
rmDispatcher.register(RMAppEventType.class, new ApplicationEventDispatcher(rmContext));
//为RM中的RMApp对象的一次尝试对应的对象RMAppAttempt在rmDispatcher注册事件处理器类
rmDispatcher.register(RMAppAttemptEventType.class, new ApplicationAttemptEventDispatcher(rmContext));
//为RM中的节点呈现对象RMNode在rmDispatcher注册事件处理器类
rmDispatcher.register(RMNodeEventType.class, new NodeEventDispatcher(rmContext));
注册的过程我们举一个例子看一下,就看为调度器在rmDispatcher中注册事件处理器类的过程吧:rmDispatcher.register(SchedulerEventType.class, schedulerDispatcher):
其中SchedulerEventType类,是事件的类型类,是一个enum,我们可以看到其中所有调度器相关的事件类型有哪些
public enum SchedulerEventType {
// Node操作对应的调度器事件类型
NODE_ADDED,
NODE_REMOVED,
NODE_UPDATE,
NODE_RESOURCE_UPDATE,
NODE_LABELS_UPDATE,
// RMApp操作对应的调度器事件类型
APP_ADDED,
APP_REMOVED,
// RMAppAttempt操作对应的调度器事件类型
APP_ATTEMPT_ADDED,
APP_ATTEMPT_REMOVED,
//ContainerAllocationExpirer对应的事件类型
CONTAINER_EXPIRED,
// Source: SchedulerAppAttempt::pullNewlyUpdatedContainer.
RELEASE_CONTAINER,
/* Source: SchedulingEditPolicy */
KILL_RESERVED_CONTAINER,
// Mark a container for preemption
MARK_CONTAINER_FOR_PREEMPTION,
// Mark a for-preemption container killable
MARK_CONTAINER_FOR_KILLABLE,
// Cancel a killable container
MARK_CONTAINER_FOR_NONKILLABLE,
//Queue Management Change
MANAGE_QUEUE
}
schedulerDispatcher就是一个调度器有关的事件对应的事件处理器类,调度器相关的事件处理器类较其他事件处理器类复杂很多,我们深入看一下,schedulerDispatcher = createSchedulerEventDispatcher() -> ResourceManager.createSchedulerEventDispatcher()
protected ResourceScheduler scheduler;
protected EventHandler<SchedulerEvent> createSchedulerEventDispatcher() {
return new EventDispatcher(this.scheduler, "SchedulerEventDispatcher");
}
public interface ResourceScheduler extends YarnScheduler, Recoverable {
}
public interface YarnScheduler extends EventHandler<SchedulerEvent> {
}
//传入的handler是一个ResourceScheduler ,继承了 YarnScheduler, 而YarnScheduler又继承了 EventHandler<SchedulerEvent>
大概了解了事件类型和,事件处理器后,再次回到上面的 RMAppManger.submitApplication()过程:
this.rmContext.getDispatcher().getEventHandler()
.handle(new RMAppEvent(applicationId, RMAppEventType.START));
this.rmContext.getDispatcher().getEventHandler()这个过程我们上面已经了解了,rmDispatcher把事件交给之前注册过的RMApp对应的事件处理器之后,handle()函数就是对于具体事件类型,事件处理器的操作了:
ApplicationEventDispatcher.handle():
public void handle(RMAppEvent event) {
ApplicationId appID = event.getApplicationId();
RMApp rmApp = this.rmContext.getRMApps().get(appID);
if (rmApp != null) {
try {
rmApp.handle(event);
} catch (Throwable t) {
LOG.error("Error in handling event type " + event.getType()
+ " for application " + appID, t);
}
}
}
事件处理器,把对应的事件类型转交给RMApp,然后调用rmApp.handle(event),事件上调用的是RMAppImpl.handle(event),为什么可以转交给RMAppImpl处理呢,因为RMAppImpl也是实现了EventHandler,RMAppImpl的类图如下,实现的接口RMApp是继承自EventHandler的:
ApplicationEventDispatcher.handle() -> RMAppImpl.handle() :
public void handle(RMAppEvent event) {
this.writeLock.lock();
try {
ApplicationId appID = event.getApplicationId();
LOG.debug("Processing event for " + appID + " of type "
+ event.getType());
final RMAppState oldState = getState();
try {
/* keep the master in sync with the state machine */
//根据事件类型,完成相应的状态机跳变操作
this.stateMachine.doTransition(event.getType(), event);
} catch (InvalidStateTransitionException e) {
LOG.error("App: " + appID
+ " can't handle this event at current state", e);
onInvalidStateTransition(event.getType(), oldState);
}
// Log at INFO if we're not recovering or not in a terminal state.
// Log at DEBUG otherwise.
if ((oldState != getState()) &&
(((recoveredFinalState == null)) ||
(event.getType() != RMAppEventType.RECOVER))) {
LOG.info(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
getState(), event.getType()));
} else if ((oldState != getState()) && LOG.isDebugEnabled()) {
LOG.debug(String.format(STATE_CHANGE_MESSAGE, appID, oldState,
getState(), event.getType()));
}
} finally {
this.writeLock.unlock();
}
}
现在传入的事件类型 event.getType()即为: RMAppEventType.START。RMAppImpl通过StateMachineFactory,初始化状态机跳变的函数,我们截取部分来看一下,初始化的RMApp的初始化状态是RMAppState.NEW状态,接下来是添加的从NEW状态转移到其他状态对应的跳变操作函数。当然StateMachineFactory不仅仅是从NEW状态跳变相关的部分,还保存了其他状态进行转移的跳变表。
private static final StateMachineFactory<RMAppImpl,
RMAppState,
RMAppEventType,
RMAppEvent> stateMachineFactory
= new StateMachineFactory<RMAppImpl,
RMAppState,
RMAppEventType,
RMAppEvent>(RMAppState.NEW)
// Transitions from NEW state
.addTransition(RMAppState