夜阑听风

仰望新技术,低头敲代码

关于Yarn源码的那些事(六)

终于可以继续写ApplicationMaster提交和运行的整体流程了。

在上次分析到RMAppAttemptImpl的时候,觉得自己对于调度器和状态机了解地不是很清楚,因此暂停,先把一些需要了解的概念分析清楚,今天,继续分析ApplicationMaster提交运行的整体流程:

上文中,我们提到,RMAppAttemptImpl的状态从RMAppAttemptState.SCHEDULED,在事件RMAppAttemptEventType.CONTAINER_ALLOCATED的触发下,转换为RMAppAttemptedState.ALLOCATED_SAVING,同时,该转换操作为:AMContainerAllocatedTransition,我们看看其代码:

// Set the masterContainer
			appAttempt.setMasterContainer(amContainerAllocation.getContainers()
					.get(0));
			RMContainerImpl rmMasterContainer = (RMContainerImpl) appAttempt.scheduler
					.getRMContainer(appAttempt.getMasterContainer().getId());
			rmMasterContainer.setAMContainer(true);
			// The node set in NMTokenSecrentManager is used for marking whether
			// the
			// NMToken has been issued for this node to the AM.
			// When AM container was allocated to RM itself, the node which
			// allocates
			// this AM container was marked as the NMToken already sent. Thus,
			// clear this node set so that the following allocate requests from
			// AM are
			// able to retrieve the corresponding NMToken.
			appAttempt.rmContext.getNMTokenSecretManager()
					.clearNodeSetForAttempt(appAttempt.applicationAttemptId);
			appAttempt.getSubmissionContext().setResource(
					appAttempt.getMasterContainer().getResource());
			appAttempt.storeAttempt();
			return RMAppAttemptState.ALLOCATED_SAVING;

我们可以看到,里面对于我们本次提交的这个对象实例,主要是取出其submissionContext,即运行环境上下文,赋予了相应的资源,其中storeAttempt方法:

private void storeAttempt() {
		// store attempt data in a non-blocking manner to prevent dispatcher
		// thread starvation and wait for state to be saved
		LOG.info("Storing attempt: AppId: "
				+ getAppAttemptId().getApplicationId() + " AttemptId: "
				+ getAppAttemptId() + " MasterContainer: " + masterContainer);
		rmContext.getStateStore().storeNewApplicationAttempt(this);
	}

这里,把RMAppAttemptImpl存储到了全局存储内,这个RMStateStore初始化时候,如果相应配置是false,即不会恢复以前的应用,默认是个NullRMStateStore。

我们接着看,需要注意,这时候RMAppAttemptImpl的状态已经更新为了RMAppAttemptState.ALLOCATED_SAVING,那么,状态机的变化必然会触发相应的操作,但是这里原先状态为RMAppAttemptState.ALLOCATED_SAVING牵涉到的转换有三种,到底是哪个会触发呢?

addTransition(RMAppAttemptState.ALLOCATED_SAVING,
					RMAppAttemptState.ALLOCATED,
					RMAppAttemptEventType.ATTEMPT_NEW_SAVED,
					new AttemptStoredTransition())

			// App could be killed by the client. So need to handle this.
			.addTransition(
					RMAppAttemptState.ALLOCATED_SAVING,
					RMAppAttemptState.FINAL_SAVING,
					RMAppAttemptEventType.KILL,
					new FinalSavingTransition(new BaseFinalTransition(
							RMAppAttemptState.KILLED), RMAppAttemptState.KILLED))
			.addTransition(
					RMAppAttemptState.ALLOCATED_SAVING,
					RMAppAttemptState.FINAL_SAVING,
					RMAppAttemptEventType.CONTAINER_FINISHED,
					new FinalSavingTransition(
							new AMContainerCrashedBeforeRunningTransition(),
							RMAppAttemptState.FAILED))

毫无疑问,我们必须要找到接下来的触发事件的类型,这个事件,其实就在存储的触发内:

public synchronized void storeNewApplicationAttempt(RMAppAttempt appAttempt) {
		Credentials credentials = getCredentialsFromAppAttempt(appAttempt);

		AggregateAppResourceUsage resUsage = appAttempt
				.getRMAppAttemptMetrics().getAggregateAppResourceUsage();
		ApplicationAttemptState attemptState = new ApplicationAttemptState(
				appAttempt.getAppAttemptId(), appAttempt.getMasterContainer(),
				credentials, appAttempt.getStartTime(),
				resUsage.getMemorySeconds(), resUsage.getVcoreSeconds());

		dispatcher.getEventHandler().handle(
				new RMStateStoreAppAttemptEvent(attemptState));
	}

我们仔细看下这个方法:这个dispatcher实际上是RMStateStore内部的调度器,而非RM的全局调度器,这个类比较特殊,同时持有两个调度器:rmDispatcher是全局调度器,dispatcher是自有的调度器:

dispatcher.register(RMStateStoreEventType.class,
				new ForwardingEventHandler());

我们发现,这个事件交给了ForwardingEventHandler来处理,我们看看其处理逻辑:

@Override
		public void handle(RMStateStoreEvent event) {
			handleStoreEvent(event);
		}
// Dispatcher related code
	protected void handleStoreEvent(RMStateStoreEvent event) {
		try {
			this.stateMachine.doTransition(event.getType(), event);
		} catch (InvalidStateTransitonException e) {
			LOG.error("Can't handle this event at current state", e);
		}
	}

那么,我们必须看下这个类的类型了:RMStateStoreAppAttemptEvent:

public class RMStateStoreAppAttemptEvent extends RMStateStoreEvent {
	ApplicationAttemptState attemptState;

	public RMStateStoreAppAttemptEvent(ApplicationAttemptState attemptState) {
		super(RMStateStoreEventType.STORE_APP_ATTEMPT);
		this.attemptState = attemptState;
	}
	public ApplicationAttemptState getAppAttemptState() {
		return attemptState;
	}
}

这里,对于RMStateStore来说,一个STORE_APP_ATTEMPT的事件触发:

addTransition(RMStateStoreState.DEFAULT,
					RMStateStoreState.DEFAULT,
					RMStateStoreEventType.STORE_APP_ATTEMPT,
					new StoreAppAttemptTransition())

我们看下这里的StoreAppAttemptTransition:

ApplicationAttemptState attemptState = ((RMStateStoreAppAttemptEvent) event)
					.getAppAttemptState();
			try {
				ApplicationAttemptStateData attemptStateData = ApplicationAttemptStateData
						.newInstance(attemptState);
				if (LOG.isDebugEnabled()) {
					LOG.debug("Storing info for attempt: "
							+ attemptState.getAttemptId());
				}
				store.storeApplicationAttemptStateInternal(
						attemptState.getAttemptId(), attemptStateData);
				store.notifyApplicationAttempt(new RMAppAttemptEvent(
						attemptState.getAttemptId(),
						RMAppAttemptEventType.ATTEMPT_NEW_SAVED));
			}

摘抄出其中的重点代码,先取出提交的事件状态,然后Notify相关调度器:

private void notifyApplicationAttempt(RMAppAttemptEvent event) {
		rmDispatcher.getEventHandler().handle(event);
	}

这个调度器使用了全局的调度器,把事件放到了rmDispatcher的队列里,我们看看交给谁来处理:

rmDispatcher.register(RMAppAttemptEventType.class,
					new ApplicationAttemptEventDispatcher(rmContext));

这个类的处理方法:

public void handle(RMAppAttemptEvent event) {
			ApplicationAttemptId appAttemptID = event.getApplicationAttemptId();
			ApplicationId appAttemptId = appAttemptID.getApplicationId();
			RMApp rmApp = this.rmContext.getRMApps().get(appAttemptId);
			if (rmApp != null) {
				RMAppAttempt rmAppAttempt = rmApp.getRMAppAttempt(appAttemptID);
				if (rmAppAttempt != null) {
					try {
						rmAppAttempt.handle(event);
					} catch (Throwable t) {
						LOG.error(
								"Error in handling event type "
										+ event.getType()
										+ " for applicationAttempt "
										+ appAttemptId, t);
					}
				}
			}
		}

看看RMAppAttemptImpl内的handle方法,原先的事件类型我们明确了,是RMAppAttemptState_SCHEDULED,理论上来说是这样的,因为最后一步的返回此时还没有完毕:

appAttempt.storeAttempt();
return RMAppAttemptState.ALLOCATED_SAVING;
但实际上并非如此,因为storeAttempt方法其中使用了异步的调度机制,所以,返回来,我们这边的RMContainerImpl的状态,或者其中的状态机的状态已经变成了:ALLOCATED_SAVING,而后面传回来的触发事件类型是:
RMAppAttemptEventType.ATTEMPT_NEW_SAVED

好的,那我们看看会触发什么改变:

.addTransition(RMAppAttemptState.ALLOCATED_SAVING,
					RMAppAttemptState.ALLOCATED,
					RMAppAttemptEventType.ATTEMPT_NEW_SAVED,
					new AttemptStoredTransition())

触发了这个转变,看下:

private static final class AttemptStoredTransition extends BaseTransition {
		@Override
		public void transition(RMAppAttemptImpl appAttempt,
				RMAppAttemptEvent event) {
			appAttempt.launchAttempt();
		}
	}
private void launchAttempt() {
		// Send event to launch the AM Container
		eventHandler.handle(new AMLauncherEvent(AMLauncherEventType.LAUNCH,
				this));
	}

这个eventHandler是全局调度器,而AMLauncherEvent类型的事件,交给了:ApplicationMasterLauncher处理:

@Override
	public synchronized void handle(AMLauncherEvent appEvent) {
		AMLauncherEventType event = appEvent.getType();
		RMAppAttempt application = appEvent.getAppAttempt();
		switch (event) {
		case LAUNCH:
			launch(application);
			break;
		case CLEANUP:
			cleanup(application);
		default:
			break;
		}
	}

这里,我们提交的类型是LAUNCH:

private void launch(RMAppAttempt application) {
		Runnable launcher = createRunnableLauncher(application,
				AMLauncherEventType.LAUNCH);
		masterEvents.add(launcher);
	}

而这个服务启动后,会有一个线程在操作:

private class LauncherThread extends Thread {
		public LauncherThread() {
			super("ApplicationMaster Launcher");
		}
		@Override
		public void run() {
			while (!this.isInterrupted()) {
				Runnable toLaunch;
				try {
					toLaunch = masterEvents.take();
					launcherPool.execute(toLaunch);
				} catch (InterruptedException e) {
					LOG.warn(this.getClass().getName()
							+ " interrupted. Returning.");
					return;
				}
			}
		}
	}

本身来说,ApplicationMasterLauncher是一个抽象服务的继承类,在RM启动后触发了其serviceInit和serviceStart方法,所以这个内部线程会触发,我们的重点在于看看我们提交的时间内部的run方法,即AMLauncher内部的run方法:

case LAUNCH:
			try {
				LOG.info("Launching master" + application.getAppAttemptId());
				launch();
				handler.handle(new RMAppAttemptEvent(application
						.getAppAttemptId(), RMAppAttemptEventType.LAUNCHED));
			} catch (Exception ie) {
				String message = "Error launching "
						+ application.getAppAttemptId() + ". Got exception: "
						+ StringUtils.stringifyException(ie);
				LOG.info(message);
				handler.handle(new RMAppAttemptLaunchFailedEvent(application
						.getAppAttemptId(), message));
			}
			break;

注意,这里的handler,也是全局调度器,try中的方法一个个看,首先是launch方法:

private void launch() throws IOException, YarnException {
		connect();
		ContainerId masterContainerID = masterContainer.getId();
		ApplicationSubmissionContext applicationContext = application
				.getSubmissionContext();
		LOG.info("Setting up container " + masterContainer + " for AM "
				+ application.getAppAttemptId());
		ContainerLaunchContext launchContext = createAMContainerLaunchContext(
				applicationContext, masterContainerID);

		StartContainerRequest scRequest = StartContainerRequest.newInstance(
				launchContext, masterContainer.getContainerToken());
		List<StartContainerRequest> list = new ArrayList<StartContainerRequest>();
		list.add(scRequest);
		StartContainersRequest allRequests = StartContainersRequest
				.newInstance(list);

		StartContainersResponse response = containerMgrProxy
				.startContainers(allRequests);
		if (response.getFailedRequests() != null
				&& response.getFailedRequests().containsKey(masterContainerID)) {
			Throwable t = response.getFailedRequests().get(masterContainerID)
					.deSerialize();
			parseAndThrowException(t);
		} else {
			LOG.info("Done launching container " + masterContainer + " for AM "
					+ application.getAppAttemptId());
		}
	}

其中内容看起来很多,展开来看:

connect:

private void connect() throws IOException {
		ContainerId masterContainerID = masterContainer.getId();
		containerMgrProxy = getContainerMgrProxy(masterContainerID);
	}

要找到我们需要连接的containerId,并且与之建立RPC连接,并且保持这个RPC连接:

ContainerLaunchContext launchContext = createAMContainerLaunchContext(
				applicationContext, masterContainerID);

整合出一个完整的启动上下文:

StartContainerRequest scRequest = StartContainerRequest.newInstance(
				launchContext, masterContainer.getContainerToken());

新建一个请求,我们可以看到,这个请求里包含了启动的上下文,以及需要启动的master的token,必须要这个token,才能合法地启动对应的Container:

List<StartContainerRequest> list = new ArrayList<StartContainerRequest>();
		list.add(scRequest);

这段代码看起来很奇怪,毕竟就只有一个元素,为什么要建立一个list来传输?我觉得,可能是考虑到以后的扩展,毕竟,有可能要在多个container上启动程序:

StartContainersResponse response = containerMgrProxy
				.startContainers(allRequests);

关键在这里,我们前文保持着的RPC连接派上了作用,这个方法的名称也让人激动,前文的铺垫,终于到了可以启动Container的时候了:

这里,我们也能看到,默认情况下,Container启动的参数是1G1核的,所以,我们有时候会根据需求来更改Container的启动参数:

既然是RPC请求,那么,是谁来真正执行这个方法?

答案是根据ContainerId找到对应的NM,在其上的Container上,启动一个Container,牵涉到Container的启动有些繁琐,下文再叙:

阅读更多
想对作者说点什么? 我来说一句

没有更多推荐了,返回首页

不良信息举报

关于Yarn源码的那些事(六)

最多只允许输入30个字

加入CSDN,享受更精准的内容推荐,与500万程序员共同成长!
关闭
关闭