Flink源码篇 No.7-任务提交之JobManager申请资源（per-job on yarn）

第1章 启动JobManager

我们从jobManager启动的方法开始看

org.apache.flink.runtime.jobmaster.JobMaster#startJobExecution

private Acknowledge startJobExecution(JobMasterId newJobMasterId) throws Exception {

	// 验证是否在主线程
	validateRunsInMainThread();

	checkNotNull(newJobMasterId, "The new JobMasterId must not be null.");

	if (Objects.equals(getFencingToken(), newJobMasterId)) {
		log.info("Already started the job execution with JobMasterId {}.", newJobMasterId);

		return Acknowledge.get();
	}

	setNewFencingToken(newJobMasterId);

	// TODO 真正启动jobMaster（jobManager）服务
	startJobMasterServices();

	log.info("Starting execution of job {} ({}) under job master id {}.", jobGraph.getName(), jobGraph.getJobID(), newJobMasterId);

	// TODO 重置并启动调度器
	resetAndStartScheduler();

	return Acknowledge.get();
}

org.apache.flink.runtime.jobmaster.JobMaster#startJobMasterServices

private void startJobMasterServices() throws Exception {
	// TODO 启动jobManager和resourceManager、taskManager之间的心跳服务
	startHeartbeatServices();

	// start the slot pool make sure the slot pool now accepts messages for this leader
	// TODO 启动slotPool
	slotPool.start(getFencingToken(), getAddress(), getMainThreadExecutor());

	//TODO: Remove once the ZooKeeperLeaderRetrieval returns the stored address upon start
	// try to reconnect to previously known leader
	// TODO 重连接到ResourceManager
	reconnectToResourceManager(new FlinkException("Starting JobMaster component."));

	// job is ready to go, try to establish connection with resource manager
	//   - activate leader retrieval for the resource manager
	//   - on notification of the leader, the connection will be established and
	//     the slot pool will start requesting slots
	// TODO 与ResourceManager建立连接，slotPool开始请求资源
	resourceManagerLeaderRetriever.start(new ResourceManagerLeaderListener());
}

第2章 与ResourceManager重新建立连接

org.apache.flink.runtime.jobmaster.JobMaster#reconnectToResourceManager

private void reconnectToResourceManager(Exception cause) {
	// TODO 关闭与resourceManager的连接
	closeResourceManagerConnection(cause);
	// TODO 尝试重新连接resourceManager
	tryConnectToResourceManager();
}

第3章 建立连接与申请资源

org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService#start启动后

org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateListener#stateChanged

启动一个状态监听器，如果状态发生改变则调用下面的方法⬇

org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService#handleStateChange

调用下面的方法⬇

org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService#onReconnectedConnectionState

调用下面的方法⬇

org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService#retrieveLeaderInformationFromZooKeeper

调用下面的方法⬇

org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService#notifyIfNewLeaderAddress

调用下面的方法⬇

org.apache.flink.runtime.leaderretrieval.LeaderRetrievalListener#notifyLeaderAddress

接口的实现类ResourceManagerLeaderListener

org.apache.flink.runtime.jobmaster.JobMaster.ResourceManagerLeaderListener#notifyLeaderAddress

private class ResourceManagerLeaderListener implements LeaderRetrievalListener {

	@Override
	public void notifyLeaderAddress(final String leaderAddress, final UUID leaderSessionID) {
		// TODO 通知ResourceManager Leader
		runAsync(
			() -> notifyOfNewResourceManagerLeader(
				leaderAddress,
				ResourceManagerId.fromUuidOrNull(leaderSessionID)));
	}

	@Override
	public void handleError(final Exception exception) {
		handleJobMasterError(new Exception("Fatal error in the ResourceManager leader service", exception));
	}
}

 org.apache.flink.runtime.jobmaster.JobMaster#notifyOfNewResourceManagerLeader

private void notifyOfNewResourceManagerLeader(final String newResourceManagerAddress, final ResourceManagerId resourceManagerId) {
	resourceManagerAddress = createResourceManagerAddress(newResourceManagerAddress, resourceManagerId);
	// TODO 重连接resourceManager
	reconnectToResourceManager(new FlinkException(String.format("ResourceManager leader changed to new address %s", resourceManagerAddress)));
}

org.apache.flink.runtime.jobmaster.JobMaster#reconnectToResourceManager

private void reconnectToResourceManager(Exception cause) {
	// TODO 关闭与resourceManager的连接
	closeResourceManagerConnection(cause);
	// TODO 尝试重新连接resourceManager
	tryConnectToResourceManager();
}

 org.apache.flink.runtime.jobmaster.JobMaster#tryConnectToResourceManager

private void tryConnectToResourceManager() {
	if (resourceManagerAddress != null) {
		connectToResourceManager();
	}
}

org.apache.flink.runtime.jobmaster.JobMaster#connectToResourceManager

private void connectToResourceManager() {
	assert(resourceManagerAddress != null);
	assert(resourceManagerConnection == null);
	assert(establishedResourceManagerConnection == null);

	log.info("Connecting to ResourceManager {}", resourceManagerAddress);

	// TODO 创建resourceManager连接对象
	resourceManagerConnection = new ResourceManagerConnection(
		log,
		jobGraph.getJobID(),
		resourceId,
		getAddress(),
		getFencingToken(),
		resourceManagerAddress.getAddress(),
		resourceManagerAddress.getResourceManagerId(),
		scheduledExecutorService);

	// TODO 启动
	resourceManagerConnection.start();
}

org.apache.flink.runtime.registration.RegisteredRpcConnection#start

public void start() {
	checkState(!closed, "The RPC connection is already closed");
	checkState(!isConnected() && pendingRegistration == null, "The RPC connection is already started");

	// TODO 创建jobManager向resourceManager的注册
	final RetryingRegistration<F, G, S> newRegistration = createNewRegistration();

	if (REGISTRATION_UPDATER.compareAndSet(this, null, newRegistration)) {
		// TODO 启动注册，如果注册成功执行回调函数
		// org.apache.flink.runtime.jobmaster.JobMaster.ResourceManagerConnection.onRegistrationSuccess
		newRegistration.startRegistration();
	} else {
		// concurrent start operation
		newRegistration.cancel();
	}
}

注册成功执行的是ResourceManagerConnection中的回调函数onRegistrationSuccess

org.apache.flink.runtime.jobmaster.JobMaster.ResourceManagerConnection#onRegistrationSuccess

@Override
protected void onRegistrationSuccess(final JobMasterRegistrationSuccess success) {
	runAsync(() -> {
		// filter out outdated connections
		//noinspection ObjectEquality
		if (this == resourceManagerConnection) {
			// TODO 建立与resourceManager的连接
			establishResourceManagerConnection(success);
		}
	});
}

org.apache.flink.runtime.jobmaster.JobMaster#establishResourceManagerConnection

private void establishResourceManagerConnection(final JobMasterRegistrationSuccess success) {
	final ResourceManagerId resourceManagerId = success.getResourceManagerId();

	// verify the response with current connection
	if (resourceManagerConnection != null
			&& Objects.equals(resourceManagerConnection.getTargetLeaderId(), resourceManagerId)) {

		log.info("JobManager successfully registered at ResourceManager, leader id: {}.", resourceManagerId);

		final ResourceManagerGateway resourceManagerGateway = resourceManagerConnection.getTargetGateway();

		final ResourceID resourceManagerResourceId = success.getResourceManagerResourceId();

		establishedResourceManagerConnection = new EstablishedResourceManagerConnection(
			resourceManagerGateway,
			resourceManagerResourceId);

		// TODO slotPool连接resourceManager
		slotPool.connectToResourceManager(resourceManagerGateway);

		resourceManagerHeartbeatManager.monitorTarget(resourceManagerResourceId, new HeartbeatTarget<Void>() {
			@Override
			public void receiveHeartbeat(ResourceID resourceID, Void payload) {
				resourceManagerGateway.heartbeatFromJobManager(resourceID);
			}

			@Override
			public void requestHeartbeat(ResourceID resourceID, Void payload) {
				// request heartbeat will never be called on the job manager side
			}
		});
	} else {
		log.debug("Ignoring resource manager connection to {} because it's duplicated or outdated.", resourceManagerId);

	}
}

这里可以看到，是由JobManager里的slotPool来与ResourceManager建立连接。

第4章 请求slot

org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl#connectToResourceManager

@Override
public void connectToResourceManager(@Nonnull ResourceManagerGateway resourceManagerGateway) {
	this.resourceManagerGateway = checkNotNull(resourceManagerGateway);

	// work on all slots waiting for this connection
	for (PendingRequest pendingRequest : waitingForResourceManager.values()) {
		// TODO 向resourceManager请求slot
		requestSlotFromResourceManager(resourceManagerGateway, pendingRequest);
	}

	// all sent off
	waitingForResourceManager.clear();
}

org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl#requestSlotFromResourceManager

private void requestSlotFromResourceManager(
		final ResourceManagerGateway resourceManagerGateway,
		final PendingRequest pendingRequest) {

	// ...

	// TODO 请求slot
	CompletableFuture<Acknowledge> rmResponse = resourceManagerGateway.requestSlot(
		jobMasterId,
		new SlotRequest(jobId, allocationId, pendingRequest.getResourceProfile(), jobManagerAddress),
		rpcTimeout);

	// ...
}

org.apache.flink.runtime.resourcemanager.ResourceManager#requestSlot

@Override
public CompletableFuture<Acknowledge> requestSlot(
		JobMasterId jobMasterId,
		SlotRequest slotRequest,
		final Time timeout) {

	JobID jobId = slotRequest.getJobId();
	JobManagerRegistration jobManagerRegistration = jobManagerRegistrations.get(jobId);

	if (null != jobManagerRegistration) {
		if (Objects.equals(jobMasterId, jobManagerRegistration.getJobMasterId())) {
			log.info("Request slot with profile {} for job {} with allocation id {}.",
				slotRequest.getResourceProfile(),
				slotRequest.getJobId(),
				slotRequest.getAllocationId());

			try {
				// TODO slotManager申请资源
				slotManager.registerSlotRequest(slotRequest);
			} catch (ResourceManagerException e) {
				return FutureUtils.completedExceptionally(e);
			}

			return CompletableFuture.completedFuture(Acknowledge.get());
		} else {
			return FutureUtils.completedExceptionally(new ResourceManagerException("The job leader's id " +
				jobManagerRegistration.getJobMasterId() + " does not match the received id " + jobMasterId + '.'));
		}

	} else {
		return FutureUtils.completedExceptionally(new ResourceManagerException("Could not find registered job manager for job " + jobId + '.'));
	}
}

org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl#registerSlotRequest

@Override
public boolean registerSlotRequest(SlotRequest slotRequest) throws ResourceManagerException {
	checkInit();

	// TODO 检查是否是重复的请求
	if (checkDuplicateRequest(slotRequest.getAllocationId())) {
		LOG.debug("Ignoring a duplicate slot request with allocation id {}.", slotRequest.getAllocationId());

		return false;
	} else {
		PendingSlotRequest pendingSlotRequest = new PendingSlotRequest(slotRequest);

		pendingSlotRequests.put(slotRequest.getAllocationId(), pendingSlotRequest);

		try {
			// TODO 申请slot
			internalRequestSlot(pendingSlotRequest);
		} catch (ResourceManagerException e) {
			// requesting the slot failed --> remove pending slot request
			pendingSlotRequests.remove(slotRequest.getAllocationId());

			throw new ResourceManagerException("Could not fulfill slot request " + slotRequest.getAllocationId() + '.', e);
		}

		return true;
	}
}

org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl#internalRequestSlot

private void internalRequestSlot(PendingSlotRequest pendingSlotRequest) throws ResourceManagerException {
	final ResourceProfile resourceProfile = pendingSlotRequest.getResourceProfile();

	// TODO 如果由空闲的slot则直接分配；如果没有，则向外部资源管理器申请
	OptionalConsumer.of(findMatchingSlot(resourceProfile))
		.ifPresent(taskManagerSlot -> allocateSlot(taskManagerSlot, pendingSlotRequest))
		.ifNotPresent(() -> fulfillPendingSlotRequestWithPendingTaskManagerSlot(pendingSlotRequest));
}

org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl#fulfillPendingSlotRequestWithPendingTaskManagerSlot

private void fulfillPendingSlotRequestWithPendingTaskManagerSlot(PendingSlotRequest pendingSlotRequest) throws ResourceManagerException {
	ResourceProfile resourceProfile = pendingSlotRequest.getResourceProfile();
	Optional<PendingTaskManagerSlot> pendingTaskManagerSlotOptional = findFreeMatchingPendingTaskManagerSlot(resourceProfile);

	// TODO 如果没有足够的slot
	if (!pendingTaskManagerSlotOptional.isPresent()) {
		// TODO 申请资源
		pendingTaskManagerSlotOptional = allocateResource(resourceProfile);
	}

	OptionalConsumer.of(pendingTaskManagerSlotOptional)
		.ifPresent(pendingTaskManagerSlot -> assignPendingTaskManagerSlot(pendingSlotRequest, pendingTaskManagerSlot))
		.ifNotPresent(() -> {
			// request can not be fulfilled by any free slot or pending slot that can be allocated,
			// check whether it can be fulfilled by allocated slots
			if (failUnfulfillableRequest && !isFulfillableByRegisteredOrPendingSlots(pendingSlotRequest.getResourceProfile())) {
				throw new UnfulfillableSlotRequestException(pendingSlotRequest.getAllocationId(), pendingSlotRequest.getResourceProfile());
			}
		});
}

org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl#allocateResource

private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
	// ...

	// TODO 分配资源
	if (!resourceActions.allocateResource(defaultWorkerResourceSpec)) {
		// resource cannot be allocated
		return Optional.empty();
	}
	
	// ...
}

org.apache.flink.runtime.resourcemanager.ResourceManager.ResourceActionsImpl#allocateResource

@Override
public boolean allocateResource(WorkerResourceSpec workerResourceSpec) {
	validateRunsInMainThread();
	// TODO 启动新的工作节点
	return startNewWorker(workerResourceSpec);
}

org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager#startNewWorker

@Override
public boolean startNewWorker(WorkerResourceSpec workerResourceSpec) {
	// TODO 请求新的工作节点
	requestNewWorker(workerResourceSpec);
	return true;
}

org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager#requestNewWorker

private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
	// ...

	// TODO 资源管理器发起资源请求
	CompletableFuture<WorkerType> requestResourceFuture = resourceManagerDriver.requestResource(taskExecutorProcessSpec);
	
	// ...
}

 org.apache.flink.runtime.resourcemanager.active.ResourceManagerDriver#requestResource的具体实现类如下：

我们看YarnResourceManagerDriver

org.apache.flink.yarn.YarnResourceManagerDriver#requestResource

@Override
public CompletableFuture<YarnWorkerNode> requestResource(TaskExecutorProcessSpec taskExecutorProcessSpec) {
	checkInitialized();

	final CompletableFuture<YarnWorkerNode> requestResourceFuture = new CompletableFuture<>();

	final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.PriorityAndResource> priorityAndResourceOpt =
		taskExecutorProcessSpecContainerResourcePriorityAdapter.getPriorityAndResource(taskExecutorProcessSpec);

	if (!priorityAndResourceOpt.isPresent()) {
		requestResourceFuture.completeExceptionally(
			new ResourceManagerException(
				String.format("Could not compute the container Resource from the given TaskExecutorProcessSpec %s. " +
						"This usually indicates the requested resource is larger than Yarn's max container resource limit.",
					taskExecutorProcessSpec)));
	} else {
		final Priority priority = priorityAndResourceOpt.get().getPriority();
		final Resource resource = priorityAndResourceOpt.get().getResource();
		// TODO yarn resourceManager 客户端向Yarn请求资源，发起添加Container的请求
		resourceManagerClient.addContainerRequest(getContainerRequest(resource, priority));

		// make sure we transmit the request fast and receive fast news of granted allocations
		resourceManagerClient.setHeartbeatInterval(containerRequestHeartbeatIntervalMillis);

		requestResourceFutures.computeIfAbsent(taskExecutorProcessSpec, ignore -> new LinkedList<>()).add(requestResourceFuture);

		log.info("Requesting new TaskExecutor container with resource {}, priority {}.", taskExecutorProcessSpec, priority);
	}

	return requestResourceFuture;
}

 到这里就开始调用Yarn的resourceManager客户端，向Yarn申请资源了。