flink源码分析1-4--yarnPerJob任务提交流程--部署集群前的准备&提交任务

最新推荐文章于 2023-07-21 16:21:03 发布
最新推荐文章于 2023-07-21 16:21:03 发布
阅读量486 收藏
点赞数
分类专栏： flink源码 文章标签： flink java
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/luyunlong_it/article/details/125246182
版权

flink源码分析1-4–yarnPerJob任务提交流程–部署集群前的准备&提交任务

1、创建flink的ResourceManager
//往yarn集群提交完了任务之后，启动入口类的main方法：YarnJobClusterEntryPoint.java的main方法
	public static void main(String[] args) {
		// startup checks and logging
		EnvironmentInformation.logEnvironmentInfo(LOG, YarnJobClusterEntrypoint.class.getSimpleName(), args);
		SignalHandler.register(LOG);
		JvmShutdownSafeguard.installAsShutdownHook(LOG);

		Map<String, String> env = System.getenv();

		final String workingDirectory = env.get(ApplicationConstants.Environment.PWD.key());
		Preconditions.checkArgument(
			workingDirectory != null,
			"Working directory variable (%s) not set",
			ApplicationConstants.Environment.PWD.key());

		try {
			YarnEntrypointUtils.logYarnEnvironmentInformation(env, LOG);
		} catch (IOException e) {
			LOG.warn("Could not log YARN environment information.", e);
		}

		final Configuration dynamicParameters = ClusterEntrypointUtils.parseParametersOrExit(
			args,
			new DynamicParametersConfigurationParserFactory(),
			YarnJobClusterEntrypoint.class);
		final Configuration configuration = YarnEntrypointUtils.loadConfiguration(workingDirectory, dynamicParameters, env);

		YarnJobClusterEntrypoint yarnJobClusterEntrypoint = new YarnJobClusterEntrypoint(configuration);

		ClusterEntrypoint.runClusterEntrypoint(yarnJobClusterEntrypoint);
	}

/*
 *当前类：org.apache.flink.runtime.entrypoint.ClusterEntrypoint
 *YarnJobClusterEntryPoint.main()->ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint);
 */
	public static void runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint) {

		final String clusterEntrypointName = clusterEntrypoint.getClass().getSimpleName();//只有类名，没有包名
		try {
            //######点进去
			clusterEntrypoint.startCluster();
		} catch (ClusterEntrypointException e) {
			LOG.error(String.format("Could not start cluster entrypoint %s.", clusterEntrypointName), e);
			System.exit(STARTUP_FAILURE_RETURN_CODE);
		}

		clusterEntrypoint.getTerminationFuture().whenComplete((applicationStatus, throwable) -> {
			final int returnCode;

			if (throwable != null) {
				returnCode = RUNTIME_FAILURE_RETURN_CODE;
			} else {
				returnCode = applicationStatus.processExitCode();
			}

			LOG.info("Terminating cluster entrypoint process {} with exit code {}.", clusterEntrypointName, returnCode, throwable);
			System.exit(returnCode);
		});
	}


	public void startCluster() throws ClusterEntrypointException {
		LOG.info("Starting {}.", getClass().getSimpleName());

		try {
			replaceGracefulExitWithHaltIfConfigured(configuration);
			PluginManager pluginManager = PluginUtils.createPluginManagerFromRootFolder(configuration);
			configureFileSystems(configuration, pluginManager);

			SecurityContext securityContext = installSecurityContext(configuration);

			securityContext.runSecured((Callable<Void>) () -> {
				runCluster(configuration, pluginManager);

				return null;
			});
		} catch (Throwable t) {
			final Throwable strippedThrowable = ExceptionUtils.stripException(t, UndeclaredThrowableException.class);

			try {
				// clean up any partial state
				shutDownAsync(
					ApplicationStatus.FAILED,
					ExceptionUtils.stringifyException(strippedThrowable),
					false).get(INITIALIZATION_SHUTDOWN_TIMEOUT.toMilliseconds(), TimeUnit.MILLISECONDS);
			} catch (InterruptedException | ExecutionException | TimeoutException e) {
				strippedThrowable.addSuppressed(e);
			}

			throw new ClusterEntrypointException(
				String.format("Failed to initialize the cluster entrypoint %s.", getClass().getSimpleName()),
				strippedThrowable);
		}
	}


	private void runCluster(Configuration configuration, PluginManager pluginManager) throws Exception {
		synchronized (lock) {

			/*TODO 初始化服务：Rpc相关：包含了后面例如haServices的一大堆*/
			initializeServices(configuration, pluginManager);

			// write host information into configuration
			configuration.setString(JobManagerOptions.ADDRESS, commonRpcService.getAddress());
			configuration.setInteger(JobManagerOptions.PORT, commonRpcService.getPort());

			//拿着配置创建了componentFactory--dispatcherResourceManagerComponentFactory，包含3个工厂：JobRestEndpointFactory、resourceManagerFactory、
			// DefaultDispatcherRunnerFactory（传入jobGraphRetriver，调用了createJobRunner方法，可能是分发jobRunner的工厂）
			final DispatcherResourceManagerComponentFactory dispatcherResourceManagerComponentFactory = createDispatcherResourceManagerComponentFactory(configuration);

			/*TODO 创建和启动 JobManager里的组件：Dispatcher、ResourceManager、JobMaster*/
			clusterComponent = dispatcherResourceManagerComponentFactory.create(
				configuration,
				ioExecutor,
				commonRpcService,
				haServices,
				blobServer,
				heartbeatServices,
				metricRegistry,
				archivedExecutionGraphStore,
				new RpcMetricQueryServiceRetriever(metricRegistry.getMetricQueryServiceRpcService()),
				this);

			clusterComponent.getShutDownFuture().whenComplete(
				(ApplicationStatus applicationStatus, Throwable throwable) -> {
					if (throwable != null) {
						shutDownAsync(
							ApplicationStatus.UNKNOWN,
							ExceptionUtils.stringifyException(throwable),
							false);
					} else {
						// This is the general shutdown path. If a separate more specific shutdown was
						// already triggered, this will do nothing
						shutDownAsync(
							applicationStatus,
							null,
							true);
					}
				});
		}
	}

/*
 *当前类：org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory
 *YarnJobClusterEntryPoint.main()->ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint)->startCluster()->runCluster(Configuration configuration, PluginManager pluginManager)->dispatcherResourceManagerComponentFactory.create();
 */
	@Override
	public DispatcherResourceManagerComponent create(
			Configuration configuration,
			Executor ioExecutor,
			RpcService rpcService,
			HighAvailabilityServices highAvailabilityServices,
			BlobServer blobServer,
			HeartbeatServices heartbeatServices,
			MetricRegistry metricRegistry,
			ArchivedExecutionGraphStore archivedExecutionGraphStore,
			MetricQueryServiceRetriever metricQueryServiceRetriever,
			FatalErrorHandler fatalErrorHandler) throws Exception {

		LeaderRetrievalService dispatcherLeaderRetrievalService = null;
		LeaderRetrievalService resourceManagerRetrievalService = null;
		WebMonitorEndpoint<?> webMonitorEndpoint = null;
		ResourceManager<?> resourceManager = null;
		DispatcherRunner dispatcherRunner = null;

		try {
			dispatcherLeaderRetrievalService = highAvailabilityServices.getDispatcherLeaderRetriever();

			resourceManagerRetrievalService = highAvailabilityServices.getResourceManagerLeaderRetriever();

			//传入rpcService创建dispatcherGatewayRetriever和resourceManagerGatewayRetriever，为了之后创建webMonitor的终端和MetricFetcher准备的，
			// 在后面还会传入dispatcherLeaderRetrievalService和resourceManagerRetrievalService的start方法
			final LeaderGatewayRetriever<DispatcherGateway> dispatcherGatewayRetriever = new RpcGatewayRetriever<>(
				rpcService,
				DispatcherGateway.class,
				DispatcherId::fromUuid,
				new ExponentialBackoffRetryStrategy(12, Duration.ofMillis(10), Duration.ofMillis(50)));

			final LeaderGatewayRetriever<ResourceManagerGateway> resourceManagerGatewayRetriever = new RpcGatewayRetriever<>(
				rpcService,
				ResourceManagerGateway.class,
				ResourceManagerId::fromUuid,
				new ExponentialBackoffRetryStrategy(12, Duration.ofMillis(10), Duration.ofMillis(50)));

			//webMonitor的executor和metric的一些东西
			final ScheduledExecutorService executor = WebMonitorEndpoint.createExecutorService(
				configuration.getInteger(RestOptions.SERVER_NUM_THREADS),
				configuration.getInteger(RestOptions.SERVER_THREAD_PRIORITY),
				"DispatcherRestEndpoint");

			final long updateInterval = configuration.getLong(MetricOptions.METRIC_FETCHER_UPDATE_INTERVAL);
			final MetricFetcher metricFetcher = updateInterval == 0
				? VoidMetricFetcher.INSTANCE
				: MetricFetcherImpl.fromConfiguration(
					configuration,
					metricQueryServiceRetriever,
					dispatcherGatewayRetriever,
					executor);

			webMonitorEndpoint = restEndpointFactory.createRestEndpoint(
				configuration,
				dispatcherGatewayRetriever,
				resourceManagerGatewayRetriever,
				blobServer,
				executor,
				metricFetcher,
				highAvailabilityServices.getClusterRestEndpointLeaderElectionService(),
				fatalErrorHandler);

			log.debug("Starting Dispatcher REST endpoint.");
			webMonitorEndpoint.start();

			final String hostname = RpcUtils.getHostname(rpcService);

			/*TODO 创建 ResourceManager：Yarn模式的 ResourceManager*/
			resourceManager = resourceManagerFactory.createResourceManager(
				configuration,
				ResourceID.generate(),
				rpcService,
				highAvailabilityServices,
				heartbeatServices,
				fatalErrorHandler,
				new ClusterInformation(hostname, blobServer.getPort()),
				webMonitorEndpoint.getRestBaseUrl(),
				metricRegistry,
				hostname,
				ioExecutor);

			final HistoryServerArchivist historyServerArchivist = HistoryServerArchivist.createHistoryServerArchivist(configuration, webMonitorEndpoint, ioExecutor);

			final PartialDispatcherServices partialDispatcherServices = new PartialDispatcherServices(
				configuration,
				highAvailabilityServices,
				resourceManagerGatewayRetriever,
				blobServer,
				heartbeatServices,
				() -> MetricUtils.instantiateJobManagerMetricGroup(metricRegistry, hostname),
				archivedExecutionGraphStore,
				fatalErrorHandler,
				historyServerArchivist,
				metricRegistry.getMetricQueryServiceGatewayRpcAddress(),
				ioExecutor);

			log.debug("Starting Dispatcher.");
			/*TODO 创建和启动 Dispatcher => dispatcher会创建和启动JobMaster*/
			dispatcherRunner = dispatcherRunnerFactory.createDispatcherRunner(
				highAvailabilityServices.getDispatcherLeaderElectionService(),
				fatalErrorHandler,
				new HaServicesJobGraphStoreFactory(highAvailabilityServices),
				ioExecutor,
				rpcService,
				partialDispatcherServices);

			log.debug("Starting ResourceManager.");
			/*TODO 启动 ResourceManager*/
			resourceManager.start();

			resourceManagerRetrievalService.start(resourceManagerGatewayRetriever);
			dispatcherLeaderRetrievalService.start(dispatcherGatewayRetriever);

			return new DispatcherResourceManagerComponent(
				dispatcherRunner,
				DefaultResourceManagerService.createFor(resourceManager),
				dispatcherLeaderRetrievalService,
				resourceManagerRetrievalService,
				webMonitorEndpoint,
				fatalErrorHandler);

		} catch (Exception exception) {
			// clean up all started components
			if (dispatcherLeaderRetrievalService != null) {
				try {
					dispatcherLeaderRetrievalService.stop();
				} catch (Exception e) {
					exception = ExceptionUtils.firstOrSuppressed(e, exception);
				}
			}

			if (resourceManagerRetrievalService != null) {
				try {
					resourceManagerRetrievalService.stop();
				} catch (Exception e) {
					exception = ExceptionUtils.firstOrSuppressed(e, exception);
				}
			}

			final Collection<CompletableFuture<Void>> terminationFutures = new ArrayList<>(3);

			if (webMonitorEndpoint != null) {
				terminationFutures.add(webMonitorEndpoint.closeAsync());
			}

			if (resourceManager != null) {
				terminationFutures.add(resourceManager.closeAsync());
			}

			if (dispatcherRunner != null) {
				terminationFutures.add(dispatcherRunner.closeAsync());
			}

			final FutureUtils.ConjunctFuture<Void> terminationFuture = FutureUtils.completeAll(terminationFutures);

			try {
				terminationFuture.get();
			} catch (Exception e) {
				exception = ExceptionUtils.firstOrSuppressed(e, exception);
			}

			throw new FlinkException("Could not create the DispatcherResourceManagerComponent.", exception);
		}
	}
//调用到这里一共干了几件事：创建了一个DispatcherResourceManagerComponent，里面有dispatcherRunner，ResourceManagerService（封装了resourceManager），dispatcherLeaderRetrievalService，resourceManagerRetrievalService，webMonitorEndpoint。返回clusterComponent之前，dispatcherRunner&dispatcher，resourceManager，dispatcherLeaderRetrievalService，resourceManagerRetrievalService，webMonitorEndpoint均已创建并启动完毕。

这里还有一段创建RM的过程，暂时空着

2、创建Dispatcher（创建的时候里面就有启动的动作了，在此之前RM已经创建完了，只是没有启动）
/*
 *当前类：org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory
 *YarnJobClusterEntryPoint.main()->ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint)->startCluster()->runCluster(Configuration configuration, PluginManager pluginManager)->dispatcherResourceManagerComponentFactory.create();
 */
	@Override
	public DispatcherResourceManagerComponent create(
			Configuration configuration,
			Executor ioExecutor,
			RpcService rpcService,
			HighAvailabilityServices highAvailabilityServices,
			BlobServer blobServer,
			HeartbeatServices heartbeatServices,
			MetricRegistry metricRegistry,
			ArchivedExecutionGraphStore archivedExecutionGraphStore,
			MetricQueryServiceRetriever metricQueryServiceRetriever,
			FatalErrorHandler fatalErrorHandler) throws Exception {

		LeaderRetrievalService dispatcherLeaderRetrievalService = null;
		LeaderRetrievalService resourceManagerRetrievalService = null;
		WebMonitorEndpoint<?> webMonitorEndpoint = null;
		ResourceManager<?> resourceManager = null;
		DispatcherRunner dispatcherRunner = null;

/。。。。。。。。。。中间省略一大坨。。。。。。。。。。。
			/*TODO 创建和启动 Dispatcher => dispatcher会创建和启动JobMaster*/
			dispatcherRunner = dispatcherRunnerFactory.createDispatcherRunner(
				highAvailabilityServices.getDispatcherLeaderElectionService(),
				fatalErrorHandler,
				new HaServicesJobGraphStoreFactory(highAvailabilityServices),
				ioExecutor,
				rpcService,
				partialDispatcherServices);
/。。。。。。。。。。中间省略一大坨。。。。。。。。。。。
	}

/*
 *当前类：org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunnerFactory
 *YarnJobClusterEntryPoint.main()->ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint)->startCluster()->runCluster(Configuration configuration, PluginManager pluginManager)->dispatcherResourceManagerComponentFactory.create()->dispatcherRunnerFactory.createDispatcherRunner();
 */
	@Override
	public DispatcherRunner createDispatcherRunner(
			LeaderElectionService leaderElectionService,
			FatalErrorHandler fatalErrorHandler,
			JobGraphStoreFactory jobGraphStoreFactory,
			Executor ioExecutor,
			RpcService rpcService,
			PartialDispatcherServices partialDispatcherServices) throws Exception {

		//创建了一个dispatcher领导进程工厂对象，参与了DispatcherRunner的创建
		final DispatcherLeaderProcessFactory dispatcherLeaderProcessFactory = dispatcherLeaderProcessFactoryFactory.createFactory(
			jobGraphStoreFactory,
			ioExecutor,
			rpcService,
			partialDispatcherServices,
			fatalErrorHandler);

		return DefaultDispatcherRunner.create(
			leaderElectionService,
			fatalErrorHandler,
			dispatcherLeaderProcessFactory);
	}

/*
 *当前类：org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner
 *YarnJobClusterEntryPoint.main()->ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint clusterEntrypoint)->startCluster()->runCluster(Configuration configuration, PluginManager pluginManager)->dispatcherResourceManagerComponentFactory.create()->dispatcherRunnerFactory.createDispatcherRunner()->create();
 */
	public static DispatcherRunner create(
			LeaderElectionService leaderElectionService,
			FatalErrorHandler fatalErrorHandler,
			DispatcherLeaderProcessFactory dispatcherLeaderProcessFactory) throws Exception {
		final DefaultDispatcherRunner dispatcherRunner = new DefaultDispatcherRunner(
			leaderElectionService,
			fatalErrorHandler,
			dispatcherLeaderProcessFactory);
		return DispatcherRunnerLeaderElectionLifecycleManager.createFor(dispatcherRunner, leaderElectionService);
	}

/*DispatcherRunnerLeaderElectionLifecycleManager
*/
	public static <T extends DispatcherRunner & LeaderContender> DispatcherRunner createFor(T dispatcherRunner, LeaderElectionService leaderElectionService) throws Exception {
		return new DispatcherRunnerLeaderElectionLifecycleManager<>(dispatcherRunner, leaderElectionService);
	}


	private DispatcherRunnerLeaderElectionLifecycleManager(T dispatcherRunner, LeaderElectionService leaderElectionService) throws Exception {
		this.dispatcherRunner = dispatcherRunner;
		this.leaderElectionService = leaderElectionService;

		leaderElectionService.start(dispatcherRunner);
	}

/*StandaloneLeaderElectionService
*/
	@Override
	public void start(LeaderContender newContender) throws Exception {
		if (contender != null) {
			// Service was already started
			throw new IllegalArgumentException("Leader election service cannot be started multiple times.");
		}

		contender = Preconditions.checkNotNull(newContender);

		// directly grant leadership to the given contender
		contender.grantLeadership(HighAvailabilityServices.DEFAULT_LEADER_ID);
	}

/*DefaultDispatcherRunner
*/
	@Override
	public void grantLeadership(UUID leaderSessionID) {
		runActionIfRunning(() -> startNewDispatcherLeaderProcess(leaderSessionID));
	}

	private void startNewDispatcherLeaderProcess(UUID leaderSessionID) {
		stopDispatcherLeaderProcess();

		dispatcherLeaderProcess = createNewDispatcherLeaderProcess(leaderSessionID);

		final DispatcherLeaderProcess newDispatcherLeaderProcess = dispatcherLeaderProcess;
		FutureUtils.assertNoException(
			previousDispatcherLeaderProcessTerminationFuture.thenRun(newDispatcherLeaderProcess::start));
	}

/*AbstractDispatcherLeaderProcess
*/
	@Override
	public final void start() {
		runIfStateIs(
			State.CREATED,
			this::startInternal);
	}

	private void startInternal() {
		log.info("Start {}.", getClass().getSimpleName());
		state = State.RUNNING;
		onStart();
	}

/*JobDispatcherLeaderProcess
*/
	@Override
	protected void onStart() {
		final DispatcherGatewayService dispatcherService = dispatcherGatewayServiceFactory.create(
			DispatcherId.fromUuid(getLeaderSessionId()),
			Collections.singleton(jobGraph),
			ThrowingJobGraphWriter.INSTANCE);

		completeDispatcherSetup(dispatcherService);
	}

/*DefaultDispatcherGatewayServiceFactory
*/
	@Override
	public AbstractDispatcherLeaderProcess.DispatcherGatewayService create(
			DispatcherId fencingToken,
			Collection<JobGraph> recoveredJobs,
			JobGraphWriter jobGraphWriter) {

		final Dispatcher dispatcher;
		try {
			/*TODO 创建Dispatcher*/
			dispatcher = dispatcherFactory.createDispatcher(
				rpcService,
				fencingToken,
				recoveredJobs,
				(dispatcherGateway, scheduledExecutor, errorHandler) -> new NoOpDispatcherBootstrap(),
				PartialDispatcherServicesWithJobGraphStore.from(partialDispatcherServices, jobGraphWriter));
		} catch (Exception e) {
			throw new FlinkRuntimeException("Could not create the Dispatcher rpc endpoint.", e);
		}

		/*TODO 启动 Dispatcher，接着看 onStart()，要看Dispatcher的onStart()*/
		dispatcher.start();
3、Dispatcher启动JobMaster
/*Dispatcher
*/
	@Override
	public void onStart() throws Exception {
		try {
			/*TODO 启动 dispatcher服务*/
			startDispatcherServices();
		} catch (Throwable t) {
			final DispatcherException exception = new DispatcherException(String.format("Could not start the Dispatcher %s", getAddress()), t);
			onFatalError(exception);
			throw exception;
		}
/此方法到此只有一半，还有一半在后面///
        

	private void startDispatcherServices() throws Exception {
		try {
			registerDispatcherMetrics(jobManagerMetricGroup);
		} catch (Exception e) {
			handleStartDispatcherServicesException(e);
		}
	}
        
	
    private void registerDispatcherMetrics(MetricGroup jobManagerMetricGroup) {
		jobManagerMetricGroup.gauge(MetricNames.NUM_RUNNING_JOBS,
			() -> (long) runningJobs.size());
	}
        
        
//onStart方法剩下的一半在这里//
		/*TODO 启动JobMaster*/
		startRecoveredJobs();
		this.dispatcherBootstrap = this.dispatcherBootstrapFactory.create(
				getSelfGateway(DispatcherGateway.class),
				this.getRpcService().getScheduledExecutor() ,
				this::onFatalError);
	}


	private void startRecoveredJobs() {
		for (JobGraph recoveredJob : recoveredJobs) {
			runRecoveredJob(recoveredJob);
		}
		recoveredJobs.clear();
	}


	private void runRecoveredJob(final JobGraph recoveredJob) {
		checkNotNull(recoveredJob);
		try {
			runJob(recoveredJob, ExecutionType.RECOVERY);
		} catch (Throwable throwable) {
			onFatalError(new DispatcherException(String.format("Could not start recovered job %s.", recoveredJob.getJobID()), throwable));
		}
	}


	private void runJob(JobGraph jobGraph, ExecutionType executionType) {
		Preconditions.checkState(!runningJobs.containsKey(jobGraph.getJobID()));
		long initializationTimestamp = System.currentTimeMillis();
		CompletableFuture<JobManagerRunner> jobManagerRunnerFuture = createJobManagerRunner(jobGraph, initializationTimestamp);

		DispatcherJob dispatcherJob = DispatcherJob.createFor(
				jobManagerRunnerFuture,
				jobGraph.getJobID(),
				jobGraph.getName(),
				initializationTimestamp);
		runningJobs.put(jobGraph.getJobID(), dispatcherJob);

		final JobID jobId = jobGraph.getJobID();

		final CompletableFuture<CleanupJobState> cleanupJobStateFuture = dispatcherJob.getResultFuture().handleAsync(
			(dispatcherJobResult, throwable) -> {
				Preconditions.checkState(runningJobs.get(jobId) == dispatcherJob, "The job entry in runningJobs must be bound to the lifetime of the DispatcherJob.");

				if (dispatcherJobResult != null) {
					return handleDispatcherJobResult(jobId, dispatcherJobResult, executionType);
				} else {
					return dispatcherJobFailed(jobId, throwable);
				}
			}, getMainThreadExecutor());

		final CompletableFuture<Void> jobTerminationFuture = cleanupJobStateFuture
			.thenApply(cleanupJobState -> removeJob(jobId, cleanupJobState))
			.thenCompose(Function.identity());

		FutureUtils.assertNoException(jobTerminationFuture);
		registerDispatcherJobTerminationFuture(jobId, jobTerminationFuture);
	}


	CompletableFuture<JobManagerRunner> createJobManagerRunner(JobGraph jobGraph, long initializationTimestamp) {
		final RpcService rpcService = getRpcService();
		return CompletableFuture.supplyAsync(
			() -> {
				try {
					/*TODO 创建JobMaster */
					JobManagerRunner runner = jobManagerRunnerFactory.createJobManagerRunner(
						jobGraph,
						configuration,
						rpcService,
						highAvailabilityServices,
						heartbeatServices,
						jobManagerSharedServices,
						new DefaultJobManagerJobMetricGroupFactory(jobManagerMetricGroup),
						fatalErrorHandler,
						initializationTimestamp);
					/*TODO 启动JobMaster*/
					runner.start();
					return runner;
				} catch (Exception e) {
					throw new CompletionException(new JobInitializationException(jobGraph.getJobID(), "Could not instantiate JobManager.", e));
				}
			},
			ioExecutor); // do not use main thread executor. Otherwise, Dispatcher is blocked on JobManager creation
	}


//在第一步创建jobManagerRunner的时候，层层调用，最后会在底层创建一个JobMaster对象，封装赋值到JobManagerRunner的成员变量里面去，然后再启动这个Runner。

/*JobManagerRunnerImpl
*/
	public void start() throws Exception {
		try {
			leaderElectionService.start(this);
		} catch (Exception e) {
			log.error("Could not start the JobManager because the leader election service did not start.", e);
			throw new Exception("Could not start the leader election service.", e);
		}
	}


	@Override
	public void grantLeadership(final UUID leaderSessionID) {
		synchronized (lock) {
			if (shutdown) {
				log.debug("JobManagerRunner cannot be granted leadership because it is already shut down.");
				return;
			}

			leadershipOperation = leadershipOperation.thenCompose(
				(ignored) -> {
					synchronized (lock) {
						return verifyJobSchedulingStatusAndStartJobManager(leaderSessionID);
					}
				});

			handleException(leadershipOperation, "Could not start the job manager.");
		}
	}

	private CompletableFuture<Void> verifyJobSchedulingStatusAndStartJobManager(UUID leaderSessionId) {
		final CompletableFuture<JobSchedulingStatus> jobSchedulingStatusFuture = getJobSchedulingStatus();

		return jobSchedulingStatusFuture.thenCompose(
			jobSchedulingStatus -> {
				if (jobSchedulingStatus == JobSchedulingStatus.DONE) {
					return jobAlreadyDone();
				} else {
					return startJobMaster(leaderSessionId);
				}
			});
	}



/*JobMaster
*/
	public CompletableFuture<Acknowledge> start(final JobMasterId newJobMasterId) throws Exception {
		// make sure we receive RPC and async calls
		start();
		/*TODO 异步不阻塞 调用*/
		return callAsyncWithoutFencing(() -> startJobExecution(newJobMasterId), RpcUtils.INF_TIMEOUT);
	}


	private Acknowledge startJobExecution(JobMasterId newJobMasterId) throws Exception {

		validateRunsInMainThread();

		checkNotNull(newJobMasterId, "The new JobMasterId must not be null.");

		if (Objects.equals(getFencingToken(), newJobMasterId)) {
			log.info("Already started the job execution with JobMasterId {}.", newJobMasterId);

			return Acknowledge.get();
		}

		setNewFencingToken(newJobMasterId);

		/*TODO 真正启动JobMaster服务*/
		startJobMasterServices();

		log.info("Starting execution of job {} ({}) under job master id {}.", jobGraph.getName(), jobGraph.getJobID(), newJobMasterId);

		/*TODO 重置和启动调度器*/
		resetAndStartScheduler();

		return Acknowledge.get();
	}
4、ResourceManager初始化和启动SlotManager
/*ResourceManager
*/
//dispatcher创建启动完毕后下一步是RM的启动，调用start方法，直接看到RM的onstart方法就可以
public final void onStart() throws Exception {
... ...
startResourceManagerServices();
... ...
}


private void startResourceManagerServices() throws Exception {
try {
	leaderElectionService = highAvailabilityServices.getResourceManagerLeaderElectionService();
	initialize();
	leaderElectionService.start(this);
	jobLeaderIdService.start(new JobLeaderIdActionsImpl());
	registerTaskExecutorMetrics();
} catch (Exception e) {
	handleStartResourceManagerServicesException(e);
	}
}


/*ActiveResourceManager
*/
	@Override
	protected void initialize() throws ResourceManagerException {
		try {
			resourceManagerDriver.initialize(
					this,
					new GatewayMainThreadExecutor(),
					ioExecutor);
		} catch (Exception e) {
			throw new ResourceManagerException("Cannot initialize resource provider.", e);
		}
	}

/*AbstractResourceManagerDriver
*/
	@Override
	public final void initialize(
			ResourceEventHandler<WorkerType> resourceEventHandler,
			ScheduledExecutor mainThreadExecutor,
			Executor ioExecutor) throws Exception {
		this.resourceEventHandler = Preconditions.checkNotNull(resourceEventHandler);
		this.mainThreadExecutor = Preconditions.checkNotNull(mainThreadExecutor);
		this.ioExecutor = Preconditions.checkNotNull(ioExecutor);

		initializeInternal();
	}

/*YarnResourceManagerDriver
*/
	@Override
	protected void initializeInternal() throws Exception {
		final YarnContainerEventHandler yarnContainerEventHandler = new YarnContainerEventHandler();
		try {
			/*TODO 创建Yarn的ResourceManager的客户端，并且初始化和启动*/
			resourceManagerClient = yarnResourceManagerClientFactory.createResourceManagerClient(
				yarnHeartbeatIntervalMillis,
				yarnContainerEventHandler);
			resourceManagerClient.init(yarnConfig);
			resourceManagerClient.start();

			//注册AppMaster
			final RegisterApplicationMasterResponse registerApplicationMasterResponse = registerApplicationMaster();
			//获取AppMaster的container
			getContainersFromPreviousAttempts(registerApplicationMasterResponse);
			taskExecutorProcessSpecContainerResourcePriorityAdapter =
				new TaskExecutorProcessSpecContainerResourcePriorityAdapter(
					registerApplicationMasterResponse.getMaximumResourceCapability(),
					ExternalResourceUtils.getExternalResources(flinkConfig, YarnConfigOptions.EXTERNAL_RESOURCE_YARN_CONFIG_KEY_SUFFIX));
		} catch (Exception e) {
			throw new ResourceManagerException("Could not start resource manager client.", e);
		}

		/*TODO 创建yarn的 NodeManager的客户端，并且初始化和启动*/
		nodeManagerClient = yarnNodeManagerClientFactory.createNodeManagerClient(yarnContainerEventHandler);
		nodeManagerClient.init(yarnConfig);
		nodeManagerClient.start();
	}
//回到leaderElectionService.start(this);

/*ResourceManager
*/
	@Override
	public void grantLeadership(final UUID newLeaderSessionID) {
		final CompletableFuture<Boolean> acceptLeadershipFuture = clearStateFuture
			.thenComposeAsync((ignored) -> tryAcceptLeadership(newLeaderSessionID), getUnfencedMainThreadExecutor());

		final CompletableFuture<Void> confirmationFuture = acceptLeadershipFuture.thenAcceptAsync(
			(acceptLeadership) -> {
				if (acceptLeadership) {
					// confirming the leader session ID might be blocking,
					leaderElectionService.confirmLeadership(newLeaderSessionID, getAddress());
				}
			},
			ioExecutor);

		confirmationFuture.whenComplete(
			(Void ignored, Throwable throwable) -> {
				if (throwable != null) {
					onFatalError(ExceptionUtils.stripCompletionException(throwable));
				}
			});
	}


	private CompletableFuture<Boolean> tryAcceptLeadership(final UUID newLeaderSessionID) {
		if (leaderElectionService.hasLeadership(newLeaderSessionID)) {
			final ResourceManagerId newResourceManagerId = ResourceManagerId.fromUuid(newLeaderSessionID);

			log.info("ResourceManager {} was granted leadership with fencing token {}", getAddress(), newResourceManagerId);

			// clear the state if we've been the leader before
			if (getFencingToken() != null) {
				clearStateInternal();
			}

			setFencingToken(newResourceManagerId);

			startServicesOnLeadership();

			return prepareLeadershipAsync().thenApply(ignored -> true);
		} else {
			return CompletableFuture.completedFuture(false);
		}
	}


	private void startServicesOnLeadership() {
		/*TODO 启动心跳服务：TaskManager、JobMaster*/
		startHeartbeatServices();

		/*TODO 启动slotManager*/
		slotManager.start(getFencingToken(), getMainThreadExecutor(), new ResourceActionsImpl());

		onLeadership();
	}


//启动心跳服务
	private void startHeartbeatServices() {
		taskManagerHeartbeatManager = heartbeatServices.createHeartbeatManagerSender(
			resourceId,
			new TaskManagerHeartbeatListener(),
			getMainThreadExecutor(),
			log);

		jobManagerHeartbeatManager = heartbeatServices.createHeartbeatManagerSender(
			resourceId,
			new JobManagerHeartbeatListener(),
			getMainThreadExecutor(),
			log);
	}


//启动slotManager
/*SlotManagerImpl
*/
	@Override
	public void start(ResourceManagerId newResourceManagerId, Executor newMainThreadExecutor, ResourceActions newResourceActions) {
		LOG.info("Starting the SlotManager.");

		this.resourceManagerId = Preconditions.checkNotNull(newResourceManagerId);
		mainThreadExecutor = Preconditions.checkNotNull(newMainThreadExecutor);
		resourceActions = Preconditions.checkNotNull(newResourceActions);

		started = true;

		taskManagerTimeoutsAndRedundancyCheck = scheduledExecutor.scheduleWithFixedDelay(
			() -> mainThreadExecutor.execute(
				() -> checkTaskManagerTimeoutsAndRedundancy()),
			0L,
			taskManagerTimeout.toMilliseconds(),
			TimeUnit.MILLISECONDS);

		slotRequestTimeoutCheck = scheduledExecutor.scheduleWithFixedDelay(
			() -> mainThreadExecutor.execute(
				() -> checkSlotRequestTimeouts()),
			0L,
			slotRequestTimeout.toMilliseconds(),
			TimeUnit.MILLISECONDS);

		registerSlotManagerMetrics();
	}


	@VisibleForTesting
	void checkTaskManagerTimeoutsAndRedundancy() {
		if (!taskManagerRegistrations.isEmpty()) {
			long currentTime = System.currentTimeMillis();

			ArrayList<TaskManagerRegistration> timedOutTaskManagers = new ArrayList<>(taskManagerRegistrations.size());

			// first retrieve the timed out TaskManagers
			for (TaskManagerRegistration taskManagerRegistration : taskManagerRegistrations.values()) {
				if (currentTime - taskManagerRegistration.getIdleSince() >= taskManagerTimeout.toMilliseconds()) {
					// we collect the instance ids first in order to avoid concurrent modifications by the
					// ResourceActions.releaseResource call
					timedOutTaskManagers.add(taskManagerRegistration);
				}
			}

			int slotsDiff = redundantTaskManagerNum * numSlotsPerWorker - freeSlots.size();
			if (freeSlots.size() == slots.size()) {
				// No need to keep redundant taskManagers if no job is running.
                // 如果没有 job 在运行，释放 taskmanager
				releaseTaskExecutors(timedOutTaskManagers, timedOutTaskManagers.size());
			} else if (slotsDiff > 0) {
				// Keep enough redundant taskManagers from time to time.
                // 保证随时有足够的 taskmanager
				int requiredTaskManagers = MathUtils.divideRoundUp(slotsDiff, numSlotsPerWorker);
				allocateRedundantTaskManagers(requiredTaskManagers);
			} else {
				// second we trigger the release resource callback which can decide upon the resource release
				int maxReleaseNum = (-slotsDiff) / numSlotsPerWorker;
				releaseTaskExecutors(timedOutTaskManagers, Math.min(maxReleaseNum, timedOutTaskManagers.size()));
			}
		}
	}
5、SlotPool向RM申请资源
//JobMaster
	private Acknowledge startJobExecution(JobMasterId newJobMasterId) throws Exception {
	... ...
		// 启动 JobMaster
		startJobMasterServices();
		log.info("Starting execution of job {} ({}) under job master id {}.", jobGraph.getName(), 
		jobGraph.getJobID(), newJobMasterId);
		// 重置开始调度
		resetAndStartScheduler();
	... ...
	}


	private void startJobMasterServices() throws Exception {
		/*TODO 启动心跳服务：taskmanager、resourcemanager*/
		startHeartbeatServices();

		// start the slot pool make sure the slot pool now accepts messages for this leader
		/*TODO 启动 slotpool*/
		slotPool.start(getFencingToken(), getAddress(), getMainThreadExecutor());

		//TODO: Remove once the ZooKeeperLeaderRetrieval returns the stored address upon start
		// try to reconnect to previously known leader
        // 重连接到之前已知的 ResourceManager
		reconnectToResourceManager(new FlinkException("Starting JobMaster component."));

		// job is ready to go, try to establish connection with resource manager
		//   - activate leader retrieval for the resource manager
		//   - on notification of the leader, the connection will be established and
		//     the slot pool will start requesting slots
		/*TODO 与ResourceManager建立连接，slotpool开始请求资源*/
        /*这一步是领导寻回的动作，用来寻回RM的领导，传进去一个“RM领导监听器”，用来监听当前RM的领导是谁并通知到
        *当前的JobMaster，并重连接到当前的领导RM
        */
		resourceManagerLeaderRetriever.start(new ResourceManagerLeaderListener());
	}

//1、启动心跳服务
	private void startHeartbeatServices() {
		taskManagerHeartbeatManager = heartbeatServices.createHeartbeatManagerSender(
			resourceId,
			new TaskManagerHeartbeatListener(),
			getMainThreadExecutor(),
			log);

		resourceManagerHeartbeatManager = heartbeatServices.createHeartbeatManager(
			resourceId,
			new ResourceManagerHeartbeatListener(),
			getMainThreadExecutor(),
			log);
	}


//2、启动slotPool
	public void start(
		@Nonnull JobMasterId jobMasterId,
		@Nonnull String newJobManagerAddress,
		@Nonnull ComponentMainThreadExecutor componentMainThreadExecutor) throws Exception {

		this.jobMasterId = jobMasterId;
		this.jobManagerAddress = newJobManagerAddress;
		this.componentMainThreadExecutor = componentMainThreadExecutor;

		scheduleRunAsync(this::checkIdleSlot, idleSlotTimeout);
		scheduleRunAsync(this::checkBatchSlotTimeout, batchSlotTimeout);

		if (log.isDebugEnabled()) {
			scheduleRunAsync(this::scheduledLogStatus, STATUS_LOG_INTERVAL_MS, TimeUnit.MILLISECONDS);
		}
	}


//3、连接到之前已知的ResourceManager
	private void reconnectToResourceManager(Exception cause) {
		closeResourceManagerConnection(cause);
		tryConnectToResourceManager();
	}

	private void tryConnectToResourceManager() {
		if (resourceManagerAddress != null) {
			connectToResourceManager();
		}
	}

	private void connectToResourceManager() {
		assert(resourceManagerAddress != null);
		assert(resourceManagerConnection == null);
		assert(establishedResourceManagerConnection == null);

		log.info("Connecting to ResourceManager {}", resourceManagerAddress);

		resourceManagerConnection = new ResourceManagerConnection(
			log,
			jobGraph.getJobID(),
			resourceId,
			getAddress(),
			getFencingToken(),
			resourceManagerAddress.getAddress(),
			resourceManagerAddress.getResourceManagerId(),
			scheduledExecutorService);

		resourceManagerConnection.start();
	}


	//RegisteredRpcConnection
	public void start() {
		checkState(!closed, "The RPC connection is already closed");
		checkState(!isConnected() && pendingRegistration == null, "The RPC connection is already started");

		/*TODO 创建注册对象*/
		final RetryingRegistration<F, G, S> newRegistration = createNewRegistration();

		if (REGISTRATION_UPDATER.compareAndSet(this, null, newRegistration)) {
			/*TODO 开始注册，注册成功之后，调用 onRegistrationSuccess()*/
			newRegistration.startRegistration();
		} else {
			// concurrent start operation
			newRegistration.cancel();
		}
	}


//4、与ResourceManager建立连接，slotpool开始请求资源
	经过下面层层调用：
	resourceManagerLeaderRetriever.start(new ResourceManagerLeaderListener());
		-> notifyOfNewResourceManagerLeader() 
			-> notifyOfNewResourceManagerLeader() 
				-> reconnectToResourceManager()
					-> tryConnectToResourceManager()
						-> connectToResourceManager()
    
    private void connectToResourceManager() {
		... ...
		resourceManagerConnection = new ResourceManagerConnection(
        log,
		jobGraph.getJobID(),
		resourceId,
		getAddress(),
		getFencingToken(),
		resourceManagerAddress.getAddress(),
		resourceManagerAddress.getResourceManagerId(),
		scheduledExecutorService);
    
    
		resourceManagerConnection.start();
    
	}


	//RegisteredRpcConnection.java
	public void start() {
		... ...
		final RetryingRegistration<F, G, S> newRegistration = createNewRegistration();
		if (REGISTRATION_UPDATER.compareAndSet(this, null, newRegistration)) {
			newRegistration.startRegistration();
		} else {
			// concurrent start operation
			newRegistration.cancel();
		}
	}


	private RetryingRegistration<F, G, S> createNewRegistration() {
		RetryingRegistration<F, G, S> newRegistration = checkNotNull(generateRegistration());

		CompletableFuture<Tuple2<G, S>> future = newRegistration.getFuture();

		future.whenCompleteAsync(
			(Tuple2<G, S> result, Throwable failure) -> {
				if (failure != null) {
					if (failure instanceof CancellationException) {
						// we ignore cancellation exceptions because they originate from cancelling
						// the RetryingRegistration
						log.debug("Retrying registration towards {} was cancelled.", targetAddress);
					} else {
						// this future should only ever fail if there is a bug, not if the registration is declined
						onRegistrationFailure(failure);
					}
				} else {
					targetGateway = result.f0;
					onRegistrationSuccess(result.f1);
				}
			}, executor);

		return newRegistration;
	}

	//先生成注册对象
	//ResourceManagerConnection IN JobMaster
	@Override
	protected RetryingRegistration<ResourceManagerId, ResourceManagerGateway, JobMasterRegistrationSuccess> generateRegistration() {
		return new RetryingRegistration<ResourceManagerId, ResourceManagerGateway, JobMasterRegistrationSuccess>(
			log,
			getRpcService(),
			"ResourceManager",
			ResourceManagerGateway.class,
			getTargetAddress(),
			getTargetLeaderId(),
			jobMasterConfiguration.getRetryingRegistrationConfiguration()) {
				
            @Override
			protected CompletableFuture<RegistrationResponse> invokeRegistration(
					ResourceManagerGateway gateway, ResourceManagerId fencingToken, long timeoutMillis) {
				Time timeout = Time.milliseconds(timeoutMillis);
					
                return gateway.registerJobManager(
					jobMasterId,
					jobManagerResourceID,
					jobManagerRpcAddress,
					jobID,
					timeout);
			}
		};
	}

	//然后调用startRegistration()方法，开始注册，注册成功之后，调用 onRegistrationSuccess()
	protected void onRegistrationSuccess(final JobMasterRegistrationSuccess success) {
		runAsync(() -> {
			// filter out outdated connections
			//noinspection ObjectEquality
			if (this == resourceManagerConnection) {
				establishResourceManagerConnection(success);
			}
		});
	}

	//ResourceManagerConnection IN JobMaster
	@Override
	protected void onRegistrationSuccess(final JobMasterRegistrationSuccess success) {
		runAsync(() -> {
			// filter out outdated connections
			//noinspection ObjectEquality
			if (this == resourceManagerConnection) {
				establishResourceManagerConnection(success);
			}
		});
	}


	//JobMaster
	private void establishResourceManagerConnection(final JobMasterRegistrationSuccess success) {
		final ResourceManagerId resourceManagerId = success.getResourceManagerId();

		// verify the response with current connection
		if (resourceManagerConnection != null
				&& Objects.equals(resourceManagerConnection.getTargetLeaderId(), resourceManagerId)) {

			log.info("JobManager successfully registered at ResourceManager, leader id: {}.", resourceManagerId);

			final ResourceManagerGateway resourceManagerGateway = resourceManagerConnection.getTargetGateway();

			final ResourceID resourceManagerResourceId = success.getResourceManagerResourceId();

			establishedResourceManagerConnection = new EstablishedResourceManagerConnection(
				resourceManagerGateway,
				resourceManagerResourceId);

			/*TODO slotpool连接到ResourceManager，请求资源*/
			slotPool.connectToResourceManager(resourceManagerGateway);

			resourceManagerHeartbeatManager.monitorTarget(resourceManagerResourceId, new HeartbeatTarget<Void>() {
				@Override
				public void receiveHeartbeat(ResourceID resourceID, Void payload) {
					resourceManagerGateway.heartbeatFromJobManager(resourceID);
				}

				@Override
				public void requestHeartbeat(ResourceID resourceID, Void payload) {
					// request heartbeat will never be called on the job manager side
				}
			});
		} else {
			log.debug("Ignoring resource manager connection to {} because it's duplicated or outdated.", resourceManagerId);

		}
	}


	//SlotPoolImpl
	@Override
	public void connectToResourceManager(@Nonnull ResourceManagerGateway resourceManagerGateway) {
		this.resourceManagerGateway = checkNotNull(resourceManagerGateway);

		// work on all slots waiting for this connection
		for (PendingRequest pendingRequest : waitingForResourceManager.values()) {
			requestSlotFromResourceManager(resourceManagerGateway, pendingRequest);
		}

		// all sent off
		waitingForResourceManager.clear();
	}


	private void requestSlotFromResourceManager(
			final ResourceManagerGateway resourceManagerGateway,
			final PendingRequest pendingRequest) {

		checkNotNull(resourceManagerGateway);
		checkNotNull(pendingRequest);

		final AllocationID allocationId = new AllocationID();
		pendingRequest.setAllocationId(allocationId);

		pendingRequests.put(pendingRequest.getSlotRequestId(), allocationId, pendingRequest);

		pendingRequest.getAllocatedSlotFuture().whenComplete(
			(AllocatedSlot allocatedSlot, Throwable throwable) -> {
				if (throwable != null) {
					// the allocation id can be remapped so we need to get it from the pendingRequest
					// where it will be updated timely
					final Optional<AllocationID> updatedAllocationId = pendingRequest.getAllocationId();

					if (updatedAllocationId.isPresent()) {
						// cancel the slot request if there is a failure
						resourceManagerGateway.cancelSlotRequest(updatedAllocationId.get());
					}
				}
			});

		log.info("Requesting new slot [{}] and profile {} with allocation id {} from resource manager.",
			pendingRequest.getSlotRequestId(), pendingRequest.getResourceProfile(), allocationId);

		//向RM的SlotManager请求slot，并拿到回复
        CompletableFuture<Acknowledge> rmResponse = resourceManagerGateway.requestSlot(
			jobMasterId,
			new SlotRequest(jobId, allocationId, pendingRequest.getResourceProfile(), jobManagerAddress),
			rpcTimeout);

		FutureUtils.whenCompleteAsyncIfNotDone(
			rmResponse,
			componentMainThreadExecutor,
			(Acknowledge ignored, Throwable failure) -> {
				// on failure, fail the request future
				if (failure != null) {
					slotRequestToResourceManagerFailed(pendingRequest.getSlotRequestId(), failure);
				}
			});
	}


	//ResourceManager
	@Override
	public CompletableFuture<Acknowledge> requestSlot(
			JobMasterId jobMasterId,
			SlotRequest slotRequest,
			final Time timeout) {

		JobID jobId = slotRequest.getJobId();
		JobManagerRegistration jobManagerRegistration = jobManagerRegistrations.get(jobId);

		if (null != jobManagerRegistration) {
			if (Objects.equals(jobMasterId, jobManagerRegistration.getJobMasterId())) {
				log.info("Request slot with profile {} for job {} with allocation id {}.",
					slotRequest.getResourceProfile(),
					slotRequest.getJobId(),
					slotRequest.getAllocationId());

				try {
					/*TODO ResourceManager内部的 slotManager去向 Yarn的ResourceManager申请资源*/
					slotManager.registerSlotRequest(slotRequest);
				} catch (ResourceManagerException e) {
					return FutureUtils.completedExceptionally(e);
				}

				return CompletableFuture.completedFuture(Acknowledge.get());
			} else {
				return FutureUtils.completedExceptionally(new ResourceManagerException("The job leader's id " +
					jobManagerRegistration.getJobMasterId() + " does not match the received id " + jobMasterId + '.'));
			}

		} else {
			return FutureUtils.completedExceptionally(new ResourceManagerException("Could not find registered job manager for job " + jobId + '.'));
		}
	}


	//SlotManagerImpl
	@Override
	public boolean registerSlotRequest(SlotRequest slotRequest) throws ResourceManagerException {
		checkInit();

		if (checkDuplicateRequest(slotRequest.getAllocationId())) {
			LOG.debug("Ignoring a duplicate slot request with allocation id {}.", slotRequest.getAllocationId());

			return false;
		} else {
			PendingSlotRequest pendingSlotRequest = new PendingSlotRequest(slotRequest);

			pendingSlotRequests.put(slotRequest.getAllocationId(), pendingSlotRequest);

			try {
				internalRequestSlot(pendingSlotRequest);
			} catch (ResourceManagerException e) {
				// requesting the slot failed --> remove pending slot request
				pendingSlotRequests.remove(slotRequest.getAllocationId());

				throw new ResourceManagerException("Could not fulfill slot request " + slotRequest.getAllocationId() + '.', e);
			}

			return true;
		}
	}


	/**
	 * Tries to allocate a slot for the given slot request. If there is no slot available, the
	 * resource manager is informed to allocate more resources and a timeout for the request is
	 * registered.
	 *
	 * @param pendingSlotRequest to allocate a slot for
	 * @throws ResourceManagerException if the slot request failed or is unfulfillable
	 */
	private void internalRequestSlot(PendingSlotRequest pendingSlotRequest) throws ResourceManagerException {
		final ResourceProfile resourceProfile = pendingSlotRequest.getResourceProfile();

		OptionalConsumer.of(findMatchingSlot(resourceProfile))
			.ifPresent(taskManagerSlot -> allocateSlot(taskManagerSlot, pendingSlotRequest))
			.ifNotPresent(() -> fulfillPendingSlotRequestWithPendingTaskManagerSlot(pendingSlotRequest));
	}

	private void fulfillPendingSlotRequestWithPendingTaskManagerSlot(PendingSlotRequest pendingSlotRequest) throws ResourceManagerException {
		ResourceProfile resourceProfile = pendingSlotRequest.getResourceProfile();
		Optional<PendingTaskManagerSlot> pendingTaskManagerSlotOptional = findFreeMatchingPendingTaskManagerSlot(resourceProfile);

		if (!pendingTaskManagerSlotOptional.isPresent()) {
			pendingTaskManagerSlotOptional = allocateResource(resourceProfile);
		}

		OptionalConsumer.of(pendingTaskManagerSlotOptional)
			.ifPresent(pendingTaskManagerSlot -> assignPendingTaskManagerSlot(pendingSlotRequest, pendingTaskManagerSlot))
			.ifNotPresent(() -> {
				// request can not be fulfilled by any free slot or pending slot that can be allocated,
				// check whether it can be fulfilled by allocated slots
				if (failUnfulfillableRequest && !isFulfillableByRegisteredOrPendingSlots(pendingSlotRequest.getResourceProfile())) {
					throw new UnfulfillableSlotRequestException(pendingSlotRequest.getAllocationId(), pendingSlotRequest.getResourceProfile());
				}
			});
	}


	private Optional<PendingTaskManagerSlot> allocateResource(ResourceProfile requestedSlotResourceProfile) {
		……………………

		if (!resourceActions.allocateResource(defaultWorkerResourceSpec)) {
			// resource cannot be allocated
			return Optional.empty();
		}

		……………………
	}


	//ResourceActionsImpl
	@Override
	public boolean allocateResource(WorkerResourceSpec workerResourceSpec) {
		validateRunsInMainThread();
		return startNewWorker(workerResourceSpec);
	}


	//ActiveResourceManager
	@Override
	public boolean startNewWorker(WorkerResourceSpec workerResourceSpec) {
		requestNewWorker(workerResourceSpec);
		return true;
	}


	private void requestNewWorker(WorkerResourceSpec workerResourceSpec) {
        //拿一些内存和资源的配置
		final TaskExecutorProcessSpec taskExecutorProcessSpec =
				TaskExecutorProcessUtils.processSpecFromWorkerResourceSpec(flinkConfig, workerResourceSpec);
		final int pendingCount = pendingWorkerCounter.increaseAndGet(workerResourceSpec);

		log.info("Requesting new worker with resource spec {}, current pending count: {}.",
				workerResourceSpec,
				pendingCount);

		//申请资源
        CompletableFuture<WorkerType> requestResourceFuture = resourceManagerDriver.requestResource(taskExecutorProcessSpec);
		FutureUtils.assertNoException(
				requestResourceFuture.handle((worker, exception) -> {
					if (exception != null) {
						final int count = pendingWorkerCounter.decreaseAndGet(workerResourceSpec);
						log.warn("Failed requesting worker with resource spec {}, current pending count: {}, exception: {}",
								workerResourceSpec,
								count,
								exception);
						requestWorkerIfRequired();
					} else {
						final ResourceID resourceId = worker.getResourceID();
						workerNodeMap.put(resourceId, worker);
						currentAttemptUnregisteredWorkers.put(resourceId, workerResourceSpec);
						log.info("Requested worker {} with resource spec {}.",
								resourceId.getStringWithMetadata(),
								workerResourceSpec);
					}
					return null;
				}));
	}


	//YarnResourceManagerDriver
	@Override
	public CompletableFuture<YarnWorkerNode> requestResource(TaskExecutorProcessSpec taskExecutorProcessSpec) {
		checkInitialized();

		final CompletableFuture<YarnWorkerNode> requestResourceFuture = new CompletableFuture<>();

		final Optional<TaskExecutorProcessSpecContainerResourcePriorityAdapter.PriorityAndResource> priorityAndResourceOpt =
			taskExecutorProcessSpecContainerResourcePriorityAdapter.getPriorityAndResource(taskExecutorProcessSpec);

		if (!priorityAndResourceOpt.isPresent()) {
			requestResourceFuture.completeExceptionally(
				new ResourceManagerException(
					String.format("Could not compute the container Resource from the given TaskExecutorProcessSpec %s. " +
							"This usually indicates the requested resource is larger than Yarn's max container resource limit.",
						taskExecutorProcessSpec)));
		} else {
			final Priority priority = priorityAndResourceOpt.get().getPriority();
			final Resource resource = priorityAndResourceOpt.get().getResource();
			resourceManagerClient.addContainerRequest(getContainerRequest(resource, priority));

			// make sure we transmit the request fast and receive fast news of granted allocations
			resourceManagerClient.setHeartbeatInterval(containerRequestHeartbeatIntervalMillis);

			requestResourceFutures.computeIfAbsent(taskExecutorProcessSpec, ignore -> new LinkedList<>()).add(requestResourceFuture);

			log.info("Requesting new TaskExecutor container with resource {}, priority {}.", taskExecutorProcessSpec, priority);
		}

		return requestResourceFuture;
	}
6、启动TaskManager
//YarnTaskExecutorRunner.java
	public static void main(String[] args) {
		EnvironmentInformation.logEnvironmentInfo(LOG, "YARN TaskExecutor runner", args);
		SignalHandler.register(LOG);
		JvmShutdownSafeguard.installAsShutdownHook(LOG);
		runTaskManagerSecurely(args);
	}


	private static void runTaskManagerSecurely(String[] args) {
		try {
				LOG.debug("All environment variables: {}", ENV);
            	final String currDir = ENV.get(Environment.PWD.key());
				LOG.info("Current working Directory: {}", currDir);
				final Configuration configuration = TaskManagerRunner.loadConfiguration(args);
				setupAndModifyConfiguration(configuration, currDir, ENV);
				TaskManagerRunner.runTaskManagerSecurely(configuration);
			}
		catch (Throwable t) {
            final Throwable strippedThrowable = ExceptionUtils.stripException(t, 
UndeclaredThrowableException.class);
			// make sure that everything whatever ends up in the log
			LOG.error("YARN TaskManager initialization failed.", strippedThrowable);
			System.exit(INIT_ERROR_EXIT_CODE);
		}
	}


//TaskManagerRunner.java
	public static void runTaskManagerSecurely(Configuration configuration) throws Exception {
		replaceGracefulExitWithHaltIfConfigured(configuration);
		final PluginManager pluginManager = PluginUtils.createPluginManagerFromRootFolder(configuration);
		FileSystem.initialize(configuration, pluginManager);
		SecurityUtils.install(new SecurityConfiguration(configuration));
		SecurityUtils.getInstalledContext().runSecured(() -> {
            runTaskManager(configuration, pluginManager);
            return null;
		});
	}


	public static void runTaskManager(Configuration configuration, PluginManager pluginManager) throws 			Exception {
		final TaskManagerRunner taskManagerRunner = new TaskManagerRunner(configuration, 							pluginManager,TaskManagerRunner::createTaskExecutorService);
		
        taskManagerRunner.start();
	}

	public void start() throws Exception {
		taskExecutorService.start();
	}


//TaskExecutorToServiceAdapter.java
	public void start() {
		taskExecutor.start();
	}


//TaskExecutor.java
	public void onStart() throws Exception {
		try {
			startTaskExecutorServices();
		} catch (Throwable t) {
            final TaskManagerException exception = new TaskManagerException(String.format("Could not start the TaskExecutor %s", getAddress()), t);
			onFatalError(exception);
			throw exception;
		}
		startRegistrationTimeout();
    }
7、注册并分配slot
//TaskExecutor.java
	private void startTaskExecutorServices() throws Exception {
		try {
			// start by connecting to the ResourceManager
			resourceManagerLeaderRetriever.start(new ResourceManagerLeaderListener());
			// tell the task slot table who's responsible for the task slot actions
			taskSlotTable.start(new SlotActionsImpl(), getMainThreadExecutor());
			// start the job leader service
			jobLeaderService.start(getAddress(), getRpcService(), haServices, new 
			JobLeaderListenerImpl());
			fileCache = new FileCache(taskManagerConfiguration.getTmpDirectories(), 
			blobCacheService.getPermanentBlobService());
		} catch (Exception e) {
			handleStartTaskExecutorServicesException(e);
		}
	}

//以上的逻辑和之前注册一样：领导寻回，连接到当前的领导，然后开始注册直到执行后来的动作，领导寻回动作开始调用start的时候，走到通用的方法，然后找到TaskExecutor那边的具体实现，粗略流程如下：找到RM的领导，传入RM监听器通知TaskExecutor当前领导是谁并重连接到领导，现在调用到TaskExecutor 的 resourceManagerConnection.start()方法。
resourceManagerLeaderRetriever.start(new ResourceManagerLeaderListener());
-> notifyOfNewResourceManagerLeader() 
-> TaskExecutor 的 notifyOfNewResourceManagerLeader() 
-> TaskExecutor 的 reconnectToResourceManager()
-> TaskExecutor 的 tryConnectToResourceManager()
-> TaskExecutor 的 connectToResourceManager()
-> TaskExecutor 的 resourceManagerConnection.start()
    

//RegisteredRpcConnection.java（开始连接，连接的逻辑里面从注册开始，又走到了注册的那一套通用方法）
	public void start() {
		checkState(!closed, "The RPC connection is already closed");
		checkState(!isConnected() && pendingRegistration == null, "The RPC connection is already started");

		/*TODO 创建注册对象*/
		final RetryingRegistration<F, G, S> newRegistration = createNewRegistration();

		if (REGISTRATION_UPDATER.compareAndSet(this, null, newRegistration)) {
			/*TODO 开始注册，注册成功之后，调用 onRegistrationSuccess()*/
			newRegistration.startRegistration();
		} else {
			// concurrent start operation
			newRegistration.cancel();
		}
	}



	private RetryingRegistration<F, G, S> createNewRegistration() {
		RetryingRegistration<F, G, S> newRegistration = checkNotNull(generateRegistration());

		…………………………

		return newRegistration;
	}


//TaskExecutorToResourceManagerConnection（从通用到具体，最后是返回了一个内部类，就是TaskExecutor到RM的连接类当中的RM注册内部类的对象，然后等注册成功之后，通过之前传进来的listener--ResourceManagerRegistrationListener，调用onRegistrationSuccess方法，因为开始注册是从TaskExecutor这里开始走的，大概 —_—|||）
	@Override
	protected RetryingRegistration<ResourceManagerId, ResourceManagerGateway, TaskExecutorRegistrationSuccess> generateRegistration() {
		return new TaskExecutorToResourceManagerConnection.ResourceManagerRegistration(
			log,
			rpcService,
			getTargetAddress(),
			getTargetLeaderId(),
			retryingRegistrationConfiguration,
			taskExecutorRegistration);
	}


//回到RegisteredRpcConnection的start方法当中的newRegistration.startRegistration();创建注册对象完毕，开始注册
//RetryingRegistration
	public void startRegistration() {
		if (canceled) {
			// we already got canceled
			return;
		}

		try {
			// trigger resolution of the target address to a callable gateway
			final CompletableFuture<G> rpcGatewayFuture;

			if (FencedRpcGateway.class.isAssignableFrom(targetType)) {
				rpcGatewayFuture = (CompletableFuture<G>) rpcService.connect(
					targetAddress,
					fencingToken,
					targetType.asSubclass(FencedRpcGateway.class));
			} else {
				rpcGatewayFuture = rpcService.connect(targetAddress, targetType);
			}

			// upon success, start the registration attempts
			CompletableFuture<Void> rpcGatewayAcceptFuture = rpcGatewayFuture.thenAcceptAsync(
				(G rpcGateway) -> {
					log.info("Resolved {} address, beginning registration", targetName);
					//点进去，准备调用注册逻辑
                    register(rpcGateway, 1, retryingRegistrationConfiguration.getInitialRegistrationTimeoutMillis());
				},
				rpcService.getExecutor());

			// upon failure, retry, unless this is cancelled
			rpcGatewayAcceptFuture.whenCompleteAsync(
				(Void v, Throwable failure) -> {
					if (failure != null && !canceled) {
						final Throwable strippedFailure = ExceptionUtils.stripCompletionException(failure);
						if (log.isDebugEnabled()) {
							log.debug(
								"Could not resolve {} address {}, retrying in {} ms.",
								targetName,
								targetAddress,
								retryingRegistrationConfiguration.getErrorDelayMillis(),
								strippedFailure);
						} else {
							log.info(
								"Could not resolve {} address {}, retrying in {} ms: {}",
								targetName,
								targetAddress,
								retryingRegistrationConfiguration.getErrorDelayMillis(),
								strippedFailure.getMessage());
						}

                        //如果失败，过段时间重试
						startRegistrationLater(retryingRegistrationConfiguration.getErrorDelayMillis());
					}
				},
				rpcService.getExecutor());
		}
		catch (Throwable t) {
			completionFuture.completeExceptionally(t);
			cancel();
		}
	}


	private void register(final G gateway, final int attempt, final long timeoutMillis) {
		// eager check for canceling to avoid some unnecessary work
		if (canceled) {
			return;
		}

		try {
			log.debug("Registration at {} attempt {} (timeout={}ms)", targetName, attempt, timeoutMillis);
			CompletableFuture<RegistrationResponse> registrationFuture = invokeRegistration(gateway, fencingToken, timeoutMillis);

			// ……………………后面省略一大坨，总之是处理后事的逻辑，成功了通知一下TaskExecutor，失败了就重试
	}
        
        
        //TaskExecutorToResourceManagerConnection.java 的内部类 ResourceManagerRegistration
		/*TODO 开始注册，调用的是这个方法*/
		@Override
		protected CompletableFuture<RegistrationResponse> invokeRegistration(
				ResourceManagerGateway resourceManager, ResourceManagerId fencingToken, long timeoutMillis) throws Exception {

			Time timeout = Time.milliseconds(timeoutMillis);
			return resourceManager.registerTaskExecutor(
				taskExecutorRegistration,
				timeout);
		}
        
        
        
	//终于调到了TaskExecutor的内部类ResourceManagerRegistrationListener当中的onRegistrationSuccess
		@Override
		public void onRegistrationSuccess(TaskExecutorToResourceManagerConnection connection, TaskExecutorRegistrationSuccess success) {
			final ResourceID resourceManagerId = success.getResourceManagerId();
			final InstanceID taskExecutorRegistrationId = success.getRegistrationId();
			final ClusterInformation clusterInformation = success.getClusterInformation();
			final ResourceManagerGateway resourceManagerGateway = connection.getTargetGateway();

			runAsync(
				() -> {
					// filter out outdated connections
					//noinspection ObjectEquality
					if (resourceManagerConnection == connection) {
						try {
                            //注册成功，与RM建立连接
							establishResourceManagerConnection(
								resourceManagerGateway,
								resourceManagerId,
								taskExecutorRegistrationId,
								clusterInformation);
						} catch (Throwable t) {
							log.error("Establishing Resource Manager connection in Task Executor failed", t);
						}
					}
				});
		}
        
        
        
    private void establishResourceManagerConnection(
		ResourceManagerGateway resourceManagerGateway,
		ResourceID resourceManagerResourceId,
		InstanceID taskExecutorRegistrationId,
		ClusterInformation clusterInformation) {

		final CompletableFuture<Acknowledge> slotReportResponseFuture = resourceManagerGateway.sendSlotReport(
			getResourceID(),
			taskExecutorRegistrationId,
			taskSlotTable.createSlotReport(getResourceID()),
			taskManagerConfiguration.getTimeout());

		……………………………………

		stopRegistrationTimeout();
	}
        
        
    //ResourceManager.java
    @Override
	public CompletableFuture<Acknowledge> sendSlotReport(ResourceID taskManagerResourceId, InstanceID taskManagerRegistrationId, SlotReport slotReport, Time timeout) {
		final WorkerRegistration<WorkerType> workerTypeWorkerRegistration = taskExecutors.get(taskManagerResourceId);

		if (workerTypeWorkerRegistration.getInstanceID().equals(taskManagerRegistrationId)) {
			if (slotManager.registerTaskManager(workerTypeWorkerRegistration, slotReport)) {
				onWorkerRegistered(workerTypeWorkerRegistration.getWorker());
			}
			return CompletableFuture.completedFuture(Acknowledge.get());
		} else {
			return FutureUtils.completedExceptionally(new ResourceManagerException(String.format("Unknown TaskManager registration id %s.", taskManagerRegistrationId)));
		}
	}
        
    
        
	//SlotManagerImpl.java
	@Override
	public boolean registerTaskManager(final TaskExecutorConnection taskExecutorConnection, SlotReport initialSlotReport) {
		checkInit();

		LOG.debug("Registering TaskManager {} under {} at the SlotManager.", taskExecutorConnection.getResourceID().getStringWithMetadata(), taskExecutorConnection.getInstanceID());

		// we identify task managers by their instance id
        // 通过实例 id 判断某个 taskmanager 是否已经注册过
		if (taskManagerRegistrations.containsKey(taskExecutorConnection.getInstanceID())) {
            // 报告已注册过的 taskmanager 的 slot 分配情况，更新 slot 情况
			reportSlotStatus(taskExecutorConnection.getInstanceID(), initialSlotReport);
			return false;
		} else {
			if (isMaxSlotNumExceededAfterRegistration(initialSlotReport)) {
				LOG.info("The total number of slots exceeds the max limitation {}, release the excess resource.", maxSlotNum);
				resourceActions.releaseResource(taskExecutorConnection.getInstanceID(), new FlinkException("The total number of slots exceeds the max limitation."));
				return false;
			}

			// first register the TaskManager
			ArrayList<SlotID> reportedSlots = new ArrayList<>();

			for (SlotStatus slotStatus : initialSlotReport) {
				reportedSlots.add(slotStatus.getSlotID());
			}

			TaskManagerRegistration taskManagerRegistration = new TaskManagerRegistration(
				taskExecutorConnection,
				reportedSlots);

			taskManagerRegistrations.put(taskExecutorConnection.getInstanceID(), taskManagerRegistration);

			// next register the new slots
            // 注册新的 slot，根据 slot 请求进行分配
			for (SlotStatus slotStatus : initialSlotReport) {
				registerSlot(
					slotStatus.getSlotID(),
					slotStatus.getAllocationID(),
					slotStatus.getJobID(),
					slotStatus.getResourceProfile(),
					taskExecutorConnection);
			}

			return true;
		}

	}

//RM向TaskManager分配slot
//SlotManagerImpl.java
	private void registerSlot(
			SlotID slotId,
			AllocationID allocationId,
			JobID jobId,
			ResourceProfile resourceProfile,
			TaskExecutorConnection taskManagerConnection) {

		if (slots.containsKey(slotId)) {
			// remove the old slot first
			removeSlot(
				slotId,
				new SlotManagerException(
					String.format(
						"Re-registration of slot %s. This indicates that the TaskExecutor has re-connected.",
						slotId)));
		}

		/*TODO 创建和注册 新的这些 slot*/
		final TaskManagerSlot slot = createAndRegisterTaskManagerSlot(slotId, resourceProfile, taskManagerConnection);

		final PendingTaskManagerSlot pendingTaskManagerSlot;

		//先校验allocationId，先拿到挂起的slot，即先获取slotPool那边待分配的slot
        if (allocationId == null) {
			pendingTaskManagerSlot = findExactlyMatchingPendingTaskManagerSlot(resourceProfile);
		} else {
			pendingTaskManagerSlot = null;
		}

        //接着校验是否有待分配的slot
		if (pendingTaskManagerSlot == null) {
            //到此说明没有需要分配的slot
			updateSlot(slotId, allocationId, jobId);
		} else {
            //到此说明有slot请求需要被满足，先把挂起的槽从列表里删掉，再把处理挂起的slot的请求获取到，传下去
			pendingSlots.remove(pendingTaskManagerSlot.getTaskManagerSlotId());
			final PendingSlotRequest assignedPendingSlotRequest = pendingTaskManagerSlot.getAssignedPendingSlotRequest();

			/*TODO 分配slot*/
			if (assignedPendingSlotRequest == null) {
				/*TODO 表示 挂起的请求都已经满足了，你暂时没事*/
				handleFreeSlot(slot);
			} else {
				/*TODO 表示 你要被分配给某个请求，这步动作中，先解除待分配slot的挂起状态，然后在分配RM那边的slot给TM，最后准备提供给JM*/
				assignedPendingSlotRequest.unassignPendingTaskManagerSlot();
				allocateSlot(slot, assignedPendingSlotRequest);
			}
		}
	}


	private void allocateSlot(TaskManagerSlot taskManagerSlot, PendingSlotRequest pendingSlotRequest) {
		Preconditions.checkState(taskManagerSlot.getState() == SlotState.FREE);

		TaskExecutorConnection taskExecutorConnection = taskManagerSlot.getTaskManagerConnection();
		TaskExecutorGateway gateway = taskExecutorConnection.getTaskExecutorGateway();

		final CompletableFuture<Acknowledge> completableFuture = new CompletableFuture<>();
		final AllocationID allocationId = pendingSlotRequest.getAllocationId();
		final SlotID slotId = taskManagerSlot.getSlotId();
		final InstanceID instanceID = taskManagerSlot.getInstanceId();

		taskManagerSlot.assignPendingSlotRequest(pendingSlotRequest);
		pendingSlotRequest.setRequestFuture(completableFuture);

		returnPendingTaskManagerSlotIfAssigned(pendingSlotRequest);

		TaskManagerRegistration taskManagerRegistration = taskManagerRegistrations.get(instanceID);

		if (taskManagerRegistration == null) {
			throw new IllegalStateException("Could not find a registered task manager for instance id " +
				instanceID + '.');
		}

		taskManagerRegistration.markUsed();

		// RPC call to the task manager
		/*TODO 分配完之后，通知 TM提供 slot给 JM*/
		CompletableFuture<Acknowledge> requestFuture = gateway.requestSlot(
			slotId,
			pendingSlotRequest.getJobId(),
			allocationId,
			pendingSlotRequest.getResourceProfile(),
			pendingSlotRequest.getTargetAddress(),
			resourceManagerId,
			taskManagerRequestTimeout);

		requestFuture.whenComplete(
			(Acknowledge acknowledge, Throwable throwable) -> {
				if (acknowledge != null) {
					completableFuture.complete(acknowledge);
				} else {
					completableFuture.completeExceptionally(throwable);
				}
			});

		completableFuture.whenCompleteAsync(
			(Acknowledge acknowledge, Throwable throwable) -> {
				try {
					if (acknowledge != null) {
						updateSlot(slotId, allocationId, pendingSlotRequest.getJobId());
					} else {
						if (throwable instanceof SlotOccupiedException) {
							SlotOccupiedException exception = (SlotOccupiedException) throwable;
							updateSlot(slotId, exception.getAllocationId(), exception.getJobId());
						} else {
							removeSlotRequestFromSlot(slotId, allocationId);
						}

						if (!(throwable instanceof CancellationException)) {
							handleFailedSlotRequest(slotId, allocationId, throwable);
						} else {
							LOG.debug("Slot allocation request {} has been cancelled.", allocationId, throwable);
						}
					}
				} catch (Exception e) {
					LOG.error("Error while completing the slot allocation.", e);
				}
			},
			mainThreadExecutor);
	}


	//TaskExecutor
	@Override
	public CompletableFuture<Acknowledge> requestSlot(
		final SlotID slotId,
		final JobID jobId,
		final AllocationID allocationId,
		final ResourceProfile resourceProfile,
		final String targetAddress,
		final ResourceManagerId resourceManagerId,
		final Time timeout) {
		// TODO: Filter invalid requests from the resource manager by using the instance/registration Id

		log.info("Receive slot request {} for job {} from resource manager with leader id {}.",
			allocationId, jobId, resourceManagerId);

		if (!isConnectedToResourceManager(resourceManagerId)) {
			final String message = String.format("TaskManager is not connected to the resource manager %s.", resourceManagerId);
			log.debug(message);
			return FutureUtils.completedExceptionally(new TaskManagerException(message));
		}

		try {
			/*TODO 根据 RM的命令，分配自己的slot*/
			allocateSlot(
				slotId,
				jobId,
				allocationId,
				resourceProfile);
		} catch (SlotAllocationException sae) {
			return FutureUtils.completedExceptionally(sae);
		}

		final JobTable.Job job;

		try {
			job = jobTable.getOrCreateJob(jobId, () -> registerNewJobAndCreateServices(jobId, targetAddress));
		} catch (Exception e) {
			// free the allocated slot
			try {
				taskSlotTable.freeSlot(allocationId);
			} catch (SlotNotFoundException slotNotFoundException) {
				// slot no longer existent, this should actually never happen, because we've
				// just allocated the slot. So let's fail hard in this case!
				onFatalError(slotNotFoundException);
			}

			// release local state under the allocation id.
			localStateStoresManager.releaseLocalStateForAllocationId(allocationId);

			// sanity check
			if (!taskSlotTable.isSlotFree(slotId.getSlotNumber())) {
				onFatalError(new Exception("Could not free slot " + slotId));
			}

			return FutureUtils.completedExceptionally(new SlotAllocationException("Could not create new job.", e));
		}

		if (job.isConnected()) {
			/*TODO 向JobManager提供 slot*/
			offerSlotsToJobManager(jobId);
		}

		return CompletableFuture.completedFuture(Acknowledge.get());
	}


	//根据 RM的命令，分配自己的slot
	private void allocateSlot(
			SlotID slotId,
			JobID jobId,
			AllocationID allocationId,
			ResourceProfile resourceProfile) throws SlotAllocationException {
		if (taskSlotTable.isSlotFree(slotId.getSlotNumber())) {

			if (taskSlotTable.allocateSlot(slotId.getSlotNumber(), jobId, allocationId, resourceProfile, taskManagerConfiguration.getTimeout())) {
				log.info("Allocated slot for {}.", allocationId);
			} else {
				log.info("Could not allocate slot for {}.", allocationId);
				throw new SlotAllocationException("Could not allocate slot.");
			}
		} else if (!taskSlotTable.isAllocated(slotId.getSlotNumber(), jobId, allocationId)) {
			final String message = "The slot " + slotId + " has already been allocated for a different job.";

			log.info(message);

			final AllocationID allocationID = taskSlotTable.getCurrentAllocation(slotId.getSlotNumber());
			throw new SlotOccupiedException(message, allocationID, taskSlotTable.getOwningJob(allocationID));
		}
	}


	//
	private void allocateSlot(
			SlotID slotId,
			JobID jobId,
			AllocationID allocationId,
			ResourceProfile resourceProfile) throws SlotAllocationException {
		if (taskSlotTable.isSlotFree(slotId.getSlotNumber())) {

			
            //
            if (taskSlotTable.allocateSlot(slotId.getSlotNumber(), jobId, allocationId, resourceProfile, taskManagerConfiguration.getTimeout())) {
				log.info("Allocated slot for {}.", allocationId);
			} else {
				log.info("Could not allocate slot for {}.", allocationId);
				throw new SlotAllocationException("Could not allocate slot.");
			}
		} else if (!taskSlotTable.isAllocated(slotId.getSlotNumber(), jobId, allocationId)) {
			final String message = "The slot " + slotId + " has already been allocated for a different job.";

			log.info(message);

			final AllocationID allocationID = taskSlotTable.getCurrentAllocation(slotId.getSlotNumber());
			throw new SlotOccupiedException(message, allocationID, taskSlotTable.getOwningJob(allocationID));
		}
	}


//向JobManager提供 slot
	private void offerSlotsToJobManager(final JobID jobId) {
		jobTable
			.getConnection(jobId)
			.ifPresent(this::internalOfferSlotsToJobManager);
	}


	private void internalOfferSlotsToJobManager(JobTable.Connection jobManagerConnection) {
		final JobID jobId = jobManagerConnection.getJobId();

		if (taskSlotTable.hasAllocatedSlots(jobId)) {
			log.info("Offer reserved slots to the leader of job {}.", jobId);

			final JobMasterGateway jobMasterGateway = jobManagerConnection.getJobManagerGateway();

			final Iterator<TaskSlot<Task>> reservedSlotsIterator = taskSlotTable.getAllocatedSlots(jobId);
			final JobMasterId jobMasterId = jobManagerConnection.getJobMasterId();

			final Collection<SlotOffer> reservedSlots = new HashSet<>(2);

			while (reservedSlotsIterator.hasNext()) {
				SlotOffer offer = reservedSlotsIterator.next().generateSlotOffer();
				reservedSlots.add(offer);
			}

			CompletableFuture<Collection<SlotOffer>> acceptedSlotsFuture = jobMasterGateway.offerSlots(
				getResourceID(),
				reservedSlots,
				taskManagerConfiguration.getTimeout());

			acceptedSlotsFuture.whenCompleteAsync(
				handleAcceptedSlotOffers(jobId, jobMasterGateway, jobMasterId, reservedSlots),
				getMainThreadExecutor());
		} else {
			log.debug("There are no unassigned slots for the job {}.", jobId);
		}
	}


//JobMaster.java
	@Override
	public CompletableFuture<Collection<SlotOffer>> offerSlots(
			final ResourceID taskManagerId,
			final Collection<SlotOffer> slots,
			final Time timeout) {

		Tuple2<TaskManagerLocation, TaskExecutorGateway> taskManager = registeredTaskManagers.get(taskManagerId);

		if (taskManager == null) {
			return FutureUtils.completedExceptionally(new Exception("Unknown TaskManager " + taskManagerId));
		}

		final TaskManagerLocation taskManagerLocation = taskManager.f0;
		final TaskExecutorGateway taskExecutorGateway = taskManager.f1;

		final RpcTaskManagerGateway rpcTaskManagerGateway = new RpcTaskManagerGateway(taskExecutorGateway, getFencingToken());

		return CompletableFuture.completedFuture(
			slotPool.offerSlots(
				taskManagerLocation,
				rpcTaskManagerGateway,
				slots));
	}


	//SlotPoolImpl.java
	@Override
	public Collection<SlotOffer> offerSlots(
			TaskManagerLocation taskManagerLocation,
			TaskManagerGateway taskManagerGateway,
			Collection<SlotOffer> offers) {

		ArrayList<SlotOffer> result = new ArrayList<>(offers.size());

		for (SlotOffer offer : offers) {
			if (offerSlot(
				taskManagerLocation,
				taskManagerGateway,
				offer)) {

				result.add(offer);
			}
		}

		return result;
	}


	boolean offerSlot(
			final TaskManagerLocation taskManagerLocation,
			final TaskManagerGateway taskManagerGateway,
			final SlotOffer slotOffer) {

		……………………

		// use the slot to fulfill pending request, in requested order
		tryFulfillSlotRequestOrMakeAvailable(allocatedSlot);

		// we accepted the request in any case. slot will be released after it idled for
		// too long and timed out
		return true;
	}
luyunlong_it
关注
专栏目录
Flink内核源码解析系列（七）：yarn-per-job提交流程
打工日常
06-08 297
上一篇文章《Flink内核源码解析系列（六）：调用执行环境的execute方法》中讲到了最终调用到AbstractJobClusterExecutor的execute方法 接下来总体看看execute方法中都做了些什么？ @Override public CompletableFuture<JobClient> execute( @Nonnull final Pipeline pipeline, @Nonnull final Con
flink源码分析1-3--yarnPerJob任务提交流程--部署集群前的准备&提交任务
luyunlong_it的博客
06-10 360
flink源码分析1-3--yarnPerJob任务提交流程--部署集群前的准备&提交任务
Flink Yarn Per Job - 提交流程一
hyunbar的博客
08-01 749
/构建nodechains，返回当前节点的物理出边；TODO将每个JobVertex的入边集合也序列化到该JobVertex的StreamConfig中(出边集合已经在setChaining的时候写入了)/*TODO最重要的函数，生成JobVertex，JobEdge等，并尽可能地将多个节点chain在一起*/TODO过渡用的出边集合,用来生成最终的JobEdge,注意不包括chain内部的边。//保证如果提交的拓扑没有改变，则每次生成的hash都是一样的。...
YARN的job提交过程
Faith_xzc
07-19 610
YARN的job提交过程job提交过程示意图job提交过程详解1.作业提交2.作业初始化3.任务分配4.任务运行5.进度和状态更新6.作业完成 job提交过程示意图 job提交过程详解 1.作业提交 （1）Client调用job.waitForCompletion方法，向整个集群提交MapReduce作业。 （2）Client向RM申请一个作业id。 （3）RM给Client返回该job资源的提交路径和作业id。 （4）Client提交jar包、切片信息和配置文件到指定的资源提交路径。 （5）Client
flink源码分析1-1--yarnPerJob任务提交流程--启动命令解析&读配置&封装
luyunlong_it的博客
06-07 642
flink源码分析--yarnPerJob任务提交流程--启动命令解析&读取配置文件&封装
Flink源码分析-Job调度部署运行流程.pdf
最新发布
09-19
Flink源码分析-Job调度部署运行流程.pdf
flink-spark-submiter:从本地IDEA提交FlinkSpark任务到Yarnk8s集群
04-13
Flink任务、Spark任务提交到集群，通常需要将可执行Jar上传到集群，手动执行任务提交指令，如果有配套的大数据平台则需要上传Jar，由调度系统进行任务提交。 对开发者来说，本地IDEA调试Flink、Spark任务不涉及对象...
docker-flink:使用 Docker-Compose 在 Docker 容器中部署 Apache Flink 集群
06-17
#Apache Flink 集群部署在 Docker 上使用 Docker-Compose ##安装###安装Docker 如果您遇到与 Docker 版本不兼容的 Docker-Compose 版本问题，请尝试curl -sSL https://get.docker.com/ubuntu/ | sudo sh ###Install ...
flink-1.14.0-bin-scala_2.11.tgz
07-18
描述中同样提到了“flink-1.14.0-bin-scala_2.11.tgz”，这意味着下载的压缩包包含了 Flink 的二进制版本，适合快速部署和运行。二进制版本通常包含所有必需的可执行文件和库，可以直接在支持的环境中运行，无需进行...
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar.zip
05-27
这个压缩包中的 `flink-shaded-hadoop-2-uber-2.7.5-10.0.jar` 文件，是 Flink 针对 Hadoop 2.7.5 版本的“Uber JAR”，它是将多个依赖库打包在一起的单一 JAR 文件，旨在简化部署和运行流程。 首先，我们来理解...
flink yarn-perjob任务提交流程
m0_46449152的博客
02-10 2222
1、由脚本bin/flink 进入org.apache.flink.client.cli.CliFrontend CliFrontend main -> cli.parseAndRun -> run -> executeProgram -> ClientUtils.executeProgram -> program.invokeInteractiveModeForExecution() -> callMainMethod -> mainMethod.invoke
Yarn 的任务提交流程
想练武，就得下功夫
07-21 519
Yarn 任务提交流程
Yarn部署模式下Flink单作业提交流程详解！！
gene20200509的博客
09-22 1164
大数据技术之flink yarn部署模式下单作业提交流程
Flink实例（五十三）: flink 作业提交（三）yarn 提交模式（二）Flink on YARN Per-Job模式
微信搜：import_bigdata，大数据领域硬核原创作者
01-23 2288
声明：本系列博客是根据SGG的视频整理而成，非常适合大家入门学习。 《2021年最新版大数据面试题全面开启更新》 Flink on YARN Per-Job模式 首先将之前在 yarn 上运行的应用和相关进程给kill掉： [root@hadoop01 ~]# yarn application -kill application_1601372571363_0001 [root@hadoop01 ~]# jps 6995 SecondaryNameNode 7204 ResourceManag
Apache Flink 的 YARN Session 提交流程
Shockang的博客
07-17 1719
从总体上来说，在 Yarn 集群上使用 Session 模式提交 Flink 作业的过程分为 3 个阶段： - 首先在 Yarn 上启动 Flink Session 模式的集群； - 其次通过 Flink Client 提交作业； - 最后进行作业调度执行。
flink任务管理yarn perJob模式代码实现
qq_27242695的博客
07-14 787
使用场景：flink云平台或者是公司内部flink任务管理平台，提交任务到yarn的流程代码 代码来源：分析yarn submit job 流程整合 功能： 1. 支持动态提交flink任务到yarn 2. 支持关闭flink任务 3. 支持实时获取任务状态 4. 支持实时获取任务jobId 5. 支持实时获取flinkWel 6. 支持回去yarn ClusterClient...
Flink on Yarn 提交流程与提交模式
hanzongqing的博客
03-10 3572
Flink
yarn-cluster提交流程per-job
mischen520的博客
06-23 315
Flink任务提交后，Client向HDFS上传Flink的Jar包和配置 向Yarn ResourceManager提交任务，ResourceManager分配Container资源 通知对应的NodeManager启动ApplicationMaster，ApplicationMaster启动后加载Flink的Jar包和配置构建环境，然后启动JobManager ApplicationMaster向ResourceManager申请资源启动TaskManager ResourceManager..
Java远程提交YARN MapReduce任务
争当鲲鹏飞万里 不做燕雀恋子巢
03-29 1609
YARN是可以通过Java程序向hadoop集群提交MapReduce任务的。与普通的任务不同的是，远程提交的Job由于读不到服务器上的mapred-site.xml和yarn-site.xml。 这样在本地eclipse中运行的程序是以yarn模式（而不是以local模式运行）运行，可在yarn的监控界面监控到任务的运行状态； package org.apache.hadoop.examp...
GPU加速内存集群计算中的大规模迭代稀疏矩阵-向量乘法
"GPU加速的大数据在内存集群计算中的迭代稀疏矩阵-向量乘法" 在当前的大数据时代，迭代稀疏矩阵-向量乘法（Iterative SpMV，ISpMV）是许多基于图的数据挖掘算法和机器学习算法的核心运算。随着数据规模的急剧增长，...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

luyunlong_it

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值