Flink 调度源码分析1：拓扑图创建与提交过程

3. 集群接收并处理 JobGraph

这部分实现在 flink-runtime 模块中的 org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.handleRequest() 方法中。在该方法中，会获取 restClient 的文件，获取 JobGraph 和对应的依赖、jar包，然后通过 dispatchergateway 提交 JobGraph。
下面代码负责继续处理 JobGraph。

CompletableFuture<Acknowledge> jobSubmissionFuture =  
        finalizedJobGraphFuture.thenCompose(  
                jobGraph -> gateway.submitJob(jobGraph, timeout));

可进而查看其调用的方法，如下所示。

gateway.submitJob(jobGraph, timeout)
-> internalSubmitJob(jobGraph)
	-> this::persistAndRunJob
		-> runJob(createJobMasterRunner(jobGraph), ExecutionType.SUBMISSION)

在 runJob() 中会创建 JobMaster，然后 JobGraph 在 JobMaster 中会被转化成 ExecutionGraph。

4. 生成 ExecutionGraph

Client 会将 JobGraph 进行提交，然后服务端会接收 JobGraph，创建 JobMaster，并将 JobGraph 转换为 ExecutionGraph。

• 集群会在创建调度器时，创建 ExecutionGraph。调度器会继承 SchedulerBase 类，在该类的构造函数中通过下面的代码创建 ExecutionGraph。

this.executionGraph =  
        createAndRestoreExecutionGraph(  
                completedCheckpointStore,  
                checkpointsCleaner,  
                checkpointIdCounter,  
                initializationTimestamp,  
                mainThreadExecutor,  
                jobStatusListener,  
                vertexParallelismStore);

查看其调用过程如下。

createAndRestoreExecutionGraph(...)
-> final ExecutionGraph newExecutionGraph =  
    executionGraphFactory.createAndRestoreExecutionGraph(...)
    -> final ExecutionGraph newExecutionGraph =  
    DefaultExecutionGraphBuilder.buildGraph(...)

• JobManager 会根据 JobGraph 生成对应的 ExecutionGraph。ExecutionGraph 需要根据并行度将每个算子创建 ExecutionGraph，每个 ExecutionGraph 对应一个实例。该功能主要实现在 flink-runtime 模块中的 org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph() 方法中。该方法的部分代码如下：

// 创建一个执行图 
final DefaultExecutionGraph executionGraph =  
        new DefaultExecutionGraph(......);  
// 设置执行图的作业名、节点等信息 
try {  
    executionGraph.setJsonPlan(JsonPlanGenerator.generatePlan(jobGraph));  
} catch (Throwable t) {  
    ......
}  
......
// 创建节点
for (JobVertex vertex : jobGraph.getVertices()) {  
    try {  
        vertex.initializeOnMaster(  
                new SimpleInitializeOnMasterContext(  
                        classLoader,  
                        vertexParallelismStore  
                                .getParallelismInfo(vertex.getID())  
                                .getParallelism()));  
    } catch (Throwable t) {......}  
}  
// topologically sort the job vertices and attach the graph to the existing one 
List<JobVertex> sortedTopology = jobGraph.getVerticesSortedTopologicallyFromSources();