FlinkSQL源码解析1--提交任务过程

提交入口

org.apache.flink.table.api.internal.TableEnvironmentImpl#executeSql
 @Override
    public TableResult executeSql(String statement) {
        List<Operation> operations = getParser().parse(statement);

        if (operations.size() != 1) {
            throw new TableException(UNSUPPORTED_QUERY_IN_EXECUTE_SQL_MSG);
        }

        Operation operation = operations.get(0);
        return executeInternal(operation);
    }

 提交过程概述

sql--》operation--》Transformation--》pipeline(StreamGraph)--》JobGraph--》deployJobCluster

具体过程

1、查询转成operation

sql-->SqlNodeList-->flinkPlanner.validate(sqlNode)-->Optional<Operation> operation = SqlNodeConverters.convertSqlNode(validated, context);

2、operation转成pipeline(StreamGraph)

org.apache.flink.table.api.internal.TableEnvironmentImpl#executeQueryOperation
executeQueryOperation--》

2.1、operation转成Transformation

List<Transformation<?>> transformations = translate(Collections.singletonList(sinkOperation));--》

org.apache.flink.table.planner.delegation.PlannerBase#translate

2.2、Transformation转成pipeline

Pipeline pipeline =
                execEnv.createPipeline(
                        transformations, tableConfig.getConfiguration(), defaultJobName);

2.3、pipeline转成streamGraph

final StreamGraph streamGraph = executionEnvironment.generateStreamGraph(transformations); // StreamGraph是Pipeline接口的实现类

3、将StreamGraph转成JobGraph

org.apache.flink.streaming.api.environment.StreamExecutionEnvironment#executeAsync(org.apache.flink.streaming.api.graph.StreamGraph)

 JobClient jobClient = execEnv.executeAsync(pipeline);
3.1、final PipelineExecutor executor = getPipelineExecutor();

        CompletableFuture<JobClient> jobClientFuture =
                executor.execute(streamGraph, configuration, userClassloader);
                
3.2、createJobGraph#new StreamingJobGraphGenerator(
                            userClassLoader, streamGraph, jobID, serializationExecutor)
                    .createJobGraph();

                    
4、将jobGraph提交到集群

org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor#execute

4.1、获取集群信息

final ClusterClientProvider<ClusterID> clusterClientProvider =
                    clusterDescriptor.deployJobCluster(

                           clusterSpecification, jobGraph, configAccessor.getDetachedMode

4.2、提交任务到集群

return CompletableFuture.completedFuture(
                    new ClusterClientJobClientAdapter<>(
                            clusterClientProvider, jobGraph.getJobID(), userCodeClassloader));