Flink源码阅读之Flinksql执行流程

最近工作中需要开发一些新的Flink sql 的connector，所以先开始研究研究Flink sql的执行流程。

基本结构

Planner接口
负责sql解析、转换成Transformation
Executor接口
负责将planner转换的Transformation生成streamGraph并执行

public interface Planner {
   

	/**
	 * Retrieves a {@link Parser} that provides methods for parsing a SQL string.
	 *
	 * @return initialized {@link Parser}
	 */
	Parser getParser();

	/**
	 * Converts a relational tree of {@link ModifyOperation}s into a set of runnable
	 * {@link Transformation}s.
	 *
	 * <p>This method accepts a list of {@link ModifyOperation}s to allow reusing common
	 * subtrees of multiple relational queries. Each query's top node should be a {@link ModifyOperation}
	 * in order to pass the expected properties of the output {@link Transformation} such as
	 * output mode (append, retract, upsert) or the expected output type.
	 *
	 * @param modifyOperations list of relational operations to plan, optimize and convert in a
	 * single run.
	 * @return list of corresponding {@link Transformation}s.
	 */
	List<Transformation<?>> translate(List<ModifyOperation> modifyOperations);

	/**
	 * Returns the AST of the specified Table API and SQL queries and the execution plan
	 * to compute the result of the given collection of {@link QueryOperation}s.
	 *
	 * @param operations The collection of relational queries for which the AST
	 * and execution plan will be returned.
	 * @param extended if the plan should contain additional properties such as
	 * e.g. estimated cost, traits
	 */
	String explain(List<Operation> operations, boolean extended);

	/**
	 * Returns completion hints for the given statement at the given cursor position.
	 * The completion happens case insensitively.
	 *
	 * @param statement Partial or slightly incorrect SQL statement
	 * @param position cursor position
	 * @return completion hints that fit at the current cursor position
	 */
	String[] getCompletionHints(String statement, int position);
}

Sql解析

Parser接口
负责sql解析
有两个实现一个是old planner，另一个是blink planner
flink对sql的解析依赖于calcite

具体实现

@Override
	public List<Operation> parse(String statement) {
   
		CalciteParser parser = calciteParserSupplier.get();
		FlinkPlannerImpl planner = validatorSupplier.get();
		// parse the sql query
		//依赖calcite将sql语句解析为sqlNode
		SqlNode parsed = parser.parse(statement);

        //将sqlnode转换为Operation
		Operation operation = SqlToOperationConverter.convert(planner, catalogManager, parsed)
			.orElseThrow(() -> new TableException("Unsupported query: " + statement));
		return Collections.singletonList(operation);
	}
	
	/**
	 * This is the main entrance for executing all kinds of DDL/DML {@code SqlNode}s, different
	 * SqlNode will have it's implementation in the #convert(type) method whose 'type' argument
	 * is subclass of {@code SqlNode}.
	 *
	 * @param flinkPlanner FlinkPlannerImpl to convertCreateTable sql node to rel node
	 * @param catalogManager CatalogManager to resolve full path for operations
	 * @param sqlNode SqlNode to execute on
	 */
	public static Optional<Operation> convert(
			FlinkPlannerImpl flinkPlanner,
			CatalogManager catalogManager,
			SqlNode sqlNode) {
   
		// validate the query
		// 校验sql的合法性
		final SqlNode validated = flinkPlanner.validate(sqlNode);
		SqlToOperationConverter converter = new SqlToOperationConverter(flinkPlanner, catalogManager);
		//对不同的ddl/dml进行转换
		if (validated instanceof SqlCreateTable) {
   
			return Optional.of(converter.convertCreateTable((SqlCreateTable) validated));
		} else if (validated instanceof SqlDropTable) {
   
			return Optional.of(converter.convertDropTable((SqlDr