Presto源码解析--提交查询

本文的Presto是基于330版本

提交查询的步骤

presto的连接方式可以有两种，分别是CLI形式的和JDBC形式的，分别对应源码中的presto-cli模块和presto-jdbc模块，后续真正提交到presto coordinator对应到presto-client模块。本文以CLI来讲解Presto如何提交查询。

Presto客户端对查询语句的提交主要分为以下3个步骤：

1. 从制定的文件、命令行参数或者Cli窗口中获取需要执行的Sql语句。
2. 将得到的Sql语句组装成一个RESTful请求，发送给Coordinator，并处理返回的饿response。
3. Cli会不停的循环分批读取查询结果并在屏幕进行动态显示，直到查询结果完全显示。

Cli方式

java  -jar presto-cli-330-xxx-SNAPSHOT-executable.jar  --server localhost:8080 --catalog hive --source dispatcher --client-tags adhoc  --user dispatcher --client-request-timeout 60m --session implicit_conversion=true

之后就会进入Presto的客户端

时序图

大体的执行流程

源码分析

执行上面的可执行jar的main class实际上是io.prestosql.cli.Presto这个类，大体上的意思就通过airline解析命令行的参数，然后运行**console.run()**方法。

package io.prestosql.cli;

import static io.airlift.airline.SingleCommand.singleCommand;

public final class Presto
{
    private Presto() {}

    public static void main(String[] args)
    {
        //根据传递的参数初始化一个Console对象，该对象中保存了启动Cli时传入的参数
        Console console = singleCommand(Console.class).parse(args);
		//如果启动的时候使用了--help或者--version则会显示帮助信息或者版本信息，然后直接退出
        if (console.helpOption.showHelpIfRequested() ||
                console.versionOption.showVersionIfRequested()) {
            return;
        }
		//进入主程序，然后根据启动Cli传入的不同参数进行不同的处理
        System.exit(console.run() ? 0 : 1);
    }
}

进入Console类的run方法中，其中大体逻辑就是先分析Cli启动时有没指定了–execute或者–file参数，如果指定了就执行executeCommand方法，否则就执行runConsole方法，这两个方法最终都会进入Console类的process方法中

public boolean run()
    {
    	//获取上面命令行中的所有参数，封装到ClientSession对象中
        ClientSession session = clientOptions.toClientSession();
        //判断有没--execute and --file 参数
        boolean hasQuery = clientOptions.execute != null;
        boolean isFromFile = !isNullOrEmpty(clientOptions.file);
		//初始化日志
        initializeLogging(clientOptions.logLevelsFile);

        String query = clientOptions.execute;
        if (hasQuery) {
            query += ";";
        }

        if (isFromFile) {
        	//--execute and --file 两个参数不能同时出现
            if (hasQuery) {
                throw new RuntimeException("both --execute and --file specified");
            }
            try {
                query = asCharSource(new File(clientOptions.file), UTF_8).read();
                //这里hasQuery的意义不再是是否有--execute参数，而是变成了--  file中有具体执行语句
                hasQuery = true;
            }
            catch (IOException e) {
                throw new RuntimeException(format("Error reading from file %s: %s", clientOptions.file, e.getMessage()));
            }
        }

        // abort any running query if the CLI is terminated
        AtomicBoolean exiting = new AtomicBoolean();
        ThreadInterruptor interruptor = new ThreadInterruptor();
        CountDownLatch exited = new CountDownLatch(1);
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            exiting.set(true);
            interruptor.interrupt();
            awaitUninterruptibly(exited, EXIT_DELAY.toMillis(), MILLISECONDS);
        }));

        try (QueryRunner queryRunner = new QueryRunner(
                //...) {
            //这里是如果指定--execute或者--file参数，则执行方法
            if (hasQuery) {
                return executeCommand(
                        queryRunner,
                        exiting,
                        query,
                        clientOptions.outputFormat,
                        clientOptions.ignoreErrors,
                        clientOptions.progress);
            }
			//这里是启动时没有指定--execute和--file参数时，启动Cli窗口接受用户输入，分析sql语句并执行
            runConsole(queryRunner, exiting);
            return true;
        }
        finally {
            exited.countDown();
            interruptor.close();
        }
    }

这里看下有–execute或者–file参数时进入的executeCommand方法

private static boolean executeCommand(
            QueryRunner queryRunner,
            AtomicBoolean exiting,
            String query,
            OutputFormat outputFormat,
            boolean ignoreErrors,
            boolean showProgress)
    {
        boolean success = true;
        //对输入的sql语句进行切分，默认分隔符是;
        StatementSplitter splitter = new StatementSplitter(query);
        for (Statement split : splitter.getCompleteStatements()) {
            if (!isEmptyStatement(split.statement())) {
                try (Terminal terminal = terminal()) {
                	//这里对一条完整的sql语句进行处理
                    if (!process(queryRunner, split.statement(), outputFormat, () -> {}, false, showProgress, terminal, System.out, System.err)) {
                        if (!ignoreErrors) {
                            return false;
                        }
                        success = false;
                    }
                }
                catch (IOException e) {
                    throw new UncheckedIOException(e);
                }
            }
            if (exiting.get()) {
                return success;
            }
        }
        if (!isEmptyStatement(splitter.getPartialStatement())) {
            System.err.println("Non-terminated statement: " + splitter.getPartialStatement());
            return false;
        }
        return success;
    }

 private static boolean process(
            QueryRunner queryRunner,
            String sql,
            OutputFormat outputFormat,
            Runnable schemaChanged,
            boolean usePager,
            boolean showProgress,
            Terminal terminal,
            PrintStream out,
            PrintStream errorChannel)
    {
        String finalSql;
        try {
        	//对查询的语句进行预处理
            finalSql = preprocessQuery(
                    terminal,
                    Optional.ofNullable(queryRunner.getSession().getCatalog()),
                    Optional.ofNullable(queryRunner.getSession().getSchema()),
                    sql);
        }
        catch (QueryPreprocessorException e) {
            //  ...
        }
		//真正执行查询的逻辑
        try (Query query = queryRunner.startQuery(finalSql)) {
        	//输出结果
            boolean success = query.renderOutput(terminal, out, errorChannel, outputFormat, usePager, showProgress);

            ClientSession session = queryRunner.getSession();

            // update catalog and schema if present
            if (query.getSetCatalog().isPresent() || query.getSetSchema().isPresent()) {
                session = ClientSession.builder(session)
                          .withCatalog(query.getSetCatalog().orElse(session.getCatalog()))
                          .withSchema(query.getSetSchema().orElse(session.getSchema()))
                          .build();
            }
			//  ...
            return success;
        }
        catch (RuntimeException e) {
            System.err.println("Error running command: " + e.getMessage());
            if (queryRunner.isDebug()) {
                e.printStackTrace(System.err);
            }
            return false;
        }
    }

QueryRunner的startQuery方法中new Query中又调用了startInternalQuery方法，startInternalQuery方法中又调用了newStatementClient方法，newStatementClient又new StatementClientV1(httpClient, session, query)，最终工作都交给了StatementClientV1。

// QueryRunner的处理处理
public class QueryRunner
        implements Closeable {
	 public Query startQuery(String query)
	    {
	        return new Query(startInternalQuery(session.get(), query), debug);
	    }
	
	    public StatementClient startInternalQuery(String query)
	    {
	        return startInternalQuery(stripTransactionId(session.get()), query);
	    }
	
	    private StatementClient startInternalQuery(ClientSession session, String query)
	    {
	        OkHttpClient.Builder builder = httpClient.newBuilder();
	        sslSetup.accept(builder);
	        OkHttpClient client = builder.build();
	        return newStatementClient(client, session, query);
	    }
    }

StatementClientV1中

public StatementClientV1(OkHttpClient httpClient, ClientSession session, String query)
    {
        //  ...
        
        //生成发给coordinator上的//v1/statement接口进行处理的请求
        Request request = buildQueryRequest(session, query);
		//执行请求
        JsonResponse<QueryResults> response = JsonResponse.execute(QUERY_RESULTS_CODEC, httpClient, request);
        if ((response.getStatusCode() != HTTP_OK) || !response.hasValue()) {
            state.compareAndSet(State.RUNNING, State.CLIENT_ERROR);
            throw requestFailedException("starting query", request, response);
        }
		//处理返回结果
        processResponse(response.getHeaders(), response.getValue());
    }