[4] Hive3.x SemanticAnalyzer and CalcitePlanner 物化视图相关源码-01

接上文Hive3.x 查询流程源码-Cli端-01

查询Hive3.x Materialized view中构建的物化视图的例子, debug查看详细执行过程
1）查询语句

SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
SET hive.support.concurrency=true;
SET hive.enforce.bucketing=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.compactor.initiator.on=true;
SET hive.compactor.worker.threads=2;
SET hive.query.results.cache.enabled=false;
SELECT  deptno, count(1) as deptno_cnt from depts group by deptno;

2）查询语句

hive> SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
hive> SET hive.support.concurrency=true;
hive> SET hive.enforce.bucketing=true;
hive> SET hive.exec.dynamic.partition.mode=nonstrict;
hive> SET hive.compactor.initiator.on=true;
hive> SET hive.compactor.worker.threads=2;
hive> SET hive.query.results.cache.enabled=false;
hive> explain SELECT  deptno, count(1) as deptno_cnt from depts group by deptno;
FAILED: SemanticException [Error 10001]: Line 1:52 Table not found 'depts'
hive> explain SELECT  deptno, count(1) as deptno_cnt from hive3_test.depts group by deptno;
OK
STAGE DEPENDENCIES:
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        TableScan
          alias: hive3_test.depts_agg
          Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE Column stats: NONE
          Select Operator
            expressions: deptno (type: int), deptno_cnt (type: bigint)
            outputColumnNames: _col0, _col1
            Statistics: Num rows: 2 Data size: 24 Basic stats: COMPLETE Column stats: NONE
            ListSink

Time taken: 1.806 seconds, Fetched: 17 row(s)

SemanticAnalyzer

void analyzeInternal(ASTNode ast, PlannerContextFactory pcf) {
     ....
     // 1. Generate Resolved Parse tree from syntax tree
    boolean needsTransform = needsTransform();
    
    // 2. Gen OP Tree from resolved Parse Tree
    Operator sinkOp = genOPTree(ast, plannerCtx);//进入CalcitePlanner::getOPTree
    //---待续
}

CalcitePlanner::getOPTree

这里入参为hive的ASTNode
Operator genOPTree(ASTNode ast, PlannerContext plannerCtx){

      ......
      // 1. Gen Optimized AST
         ASTNode newAST = getOptimizedAST();
      

}

CalcitePlanner::getOptimizedAST

 ASTNode getOptimizedAST() throws SemanticException {
    //用calcite优化查询，生成calcite的RelNode
    RelNode optimizedOptiqPlan = logicalPlan();
    //将RelNode转化为hive的ASTNode
    ASTNode optiqOptimizedAST = ASTConverter.convert(optimizedOptiqPlan, resultSchema,
            HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_COLUMN_ALIGNMENT));
    return optiqOptimizedAST;
  }

CalcitePlanner:: logicalPlan

RelNode logicalPlan() throws SemanticException {
    RelNode optimizedOptiqPlan = null;
    CalcitePlannerAction calcitePlannerAction = null;
  
    /**
     * Map of table name to names of accessed columns
     */
    Map<String, Set<String>>  this.columnAccessInfo = new ColumnAccessInfo();
    /**
     * CalcitePlannerAction is code responsible for Calcite plan generation and optimization.
     */
    calcitePlannerAction = new CalcitePlannerAction(
        prunedPartitions,
        ctx.getOpContext().getColStatsCache(),
        this.columnAccessInfo);
    //calcite 优化plan，主要优化工作在CalcitePlanner::CalcitePlannerAction::apply()
    optimizedOptiqPlan = Frameworks.withPlanner(calcitePlannerAction, Frameworks
          .newConfigBuilder().typeSystem(new HiveTypeSystemImpl()).build());
    return optimizedOptiqPlan;
  }

CalcitePlanner::CalcitePlannerAction::apply()

待续。。。