Drill代码分析

Drillbit

         ZKClusterCoordinator//协调客户端请求

         ServiceEngine//包括用户服务，控制服务，数据服务

                   UserServer//接受和反馈用户请求

                            UserWorker.submitWork//将用户请求提交到WorkManager中执行

                   ControlServer//控制查询片段的状态管理

                            ControlHandlerImpl.handle//处理控制服务请求

                                     startNewRemoteFragment//为接收到的非根查询片段启动片段管理器或者片段执行器

                                               FragmentExecutor//查询片段执行器

                                                        RootExec//查询片段执行的起点

                                                                 SingleSenderRootExec.innerNext//结果遍历

                                                                           DataTunnel.sendRecordBatch//用数据客户端把执行结果发送到数据服务器

                                               NonRootFragmentManager//内部查询片段管理器

                                                        IncomingBuffers//内部查询片段所依赖的数据缓冲

                                                                 Receiver//每个数据缓冲对应一个receiver，每个receiver需要接收多个sender的反馈结果数据

                                                                 DataCollector//为每个receiver配置一个datacollector，接收sender返回的结果。

                                                                           RawBatchBuffer//每个sender返回的结果

                                                                                    RawFragmentBatch//每个sender返回结果的一部分，结果可能比较大，需要多次发送。

                                                                                             FragmentRecordBatch//RawFragmentBatch对应的protobuf包装

                                                                                                       Message//底层protobuf消息体

                                               WorkEventBus.setFragmentManager//WorkEventBus是ControlServer和DataServer的信息传递通道，传递FragmentManager状态信息

                                     receivingFragmentFinished//查询片段接收完毕，状态改变

                                               FragmentExecutor.receivingFragmentFinished//触发内部查询片段开始执行

                                                        RootExec//查询片段执行的起点

                   DataServer//处理远程内部查询片段返回的结果

                            WorkEventBus.getFragmentManager//从WorkEventBus通道中获取本节点对应的ControlServer中FragmentManager的信息

                            DataResponseHandlerImpl.handle//处理接收到的查询片段结果

                                      NonRootFragmentManager.handle//接收数据出发NonRootFragmentManager状态改变

                                               IncomingBuffers.batchArrived//IncomingBuffers状态改变

                                                        RawFragmentBatch//级联触发RawFragmentBatch改变

                                               FragmentExecutor//如果条件满足，触发查询片段执行

         Server//ip:8047地址对应的jetty服务器，可查看当前系统状态。

                   ServletHolder//servlet对应jetty服务器对应的各种服务。

---------------------------------------------------------------------------------------------

         WorkManager//负责本节点作为主节点所对应的所有用户查询的管理

                   UserWorker//对于用户查询，为查询启动相关资源和管理器

                            Foreman//为每个用户查询分配的管理工头，负责完成此次查询

                                     DrillSqlWorker.getPlan//获取用户查询对应的物理执行计划

                                               Frameworks.newConfigBuilder//为用户查询配置执行上下文

                                                        DrillDistributionTraitDef//查询需要满足此类中的分布特性

                                                        ruleSets//设置物理计划的生成需要经过这些规则集的转换

                                                                 DrillSqlWorker.getRules//获取所有规则集

                                                                           DrillRuleSets.getDrillBasicRules//获取对逻辑执行计划进行转换的规则集

                                                                                    RelOptRule//关系表达式优化规则

                                                                           DrillRuleSets.getPhysicalRules//获取对物理执行计划进行转换的规则集

                                                                                    Prule//物理执行计划转换规则的基类

                                                                           StoragePluginRegistry.getStoragePluginRuleSet//获取存储插件中定制的规则集

                                                                                    StoragePluginOptimizerRule//存储插件规则需要实现的基类

                                                                                             HivePushPartitionFilterIntoScan//hive存储插件中定制的一个规则

                                                                 RuleSetProgram//规则集的执行器。

                                               PlannerImpl.parse//calcite中负责sql词法和语法解析的入口

                                                        SqlParserImpl.parseSqlStmtEof//calcite中语法解析实现类

                                               DefaultSqlHandler//sql抽象语法树生成的后继操作

                                                        validateNode//校验节点

                                                                 PlannerImpl.validate//calcite中实际的校验方法

                                                        convertToRel//把calcite的抽象语法树转换为calcite的代数表达式

                                                                 SqlToRelConverter//实际转换执行的类

                                                        convertToDrel//把calcite的代数表达式转换为drill中的逻辑执行计划

                                                                 PlannerImpl.transform//calcite中转换的入口

                                                                           RuleSetProgram.run//逻辑执行计划每个转换规则执行的入口

                                                                                  VolcanoPlanner.setRoot.registerImpl.fireRules//如果逻辑执行中节点满足条件，触发转换执行

                                                                                             VolcanoRuleCall.match.matchRecurse.onMatch//递归对整个节点应用转换规则

                                                                                                       RelOptRule.onMatch//转换规则类中具体的转换规则

                                                                                                                VolcanoRuleCall.convert//转换规则操作的节点的输入

                                                                                                                         VolcanoPlanner.changeTraits//应用节点需要满足的特性，特性满足过程包含转换规则

                                                                                                                                   RelSet.getOrCreateSubset//生成转换后的节点

                                                                                                                                            VolcanoPlanner.register.registerImpl//执行转换

                                                                                                                DrillRel//生成转换后的逻辑执行计划节点

                                                                                                                VolcanoRuleCall.transformTo//把生成的逻辑执行计划节点放到逻辑计划树中

                                                                                                                         RelTraitPropagationVisitor//把当前逻辑计划所需要满足的特性传播到后裔节点中

                                                                                                                         VolcanoPlanner.ensureRegistered.register.registerImpl//触发特性满足的执行

                                                                                                                                   RelNode.onRegister//对子孙节点执行转换规则

                                                        convertToPrel//转换为物理关系表达式节点，转换调用的方法和逻辑执行计划的转化类似。

                                                                 PlannerImpl.transform//calcite中转换的入口

                                                                           ...

                                                                                    Prule//物理计划转换规则的基类

                                                        convertToPop//转换为物理操作符

                                                                 Prel.getPhysicalOperator//调用物理操作符实现类中的此方法实现转换

                                                                           PhysicalOperator//生成物理操作符

                                                                           PhysicalPlanCreator.addMetadata//为物理操作符添加操作优化所需要的元数据

                                                                                    Prel.getRows//获取物理操作对应的返回行数

                                                                                             Filter.estimateFilteredRows//获得近似的行数

                                                                                                       RelMdSelectivity.getSelectivity//获取一次扫描可能返回结果的比例。

                                                                                             GroupScan.getScanStats//获取表扫描的统计结果

                                                                                             ...

                                                        convertToPlan//生成物理执行计划

                                                                 PhysicalPlan//新建物理执行计划

                                                                           Graph//构建物理执行计划对应的有向无环图

                                     getQueryWorkUnit//将物理执行计划拆分成查询片段

                                               StatsCollector//查询拆分前需要建立统计信息收集器

                                                        Stats//具体统计

                                                                 addMaxWidth//收集一个统计项

                                                                           getMaxParallelizationWidth//收集一个具体的统计项，一般在具体物理操作符中收集

                                                                 addCost//收集代价，一个操作符的代价是后裔代价的累加

                                                        Wraper//查询片段的包装

                                                                 addEndpointAffinity//为一个查询片段选择数据和计算在物理上相近的节点

                                                                           GroupScan.getOperatorAffinity//在表扫描中获取这些统计信息。

                                               SimpleParallelizer.getFragments//生成查询片段

                                                        assignEndpoints//为查询片段制定要发送到的远程节点

                                                                 Wraper.assignEndpoints//根据StatsCollector收集的统计信息确定这些对应的远程节点

                                               Materializer//把生成的查询片段转换为json形式，方便通过rpc发送出去。

                                                        visitGroupScan//物化的时候当访问到组操作节点时，需要把组操作转换为子组扫描的操作

                                                                 GroupScan.getSpecificScan//获取一个子组

                                                                           SubScan//对应一个具体的子组扫描操作，一般在存储插件中实现

                                     setupRootFragment//为用户查询建立根查询片段管理器

                                               RootFragmentManager//在本节点上建立根查询片段管理器

                                                        FragmentExecutor//根查询片段的依赖条件都满足则触发根查询片段的执行

                                     setupNonRootFragments//在系统中传递查询片段图中的非根查询片段

                                               Controller.sendFragments//把非根查询片段发送到对应的远程drill节点上执行。

---------------------------------------------------------------------------------------------