第66课：SparkSQL下Parquet中PushDown的实现学习笔记

第66课：SparkSQL下Parquet中PushDown的实现学习笔记

本期内容：

1  SparkSQL下的PushDown的价值

2  SparkSQL下的Parquet的PuahDown实现

 

Hive中也有PushDown。PushDown可以极大减少数据输入，极大的提高处理效率。

SparkSQL实现了PushDown，在Parquet文件中实现PushDown具有很重要的意义。

PushDown是一种SQL优化方式，通常用在查询。应用场景：

假设通过DataFrame，df.select(a,b,c).filter(by a).filter(by b).select(c).filter(by c)这样的查询，在optimizer阶段，需要合并多个filters(CombineFilters)，并调整算子间的顺序，例如将部分filter移到select等前面(PushPredicateThroughAggregate/Generate/Join/Project)。filter前需要操作一大批数据，但filter后只需要操作很小一部分数据，SQL优化时就希望一开始就只操作这一小部分数据，而不需要把所有数据都导入进来，因为最终还是要被过滤掉。

PushDown本身既有SQL语法的层面也有物理执行的层面。

语法层面，SparkSQL和Hive都有自己的语法实现。

下面看一下QueryExecution的源码：

/**
 * The primary workflow for executing relational queries using Spark.  Designed to allow easy
 * access to the intermediate phases of query execution for developers.
 *
 * While this is not a public class, we should avoid changing the function names for the sake of
 * changing them, because a lot of developers use the feature for debugging.
 */
class QueryExecution(valsqlContext: SQLContext, vallogical: LogicalPlan) {

  def assertAnalyzed(): Unit = sqlContext.analyzer.checkAnalysis(analyzed)

  lazy val analyzed: LogicalPlan = sqlContext.analyzer.execute(logical)

  lazy val withCachedData: LogicalPlan = {
    assertAnalyzed()
    sqlContext.cacheManager.useCachedData(analyzed)
  }

  lazy val optimizedPlan: LogicalPlan = sqlContext.optimizer.execute(withCachedData)

  lazy val sparkPlan: SparkPlan = {
    SQLContext.setActive(sqlContext)
    sqlContext.planner.plan(optimizedPlan).next()
  }

  // executedPlan should not be used to initialize any SparkPlan. It should be
  // only used for execution.
  lazy valexecutedPlan: SparkPlan = sqlContext.prepareForExecution.execute(sparkPlan)

  /** Internal version of the RDD. Avoids copies and has no schema */
  lazy valtoRdd: RDD[InternalRow] =executedPlan.execute()

  protected def stringOrError[A](f: =>A): String =
    try f.toString catch { casee: Throwable => e.toString }

  def simpleString: String = {
    s"""== Physical Plan ==
       |${stringOrError(executedPlan)}
      """.stripMargin.trim
  }

  override def toString:String = {
    def output =
      analyzed.output.map(o =>s"${o.name}:${o.dataType.simpleString}").mkString(", ")

    s"""== Parsed Logical Plan ==
       |${stringOrError(logical)}
       |== Analyzed Logical Plan ==
       |${stringOrError(output)}
       |${stringOrError(analyzed)}
       |== Optimized Logical Plan ==
       |${stringOrError(optimizedPlan)}
       |== Physical Plan ==
       |${stringOrError(executedPlan)}
    """.stripMargin.trim
  }
}

QueryExecution在具体实现时会把工作过程串联成一个WorkFlow。

SQL语句的翻译过程：

1 基本语法翻译

2 pharser翻译

3 优化

4 Logical plan

5 Physical执行计划

6 引擎上执行

例如DataFrame，df.select(a,b,c).filter(by a).filter(by b).select(c).filter(by c)这样一个SQL语句，在执行前会生成一个语法树，解析和优化，在优化阶段会把Filter合并，在合并时会考虑Filter的顺序。

下面再看一下spark.sql.catalyst的Optimizer的源码：

package org.apache.spark.sql.catalyst.optimizer

abstract class Optimizerextends RuleExecutor[LogicalPlan]

object DefaultOptimizer extends Optimizer {
  val batches=
    // SubQueries are only needed for analysis and can be removed before execution.
    Batch("Remove SubQueries",FixedPoint(100),
      EliminateSubQueries) ::
    Batch("Aggregate",FixedPoint(100),
      ReplaceDistinctWithAggregate,
      RemoveLiteralFromGroupExpressions) ::
    Batch("Operator Optimizations",FixedPoint(100),
      // Operator push down
      SetOperationPushDown,
      SamplePushDown,
      PushPredicateThroughJoin,
      PushPredicateThroughProject,
      PushPredicateThroughGenerate,
      PushPredicateThroughAggregate,
      ColumnPruning,
      // Operator combine
      ProjectCollapsing,
      CombineFilters,
      CombineLimits,
      // Constant folding
      NullPropagation,
      OptimizeIn,
      ConstantFolding,
      LikeSimplification,
      BooleanSimplification,
      RemoveDispensableExpressions,
      SimplifyFilters,
      Simpli