Spark 之 ProjectExec，

最新推荐文章于 2024-07-22 14:46:41 发布

zhixingheyi_tian

最新推荐文章于 2024-07-22 14:46:41 发布

阅读量90

点赞数

分类专栏： spark 文章标签： spark scala 大数据

本文链接：https://blog.csdn.net/zhixingheyi_tian/article/details/128374151

版权

spark 专栏收录该内容

107 篇文章 4 订阅

订阅专栏

这篇博客深入探讨了Spark SQL中的`ProjectExec`和`RDDScanExec`操作。通过`Testselect10`测试用例展示了如何执行SQL查询，解释了`ProjectExec`如何创建投影并初始化，以及`RDDScanExec`如何从RDD扫描数据并生成内部行。内容涉及到Spark SQL的执行计划、数据处理及底层代码实现。

摘要由CSDN通过智能技术生成

select 10， case


  test("Test select 10") {
    val df = sql("SELECT  10 ")
    df.explain(false)
    df.show()
    df.explain(false)
  }

ProjectExec

./sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala

case class ProjectExec(projectList: Seq[NamedExpression], child: SparkPlan)

执行函数

protected override def doExecute(): RDD[InternalRow] = {
    child.execute().mapPartitionsWithIndexInternal { (index, iter) =>
      val project = UnsafeProjection.create(projectList, child.output)
      project.initialize(index)
      iter.map(project)
    }
  }

/**
   * Returns an UnsafeProjection for given sequence of Expressions, which will be bound to
   * `inputSchema`.
   */
  def create(exprs: Seq[Expression], inputSchema: Seq[Attribute]): UnsafeProjection = {
    create(bindReferences(exprs, inputSchema))
  }

RDDScanExec 饭后的的inputSchema 是空

child.execute().mapPartitionsWithIndexInternal RDDScanExec ，返回只有一个空 row， rowcount 为1

RDDScanExec

./sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala

/** Physical plan node for scanning data from an RDD of InternalRow. */
case class RDDScanExec(
    output: Seq[Attribute],
    rdd: RDD[InternalRow],
    name: String,
    override val outputPartitioning: Partitioning = UnknownPartitioning(0),
    override val outputOrdering: Seq[SortOrder] = Nil) extends LeafExecNode with InputRDDCodegen {

protected override def doExecute(): RDD[InternalRow] = {
    val numOutputRows = longMetric("numOutputRows")
    rdd.mapPartitionsWithIndexInternal { (index, iter) =>
      val proj = UnsafeProjection.create(schema)
      proj.initialize(index)
      iter.map { r =>
        numOutputRows += 1
        proj(r)
      }
    }
  }