sparksql 解析

最新推荐文章于 2024-01-04 11:33:18 发布
最新推荐文章于 2024-01-04 11:33:18 发布
阅读量407 收藏
点赞数
分类专栏： spark

1.整体运行流程

使用下列代码对SparkSQL流程进行分析，让大家明白LogicalPlan的几种状态，理解SparkSQL整体执行流程

<code class="hljs vhdl has-numbering">// sc <span class="hljs-keyword">is</span> an existing SparkContext.
val sqlContext = <span class="hljs-keyword">new</span> org.apache.spark.sql.SQLContext(sc)
// this <span class="hljs-keyword">is</span> used <span class="hljs-keyword">to</span> implicitly convert an RDD <span class="hljs-keyword">to</span> a DataFrame.
import sqlContext.implicits._

// Define the schema using a <span class="hljs-keyword">case</span> class.
// Note: <span class="hljs-keyword">Case</span> classes <span class="hljs-keyword">in</span> Scala <span class="hljs-number">2.10</span> can support only up <span class="hljs-keyword">to</span> <span class="hljs-number">22</span> fields. <span class="hljs-keyword">To</span> work around this limit,
// you can <span class="hljs-keyword">use</span> custom classes that implement the Product interface.
<span class="hljs-keyword">case</span> class Person(name: <span class="hljs-typename">String</span>, age: Int)

// Create an RDD <span class="hljs-keyword">of</span> Person objects <span class="hljs-keyword">and</span> <span class="hljs-keyword">register</span> it as a table.
val people = sc.textFile(<span class="hljs-string">"/examples/src/main/resources/people.txt"</span>).<span class="hljs-keyword">map</span>(_.split(<span class="hljs-string">","</span>)).<span class="hljs-keyword">map</span>(p => Person(p(<span class="hljs-number">0</span>), p(<span class="hljs-number">1</span>).trim.toInt)).toDF()
people.registerTempTable(<span class="hljs-string">"people"</span>)

// SQL statements can be run by using the sql methods provided by sqlContext.
val teenagers = sqlContext.sql(<span class="hljs-string">"SELECT name, age FROM people WHERE age >= 13 AND age <= 19"</span>)
</code>

（1）查看teenagers的Schema信息

<code class="hljs vbnet has-numbering">scala> teenagers.printSchema
root
 |-- name: <span class="hljs-built_in">string</span> (nullable = <span class="hljs-literal">true</span>)
 |-- age: <span class="hljs-built_in">integer</span> (nullable = <span class="hljs-literal">false</span>)</code>

（2）查看运行流程

<code class="hljs asciidoc has-numbering">scala> teenagers.queryExecution
res3: org.apache.spark.sql.SQLContext#QueryExecution =
<span class="hljs-header">== Parsed Logical Plan ==</span>
<span class="hljs-emphasis">'Project [unresolvedalias('</span>name),unresolvedalias(<span class="hljs-emphasis">'age)]
 '</span>Filter ((<span class="hljs-emphasis">'age >= 13) && ('</span>age <= 19))
<span class="hljs-code">  'UnresolvedRelation [people], None</span>

<span class="hljs-header">== Analyzed Logical Plan ==</span>
name: string, age: int
Project [name#0,age#1]
<span class="hljs-code"> Filter ((age#1 >= 13) && (age#1 <= 19))</span>
<span class="hljs-code">  Subquery people</span>
<span class="hljs-code">   LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-header">== Optimized Logical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-header">== Physical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> Scan PhysicalRDD[name#0,age#1]</span>

Code Generation: true
</code>

QueryExecution中表示的是整体Spark SQL运行流程，从上面的输出结果可以看到，一个SQL语句要执行需要经过下列步骤：

<code class="hljs asciidoc has-numbering"><span class="hljs-header">== （1）Parsed Logical Plan ==</span>
<span class="hljs-emphasis">'Project [unresolvedalias('</span>name),unresolvedalias(<span class="hljs-emphasis">'age)]
 '</span>Filter ((<span class="hljs-emphasis">'age >= 13) && ('</span>age <= 19))
<span class="hljs-code">  'UnresolvedRelation [people], None</span>

<span class="hljs-header">== （2）Analyzed Logical Plan ==</span>
name: string, age: int
Project [name#0,age#1]
<span class="hljs-code"> Filter ((age#1 >= 13) && (age#1 <= 19))</span>
<span class="hljs-code">  Subquery people</span>
<span class="hljs-code">   LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-header">== （3）Optimized Logical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-header">== （4）Physical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> Scan PhysicalRDD[name#0,age#1]</span>

<span class="hljs-comment">//启动动态字节码生成技术（bytecode generation，CG），提升查询效率</span>
Code Generation: true</code>

2.全表查询运行流程

执行语句：

<code class="hljs fsharp has-numbering"><span class="hljs-keyword">val</span> all= sqlContext.sql(<span class="hljs-string">"SELECT * FROM people"</span>)</code>

运行流程：

<code class="hljs asciidoc has-numbering">scala> all.queryExecution
res9: org.apache.spark.sql.SQLContext#QueryExecution =
<span class="hljs-comment">//注意*号被解析为unresolvedalias(*)</span>
<span class="hljs-header">== Parsed Logical Plan ==</span>
<span class="hljs-emphasis">'Project [unresolvedalias(*)]
 '</span>UnresolvedRelation [people], None

<span class="hljs-header">== Analyzed Logical Plan ==</span>
<span class="hljs-comment">//unresolvedalias(*)被analyzed为Schema中所有的字段</span>
<span class="hljs-comment">//UnresolvedRelation [people]被analyzed为Subquery people</span>
name: string, age: int
Project [name#0,age#1]
<span class="hljs-code"> Subquery people</span>
<span class="hljs-code">  LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-header">== Optimized Logical Plan ==</span>
LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22

<span class="hljs-header">== Physical Plan ==</span>
Scan PhysicalRDD[name#0,age#1]

Code Generation: true
</code>

3. filter查询运行流程

执行语句：

<code class="hljs avrasm has-numbering">scala> val filterQuery= sqlContext<span class="hljs-preprocessor">.sql</span>(<span class="hljs-string">"SELECT * FROM people WHERE age >= 13 AND age <= 19"</span>)
<span class="hljs-label">filterQuery:</span> org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.spark</span><span class="hljs-preprocessor">.sql</span><span class="hljs-preprocessor">.DataFrame</span> = [name: string, age: int]</code>

执行流程：

<code class="hljs asciidoc has-numbering">scala> filterQuery.queryExecution
res0: org.apache.spark.sql.SQLContext#QueryExecution =
<span class="hljs-header">== Parsed Logical Plan ==</span>
<span class="hljs-emphasis">'Project [unresolvedalias(*)]
 '</span>Filter ((<span class="hljs-emphasis">'age >= 13) && ('</span>age <= 19))
<span class="hljs-code">  'UnresolvedRelation [people], None</span>

<span class="hljs-header">== Analyzed Logical Plan ==</span>
name: string, age: int
Project [name#0,age#1]
<span class="hljs-code"> //多出了Filter，后同</span>
<span class="hljs-code"> Filter ((age#1 >= 13) && (age#1 <= 19))</span>
<span class="hljs-code">  Subquery people</span>
<span class="hljs-code">   LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:20</span>

<span class="hljs-header">== Optimized Logical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:20</span>

<span class="hljs-header">== Physical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> Scan PhysicalRDD[name#0,age#1]</span>

Code Generation: true</code>

4. join查询运行流程

执行语句：

<code class="hljs sql has-numbering">val joinQuery= sqlContext.sql("<span class="hljs-operator"><span class="hljs-keyword">SELECT</span> * <span class="hljs-keyword">FROM</span> people a, people b <span class="hljs-keyword">where</span> a.age=b.age<span class="hljs-string">")</span></span></code>

查看整体执行流程

<code class="hljs lasso has-numbering">scala<span class="hljs-subst">></span> joinQuery<span class="hljs-built_in">.</span>queryExecution
res0: org<span class="hljs-built_in">.</span>apache<span class="hljs-built_in">.</span>spark<span class="hljs-built_in">.</span>sql<span class="hljs-built_in">.</span>SQLContext<span class="hljs-variable">#QueryExecution</span> <span class="hljs-subst">=</span>
<span class="hljs-comment">//注意Filter</span>
<span class="hljs-comment">//Join Inner</span>
<span class="hljs-subst">==</span> Parsed Logical Plan <span class="hljs-subst">==</span>
<span class="hljs-string">'Project [unresolvedalias(*)]
 '</span>Filter (<span class="hljs-string">'a.age = '</span>b<span class="hljs-built_in">.</span>age)
  <span class="hljs-string">'Join Inner, None
   '</span>UnresolvedRelation <span class="hljs-preprocessor">[</span>people<span class="hljs-preprocessor">]</span><span class="hljs-markup">, Some(a)
   'UnresolvedRelation </span><span class="hljs-preprocessor">[</span>people<span class="hljs-preprocessor">]</span><span class="hljs-markup">, Some(b)

== Analyzed Logical Plan ==
name: string, age: int, name: string, age: int
Project </span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#0</span>,age<span class="hljs-variable">#1</span>,name<span class="hljs-variable">#2</span>,age<span class="hljs-variable">#3</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">
 Filter (age#1 = age#3)
  Join Inner, None
   Subquery a
    Subquery people
     LogicalRDD </span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#0</span>,age<span class="hljs-variable">#1</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">, MapPartitionsRDD</span><span class="hljs-preprocessor">[</span><span class="hljs-number">4</span><span class="hljs-preprocessor">]</span><span class="hljs-markup"> at rddToDataFrameHolder at <console>:22
   Subquery b
    Subquery people
     LogicalRDD </span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#2</span>,age<span class="hljs-variable">#3</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">, MapPartitionsRDD</span><span class="hljs-preprocessor">[</span><span class="hljs-number">4</span><span class="hljs-preprocessor">]</span><span class="hljs-markup"> at rddToDataFrameHolder at <console>:22

== Optimized Logical Plan ==
Project </span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#0</span>,age<span class="hljs-variable">#1</span>,name<span class="hljs-variable">#2</span>,age<span class="hljs-variable">#3</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">
 Join Inner, Some((age#1 = age#3))
  LogicalRDD </span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#0</span>,age<span class="hljs-variable">#1</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">, MapPartitionsRDD</span><span class="hljs-preprocessor">[</span><span class="hljs-number">4</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">...

//查看其Physical Plan
scala> joinQuery.queryExecution.sparkPlan
res16: org.apache.spark.sql.execution.SparkPlan =
TungstenProject </span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#0</span>,age<span class="hljs-variable">#1</span>,name<span class="hljs-variable">#2</span>,age<span class="hljs-variable">#3</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">
 SortMergeJoin </span><span class="hljs-preprocessor">[</span>age<span class="hljs-variable">#1</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">, </span><span class="hljs-preprocessor">[</span>age<span class="hljs-variable">#3</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">
  Scan PhysicalRDD</span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#0</span>,age<span class="hljs-variable">#1</span><span class="hljs-preprocessor">]</span><span class="hljs-markup">
  Scan PhysicalRDD</span><span class="hljs-preprocessor">[</span>name<span class="hljs-variable">#2</span>,age<span class="hljs-variable">#3</span><span class="hljs-preprocessor">]</span><span class="hljs-markup"></span></code>

前面的例子与下面的例子等同，只不过其运行方式略有不同，执行语句：

<code class="hljs avrasm has-numbering">scala> val innerQuery= sqlContext<span class="hljs-preprocessor">.sql</span>(<span class="hljs-string">"SELECT * FROM people a inner join people b on a.age=b.age"</span>)
<span class="hljs-label">innerQuery:</span> org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.spark</span><span class="hljs-preprocessor">.sql</span><span class="hljs-preprocessor">.DataFrame</span> = [name: string, age: int, name: string, age: int]
</code>

查看整体执行流程：

<code class="hljs oxygene has-numbering">scala> innerQuery.queryExecution
res2: org.apache.spark.sql.SQLContext#QueryExecution =
<span class="hljs-comment">//注意Join Inner</span>
<span class="hljs-comment">//另外这里面没有Filter</span>
== Parsed Logical Plan ==
<span class="hljs-string">'Project [unresolvedalias(*)]
 '</span><span class="hljs-keyword">Join</span> Inner, Some((<span class="hljs-string">'a.age = '</span>b.age))
  <span class="hljs-string">'UnresolvedRelation [people], Some(a)
  '</span>UnresolvedRelation [people], Some(b)

== Analyzed Logical Plan ==
name: string, age: int, name: string, age: int
Project [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>,name<span class="hljs-string">#4</span>,age<span class="hljs-string">#5</span>]
 <span class="hljs-keyword">Join</span> Inner, Some((age<span class="hljs-string">#1</span> = age<span class="hljs-string">#5</span>))
  Subquery a
   Subquery people
    LogicalRDD [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>], MapPartitionsRDD[<span class="hljs-number">4</span>] at rddToDataFrameHolder at <console>:<span class="hljs-number">22</span>
  Subquery b
   Subquery people
    LogicalRDD [name<span class="hljs-string">#4</span>,age<span class="hljs-string">#5</span>], MapPartitionsRDD[<span class="hljs-number">4</span>] at rddToDataFrameHolder at <console>:<span class="hljs-number">22</span>

<span class="hljs-comment">//注意Optimized Logical Plan与Analyzed Logical Plan</span>
<span class="hljs-comment">//并没有进行特别的优化，突出这一点是为了比较后面的子查询</span>
<span class="hljs-comment">//其Analyzed和Optimized间的区别</span>
== Optimized Logical Plan ==
Project [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>,name<span class="hljs-string">#4</span>,age<span class="hljs-string">#5</span>]
 <span class="hljs-keyword">Join</span> Inner, Some((age<span class="hljs-string">#1</span> = age<span class="hljs-string">#5</span>))
  LogicalRDD [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>], MapPartitionsRDD[<span class="hljs-number">4</span>] at rddToDataFrameHolder ...

<span class="hljs-comment">//查看其Physical Plan</span>
scala> innerQuery.queryExecution.sparkPlan
res14: org.apache.spark.sql.execution.SparkPlan =
TungstenProject [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>,name<span class="hljs-string">#6</span>,age<span class="hljs-string">#7</span>]
 SortMergeJoin [age<span class="hljs-string">#1</span>], [age<span class="hljs-string">#7</span>]
  Scan PhysicalRDD[name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>]
  Scan PhysicalRDD[name<span class="hljs-string">#6</span>,age<span class="hljs-string">#7</span>]
</code>

5. 子查询运行流程

执行语句：

<code class="hljs avrasm has-numbering">scala> val subQuery=sqlContext<span class="hljs-preprocessor">.sql</span>(<span class="hljs-string">"SELECT * FROM (SELECT * FROM people WHERE age >= 13)a where a.age <= 19"</span>)
<span class="hljs-label">subQuery:</span> org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.spark</span><span class="hljs-preprocessor">.sql</span><span class="hljs-preprocessor">.DataFrame</span> = [name: string, age: int]
</code>

查看整体执行流程：

<code class="hljs asciidoc has-numbering">
scala> subQuery.queryExecution
res4: org.apache.spark.sql.SQLContext#QueryExecution =
<span class="hljs-header">== Parsed Logical Plan ==</span>
<span class="hljs-emphasis">'Project [unresolvedalias(*)]
 '</span>Filter (<span class="hljs-emphasis">'a.age <= 19)
  '</span>Subquery a
<span class="hljs-code">   'Project [unresolvedalias(*)]</span>
<span class="hljs-code">    'Filter ('age >= 13)</span>
<span class="hljs-code">     'UnresolvedRelation [people], None</span>

<span class="hljs-header">== Analyzed Logical Plan ==</span>
name: string, age: int
Project [name#0,age#1]
<span class="hljs-code"> Filter (age#1 <= 19)</span>
<span class="hljs-code">  Subquery a</span>
<span class="hljs-code">   Project [name#0,age#1]</span>
<span class="hljs-code">    Filter (age#1 >= 13)</span>
<span class="hljs-code">     Subquery people</span>
<span class="hljs-code">      LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-comment">//这里需要注意Optimized与Analyzed间的区别</span>
<span class="hljs-comment">//Filter被进行了优化</span>
<span class="hljs-header">== Optimized Logical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> LogicalRDD [name#0,age#1], MapPartitionsRDD[4] at rddToDataFrameHolder at <console>:22</span>

<span class="hljs-header">== Physical Plan ==</span>
Filter ((age#1 >= 13) && (age#1 <= 19))
<span class="hljs-code"> Scan PhysicalRDD[name#0,age#1]</span>

Code Generation: true

</code>

6. 聚合SQL运行流程

执行语句：

<code class="hljs avrasm has-numbering">scala> val aggregateQuery=sqlContext<span class="hljs-preprocessor">.sql</span>(<span class="hljs-string">"SELECT a.name,sum(a.age) FROM (SELECT * FROM people WHERE age >= 13)a where a.age <= 19 group by a.name"</span>)
<span class="hljs-label">aggregateQuery:</span> org<span class="hljs-preprocessor">.apache</span><span class="hljs-preprocessor">.spark</span><span class="hljs-preprocessor">.sql</span><span class="hljs-preprocessor">.DataFrame</span> = [name: string, _c1: bigint]
</code>

运行流程查看：

<code class="hljs oxygene has-numbering">
scala> aggregateQuery.queryExecution
res6: org.apache.spark.sql.SQLContext#QueryExecution =
<span class="hljs-comment">//注意'Aggregate ['a.name], [unresolvedalias('a.name),unresolvedalias('sum('a.age))]</span>
<span class="hljs-comment">//即group by a.name被 parsed为unresolvedalias('a.name)</span>
== Parsed Logical Plan ==
<span class="hljs-string">'Aggregate ['</span>a.name], [unresolvedalias(<span class="hljs-string">'a.name),unresolvedalias('</span>sum(<span class="hljs-string">'a.age))]
 '</span>Filter (<span class="hljs-string">'a.age <= 19)
  '</span>Subquery a
   <span class="hljs-string">'Project [unresolvedalias(*)]
    '</span>Filter (<span class="hljs-string">'age >= 13)
     '</span>UnresolvedRelation [people], None

== Analyzed Logical Plan ==
name: string, _c1: bigint
Aggregate [name<span class="hljs-string">#0</span>], [name<span class="hljs-string">#0</span>,sum(cast(age<span class="hljs-string">#1</span> <span class="hljs-keyword">as</span> bigint)) <span class="hljs-keyword">AS</span> _c1<span class="hljs-string">#9</span>L]
 Filter (age<span class="hljs-string">#1</span> <= <span class="hljs-number">19</span>)
  Subquery a
   Project [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>]
    Filter (age<span class="hljs-string">#1</span> >= <span class="hljs-number">13</span>)
     Subquery people
      LogicalRDD [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>], MapPartitionsRDD[<span class="hljs-number">4</span>] at rddToDataFrameHolder at <console>:<span class="hljs-number">22</span>

== Optimized Logical Plan ==
Aggregate [name<span class="hljs-string">#0</span>], [name<span class="hljs-string">#0</span>,sum(cast(age<span class="hljs-string">#1</span> <span class="hljs-keyword">as</span> bigint)) <span class="hljs-keyword">AS</span> _c1<span class="hljs-string">#9</span>L]
 Filter ((age<span class="hljs-string">#1</span> >= <span class="hljs-number">13</span>) && (age<span class="hljs-string">#1</span> <= <span class="hljs-number">19</span>))
  LogicalRDD [name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>], MapPartitions...

<span class="hljs-comment">//查看其Physical Plan</span>
scala> aggregateQuery.queryExecution.sparkPlan
res10: org.apache.spark.sql.execution.SparkPlan =
TungstenAggregate(key=[name<span class="hljs-string">#0</span>], functions=[(sum(cast(age<span class="hljs-string">#1</span> <span class="hljs-keyword">as</span> bigint)),mode=<span class="hljs-keyword">Final</span>,isDistinct=<span class="hljs-keyword">false</span>)], output=[name<span class="hljs-string">#0</span>,_c1<span class="hljs-string">#14</span>L])
 TungstenAggregate(key=[name<span class="hljs-string">#0</span>], functions=[(sum(cast(age<span class="hljs-string">#1</span> <span class="hljs-keyword">as</span> bigint)),mode=<span class="hljs-keyword">Partial</span>,isDistinct=<span class="hljs-keyword">false</span>)], output=[name<span class="hljs-string">#0</span>,currentSum<span class="hljs-string">#17</span>L])
  Filter ((age<span class="hljs-string">#1</span> >= <span class="hljs-number">13</span>) && (age<span class="hljs-string">#1</span> <= <span class="hljs-number">19</span>))
   Scan PhysicalRDD[name<span class="hljs-string">#0</span>,age<span class="hljs-string">#1</span>]
</code>

其它SQL语句，大家可以使用同样的方法查看其执行流程，以掌握Spark SQL背后实现的基本思想。

fengwuwer
关注
专栏目录
SparkSQL-解析
weixin_44085996的博客
09-22 845
1、新的起始点SparkSession 在老的版本中，SparkSQL提供两种SQL查询起始点，一个叫SQLContext，用于Spark自己提供的SQL查询，一个叫HiveContext，用于连接Hive的查询，SparkSession是Spark最新的SQL查询起始点，实质上是SQLContext和HiveContext的组合，所以在SQLContext和HiveContext上可用的API在SparkSession上同样是可以使用的。SparkSession内部封装了sparkCon...
SparkSQL 解析XML文件并在UDF中传入Map[]类型的参数
王天一的博客
06-25 1658
需求场景如下： 有一份xml文件存储着中图分类号-----> 教育部分类号的映射关系 部分如下图所示 现在需要解析这份xml文件，并根据对应的映射关系，将DF中的中图分类号映射为相应的教育部分类号 1：spark解析xml文件 maven中导入spark-xml对应的依赖 <dependency> <groupId>com.d...
SparkSQL应用解析
11-19
sparkSQL详细解析：包括spark的基础RDD，以及sparkSQL的两大数据抽象 DataFrames与DataSet、sparkSQL数据源、输入输出、自定义函数、自定义分区等。
spark SQL应用解析
11-24
学习sparkSQL详细解析，SparkSQL实现功能，代码实例练习，以及实战， 计算所有订单中每年的销售总数、销售总额 计算所有订单每年最大金额订单的销售额 计算所有订单中每年最畅销货品
SparkSql解析过程
HanJuly9569的博客
03-11 694
引用： Spark SQL架构和原理 Spark SQL 性能优化再进一步 CBO 基于代价的优化 Spark SQL join的三种实现方式 总结 首先用户使用spark.sql 或者 select等api提交sql 转化为未解析的逻辑计划 为什么叫为解析的逻辑计划：因为没有考虑数据的 转化为已解析的逻辑计划 转化为优化后的逻辑计划 转化为物理执行计划 计算物理执行计划的开销模型，选择一个解析为RDD去执行 例子 以下以这个sql的解析为例： SELECT sum(v) ..
Spark-Sql源码解析
08-22
详细描述了Spark-sql的原理，从sql语句如何转换为逻辑计划，然后是物理计划，最后是rdd
mysql词法分析antlr4_Antlr4 - 自定义SparkSQL解析
weixin_35732670的博客
02-08 777
Antlr4 是一个强大的解析器的生成器，可以用来读取、处理、执行或翻译结构化文本，ANTLR可以从语法上来生成一个可以构建和遍历解析树的解析器，最出名的Spark计算引擎2.x就是用它来解析SQL的，是一个牛到没朋友的家伙。IDEA测试插件安装antlr-v4-grammar分析插件g4语法文件使用的是sparkSQL的 SqlBase.g4 文件进行改造的 ArcSql.g4右键选中 mult...
sparksql解析这个json串{"show":{"ecpm":100,"isBid":0}}
最新发布
01-16
Spark SQL可以使用`get_json_object`函数来解析JSON串。下面是一个示例代码： ```sql SELECT get_json_object(sJson, '$.show.ecpm') as ecpm, get_json_object(sJson, '$.show.isBid') as isBid FROM your_...
SparkSQL基础解析（三）
大数据知识梳理
01-04 1235
Sparksql的基础解析，与简单使用
spark sql query 剖析
zhixingheyi_tian的博客
11-13 458
生成 unresolved LogicalPlan // abstract class AbstractSqlParser` /** Creates LogicalPlan for a given SQL string. */ override def parsePlan(sqlText: String): LogicalPlan = parse(sqlText) { parser =&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt...
SparkSQL详解
qq_40322236的博客
02-06 2562
Spark SQL是 Spark 用来处理结构化数据的一个模块，它提供了 2 个编程抽象：DataFrame 和 DataSet，并且作为分布式 SQL 查询引擎的作用。
spark sql解析过程详解
u012794781的博客
11-18 2381
spark sql解析过程详解
图解spark sql解析过程
一群专业码农的笔记本
10-29 2728
前言 本文主要是通过轻松漫画方式将 复杂的spark sql转为RDD原理 呈现给大家. 版本 spark 2.3.2 准备 mysql表 CREATE TABLE employee ( id BIGINT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT, name VARCHAR(15) NOT NULL, age INT NOT NULL ) ; ...
spark sql demo
liben2007的博客
03-02 678
通过pojo构造table： package sparkSql; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.func
Spark SQL 总结整理(二)：Spark SQL 解析流程、工作原理
super_wj0820的博客
09-18 3014
Spark SQL对SQL语句的处理和关系型数据库类似，即词法/语法解析、绑定、优化、执行。Spark SQL会先将SQL语句解析成一棵树，然后使用规则(Rule)对Tree进行绑定、优化等处理过程 1. Spark SQL运行架构 由上图看出，Spark SQL 的解析流程为： 1. 使用 SessionCatalog 保存元数据 在解析SQL语句之前，会创建 SparkSession，或者如...
SparkSQL 操作Hive表 Demo程序
阿尼古
03-27 2056
package org.apache.spark.examples.sql.hive; // $example on:spark_hive$ import java.io.File; import java.io.Serializable; import java.util.ArrayList; import java.util.List; import org.apache.spark.a...
Spark-SQL绑定原理深入分析
逆流而上Mr李
08-14 1256
上一篇文章Spark-SQL解析原来如此简单讲到了Spark-SQL通过Antlr4生成未解析的LogicalPlan。此时的LogicalPlan是Unresolve的，需要通过Catalog来绑定UnresolvedRelation 和UnresolvedAttribute，生成解析后的LogicalPlan。 在Spark-SQL中，Catalog主要用于各种函数资源信息和元数据信息（数据库、数据表、数据视图、数据分区与函数等）的统一管理。Spark-SQL中的Catalog体系...
SQL在Spark的解析过程（二）
小朋友2滴偷偷的在写博客
07-06 803
文章目录优化逻辑计划阶段 - Optimizer谓词下推列裁剪常量替换常量累加生成可执行的物理计划阶段 - SparkPlanner 优化逻辑计划阶段 - Optimizer 在前文的绑定逻辑计划阶段对 Unresolved LogicalPlan 进行相关 transform 操作得到了 Analyzed Logical Plan，这个 Analyzed Logical Plan 是可以直接转换...
使用explain来分析Spark应用
三丰的专栏
09-18 2268
理解Spark的执行计划对Spark调优是非常重要的。Spark提供了一个explain API去查看Spark SQL查询的执行计划。在这篇文章中我会展示给你如何使用explain去获取执行计划以至于你可以调试和分析你的Spark应用。explain在Dateset上是可用的。你可以用它在Spark 查询真正运行之前就知道Spark的执行计划。当然，Spark也提供了UI，你可以通过UI去看Spark的执行计划以及其它细节当Spark 应用正在运行的时候。并且你可以配置Spark History Serv
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值