spark-sql与elasticsearch整合&测试

最新推荐文章于 2024-01-20 12:58:01 发布
最新推荐文章于 2024-01-20 12:58:01 发布
阅读量385 收藏
点赞数
分类专栏： spark 文章标签： spark elasticsearch

1. 前置条件

  • spark是1.4.1版本
  • elasticsearch是1.7版本
  • java是1.7版本

2. 依赖jar包

需要使用elasticsearch-hadoop 
下载地址：http://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop/2.2.0-m1

3. 配置

将下载的elasticsearch-hadoop包放置到$SPARK_HOME/lib/下

<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ls <span class="hljs-attribute" style="box-sizing: border-box;">-lh</span> lib

<span class="hljs-attribute" style="box-sizing: border-box;">-rw</span><span class="hljs-attribute" style="box-sizing: border-box;">-r</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-----</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> wangyue wangyue <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">692</span>K <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>月  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">17</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">49</span> elasticsearch<span class="hljs-attribute" style="box-sizing: border-box;">-hadoop</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span><span class="hljs-attribute" style="box-sizing: border-box;">-m1</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar
<span class="hljs-attribute" style="box-sizing: border-box;">-rw</span><span class="hljs-attribute" style="box-sizing: border-box;">-rw</span><span class="hljs-attribute" style="box-sizing: border-box;">-r</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> wangyue wangyue <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">946</span>K  <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>月 <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">22</span> mysql<span class="hljs-attribute" style="box-sizing: border-box;">-connector</span><span class="hljs-attribute" style="box-sizing: border-box;">-java</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.35</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

配置$SPARK_HOME/conf/spark-env.sh 
SPARK_CLASSPATH增加elasticsearch-hadoop-2.2.0-m1.jar包

4. spark-sql写数据到elasticsearch

vim /home/cluster/data/test/people.txt，增加如下测试内容：

<code class="hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">zhang,san,20
li,si,30
wang,wu,40
li,bai,100
du,fu,101</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

启动spark-shell增加如下代码：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span>    
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span>._
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.elasticsearch</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span>._ 
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>._
import sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.implicits</span>._

//创建sqlContext
val sqlContext = new SQLContext(sc)

//定义Person　case class
case class Person(name: String, surname: String, age: Int)

//创建DataFrame
val people = sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.textFile</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"file:///home/cluster/data/test/people.txt"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(_<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.split</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(p => Person(p(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), p(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>), p(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.trim</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.toInt</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.toDF</span>()

people<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.saveToEs</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark/people"</span>)  </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul>

查询插入到elasticsearch数据：

<code class="hljs vbscript has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">GET</span> /spark/people/_search

{
   <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"took"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
   <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"timed_out"</span>: <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">false</span>,
   <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_shards"</span>: {
      <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"total"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,
      <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"successful"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,
      <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"failed"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
   },
   <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hits"</span>: {
      <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"total"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>,
      <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"max_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
      <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hits"</span>: [
         {
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5MyJpFYJkWM4nwP"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: {
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"zhang"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"san"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span>
            }
         },
         {
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5SEJpFYJkWM4nwS"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: {
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"li"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bai"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>
            }
         },
         {
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5MyJpFYJkWM4nwR"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: {
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"wang"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"wu"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">40</span>
            }
         },
         {
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5MyJpFYJkWM4nwQ"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: {
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"li"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"si"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">30</span>
            }
         },
         {
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5SEJpFYJkWM4nwT"</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>,
            <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: {
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"du"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fu"</span>,
               <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">101</span>
            }
         }
      ]
   }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li></ul>

5. spark-sql读取elasticsearch数据

Name Default value Description
path Elasticsearch index/type required
pushdown true Whether to translate (push-down) Spark SQL into Elasticsearch
strict false Whether to use exact (not analyzed) matching or not (analyzed)

  • load或read方式读取：

    <code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span>    
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span>._

val sqlContext = new SQLContext(sc)
// options for Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.3</span> need to include the target path/resource
val options13 = Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"path"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark/people"</span>,
                    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pushdown"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"true"</span>,
                    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.nodes"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"localhost"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.port"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"9200"</span>)

// Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.3</span> style
val spark13DF = sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.load</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.elasticsearch.spark.sql"</span>, options13)

// options for Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span> - the path/resource is specified separately
val options = Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pushdown"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"true"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.nodes"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"localhost"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.port"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"9200"</span>)

// Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span> style
val spark14DF = sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.read</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.elasticsearch.spark.sql"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.options</span>(options)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.load</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark/people"</span>)

查询name,age:
spark14DF<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.select</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>()<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.foreach</span>(println(_))
[zhang,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span>]
[li,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>]
[wang,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">40</span>]
[li,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">30</span>]
[du,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">101</span>]

注册临时表＆查询name
spark14DF<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.registerTempTable</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>)
val results = sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT name FROM people"</span>)
results<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(t => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Name: "</span> + t(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>()<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.foreach</span>(println)
<span class="hljs-label" style="box-sizing: border-box;">Name:</span> zhang
<span class="hljs-label" style="box-sizing: border-box;">Name:</span> li
<span class="hljs-label" style="box-sizing: border-box;">Name:</span> wang
<span class="hljs-label" style="box-sizing: border-box;">Name:</span> li
<span class="hljs-label" style="box-sizing: border-box;">Name:</span> du</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li></ul>

  • 读取elasticsearch数据并创建临时表myPeople

    <code class="hljs sql has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sqlContext.sql(
   "<span class="hljs-operator" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CREATE</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TEMPORARY</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TABLE</span> myPeople    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + 
   "</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">USING</span> org.elasticsearch.spark.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">sql</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + 
   "</span>OPTIONS ( resource <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'spark/people'</span>, nodes <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'localhost:9200'</span>)<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" ) 

sqlContext.sql("</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">select</span> * <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> myPeople<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">").collect.foreach(println)

[20,zhang,san]
[100,li,bai]
[40,wang,wu]
[30,li,si]
[101,du,fu]


sqlContext.sql(
   "</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CREATE</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TEMPORARY</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TABLE</span> myPeople    <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + 
   "</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">USING</span> org.elasticsearch.spark.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">sql</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + 
   "</span>OPTIONS ( resource <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'spark/people'</span>, nodes <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'localhost:9200'</span>,scroll_size <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'20'</span>)<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" ) 
因为使用.会导致语法异常,应该用_风格代替它. 因此,在这个例子中es.scroll.size 变成　scroll_size(由于加载数据时候es可以删除)。注意这只工作在spark1.3／1.4的spark有更严格的解析器</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li></ul>

  • esDF方式读取elasticsearch数据：

    <code class="hljs asciidoc has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org.apache.spark.sql.SQLContext       
import org.elasticsearch.spark.sql._ 

val sqlContext = new SQLContext(sc)
val people = sqlContext.esDF("spark/people") 
<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// check the associated schema</span>
println(people.schema.treeString) 

root
<span class="hljs-code" style="box-sizing: border-box;"> |-- age: long (nullable = true)</span>
<span class="hljs-code" style="box-sizing: border-box;"> |-- name: string (nullable = true)</span>
<span class="hljs-code" style="box-sizing: border-box;"> |-- surname: string (nullable = true)</span>


<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// get only the wang</span>
val wangs = sqlContext.esDF("spark/people","?q=wang" )

wangs.show()

<span class="hljs-code" style="box-sizing: border-box;">+---+</span>----<span class="hljs-code" style="box-sizing: border-box;">+-------+</span>
<span class="hljs-header" style="box-sizing: border-box;">|age|name|surname|
+---+----+-------+</span>
<span class="hljs-header" style="box-sizing: border-box;">| 40|wang|     wu|
+---+----+-------+</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

参考文章： 
https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql

作者:stark_summer 
出处：http://blog.csdn.net/stark_summer/article/details/49743687

icey__
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Elasticsearch加速SparkSQL查询-李振炜
04-14
20161210Elastic{ON} Dev China 2016
项目实战——Spark将Hive表的数据写入ElasticSearch（Java版本）
10-24
在本项目实战中，我们将探讨如何使用Java编程语言，结合Spark和Hive，将Hive中的数据高效地导入到ElasticSearch（ES）中，并利用ES的别名机制实现数据更新的平滑过渡。以下是对这个流程的详细解析： 1. **Hive数据...
Spark SQL 读写 elasticsearch
weixin_44695980的博客
07-04 1560
1、需要的maven依赖 org.elasticsearch elasticsearch-hadoop 2.2.0-m1 2、配置 将下载的elasticsearch-hadoop包放置到$SPARK_HOME/lib/下 3数据写入es vim /home/admin/people.txt，增加如下测试内容： liu,sun,20 li,si,30 wang,wu,40 li,bai,100...
Spark SQL & Elasticsearch
heartsdance的博客
02-11 1777
踩坑Spark SQL读写Elasticsearch全流程
spark导入elasticsearch
m0_67401134的博客
08-24 695
前面简单的介绍了elasticsearch。现在开始使用，当我们面对海量数据的时候，如果想把数据导入到es，肯定不能想以前那样单条导入，我使用的是spark导入到es的批量导入。可以到这里下载相应的依赖包解压使用。PS：在使用的时候一定要记得spark和scala以及es的对应关系，我使用的是spark1.6.2 ，scala使用的是2.10.4，在使用的时spark-es的jar包elasticsearch-spark_2.10-2.3.3，该版本对应的编译的时候使用的是scala2.10.5.。
计算机课程毕设：基于ElasticSearch+Spark 构建高相关性搜索服务&千人千面推荐系统.zip
12-25
总之，这个项目涵盖了大数据处理、搜索引擎技术和推荐系统的核心知识，通过实践可以深入理解Spark和ElasticSearch的集成应用，以及如何构建一个既高效又能提供个性化体验的搜索与推荐系统。对于想要在IT领域，特别是...
ElasticSearch+Spark 构建高匹配度搜索服务，个性化推荐系统源码
10-29
通过深入研究这些源码，可以了解如何将 Elasticsearch 和 Spark 整合起来，构建高效、个性化的推荐系统。 **总结** Elasticsearch 和 Spark 的结合是大数据领域中的强强联合，它们在构建高匹配度搜索服务和个性化...
大数据-使用flume+kafka+HBase+spark+ElasticSearch的用户轨迹查询大数据项目.zip
最新发布
03-08
5. **Elasticsearch**: Elasticsearch 是一个基于 Lucene 的分布式搜索引擎，用于实时、高效地搜索和分析大量数据。在本项目中，Elasticsearch 可能用于对用户轨迹数据进行全文检索和分析，支持快速的实时查询，便于...
elasticsearch-2.4.4.tar
03-26
Spark可以通过其DataFrame和Spark SQL接口与Elasticsearch进行交互，实现大规模数据处理和实时分析。 9. **大数据处理**：Elasticsearch在大数据场景下表现出色，可以处理PB级别的数据，并通过其内置的倒排索引技术...
elasticsearch-spark_2.10-2.3.2.jar
09-20
ES-Spark插件，使用该插件可以使用Spark向ES插入数据或者使用Spark读取ES的数据
Spark SQL大数据处理并写入Elasticsearch
HarvardFly的博客
10-16 1341
Spark SQL大数据处理并写入Elasticsearch SparkSQL(Spark用于处理结构化数据的模块) 通过SparkSQL导入的数据可以来自MySQL数据库、Json数据、Csv数据等，通过load这些数据可以对其做一系列计算 下面通过程序代码来详细查看SparkSQL导入数据并写入到ES中： 数据集：北京市PM2.5数据 ...
spark 集成elasticsearch
dianxiang0791的博客
10-16 142
pyspark读写elasticsearch依赖elasticsearch-hadoop包，需要首先在这里下载，版本号可以通过自行修改url解决。 """ write data to elastic search https://starsift.com/2018/01/18/integrating-pyspark-and-elasticsearch/ """ from _...
spark和elasticsearch集成
weixin_34150830的博客
12-10 195
为什么80%的码农都做不了架构师？>>> ...
[日志处理工作之九]整合Spark SQL和Elasticsearch
bigstar863的博客
11-01 6277
本文使用的Spark版本是1.4.0，Elasticsearch版本是1.5.2 1.Elasticsearch对Spark的支持详见官网：https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html 官网的文档主要涉及Spark1.2中的RDD API、Spark1.3以上版本的Data Frame API，语言
spark 写 elasticsearch
我只是个小学生 能力有限 一直抱着学习的态度
10-29 189
[root@hadoop000 bin]# ./spark-submit --master local[2] --jars /root/app/spark-2.3.0-bin-2.6.0-cdh5.7.0/jars/elasticsearch-spark-20_2.11-6.3.0.jar /root/app/spark-2.3.0-bin-2.6.0-cdh5.7.0/python/spark0...
Spark写入 ES 异常 Use 'org.elasticsearch.spark.sql' package instead
码者人生的技术博客
03-21 735
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost, exe...
Spark 读取ElasticSearch
小湘西的博客
01-20 451
查询语句只要满足ES规范就行，这里使用的是下面的包。
Spark篇：Spark读写elasticsearch
qq_38740498的博客
12-10 419
spark篇8：SparkSql读写elasticsearch 废话不多说，直接上干货 package com.iflytek.elasticsearch import org.apache.spark.sql.{SparkSession, _} import org.elasticsearch.spark.sql._ object sparkEs { def main(args: Arr...
Spark基础架构与原理详解
此外，还与其他技术如 Durid（数据库）、Elasticsearch、HBase、HDP 等进行了紧密集成，形成了一个强大的大数据处理解决方案。 Spark 的基本架构和原理涉及内存计算的高效处理、与Hadoop的整合、以及丰富的生态系统...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值