1. 前置条件
- spark是1.4.1版本
- elasticsearch是1.7版本
- java是1.7版本
2. 依赖jar包
需要使用elasticsearch-hadoop
下载地址:http://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop/2.2.0-m1
3. 配置
将下载的elasticsearch-hadoop包放置到$SPARK_HOME/lib/下
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">ls <span class="hljs-attribute" style="box-sizing: border-box;">-lh</span> lib <span class="hljs-attribute" style="box-sizing: border-box;">-rw</span><span class="hljs-attribute" style="box-sizing: border-box;">-r</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-----</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> wangyue wangyue <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">692</span>K <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>月 <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">17</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">49</span> elasticsearch<span class="hljs-attribute" style="box-sizing: border-box;">-hadoop</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.2</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span><span class="hljs-attribute" style="box-sizing: border-box;">-m1</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar <span class="hljs-attribute" style="box-sizing: border-box;">-rw</span><span class="hljs-attribute" style="box-sizing: border-box;">-rw</span><span class="hljs-attribute" style="box-sizing: border-box;">-r</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">--</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> wangyue wangyue <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">946</span>K <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>月 <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">22</span> mysql<span class="hljs-attribute" style="box-sizing: border-box;">-connector</span><span class="hljs-attribute" style="box-sizing: border-box;">-java</span><span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.35</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
配置$SPARK_HOME/conf/spark-env.sh
SPARK_CLASSPATH增加elasticsearch-hadoop-2.2.0-m1.jar包
4. spark-sql写数据到elasticsearch
vim /home/cluster/data/test/people.txt,增加如下测试内容:
<code class="hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">zhang,san,20 li,si,30 wang,wu,40 li,bai,100 du,fu,101</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
启动spark-shell增加如下代码:
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span> import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span>._ import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.elasticsearch</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span>._ import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rdd</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.RDD</span>._ import sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.implicits</span>._ //创建sqlContext val sqlContext = new SQLContext(sc) //定义Person case class case class Person(name: String, surname: String, age: Int) //创建DataFrame val people = sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.textFile</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"file:///home/cluster/data/test/people.txt"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(_<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.split</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">","</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(p => Person(p(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), p(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>), p(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.trim</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.toInt</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.toDF</span>() people<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.saveToEs</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark/people"</span>) </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul>
查询插入到elasticsearch数据:
<code class="hljs vbscript has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">GET</span> /spark/people/_search { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"took"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"timed_out"</span>: <span class="hljs-literal" style="color: rgb(0, 102, 102); box-sizing: border-box;">false</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_shards"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"total"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"successful"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"failed"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> }, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hits"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"total"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"max_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"hits"</span>: [ { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5MyJpFYJkWM4nwP"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"zhang"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"san"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> } }, { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5SEJpFYJkWM4nwS"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"li"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bai"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span> } }, { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5MyJpFYJkWM4nwR"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"wang"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"wu"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">40</span> } }, { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5MyJpFYJkWM4nwQ"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"li"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"si"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">30</span> } }, { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_index"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_type"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_id"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AVDrt5SEJpFYJkWM4nwT"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_score"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"_source"</span>: { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"du"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"surname"</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"fu"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">101</span> } } ] } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li></ul>
5. spark-sql读取elasticsearch数据
Name | Default value | Description |
---|---|---|
path | Elasticsearch index/type | required |
pushdown | true | Whether to translate (push-down) Spark SQL into Elasticsearch |
strict | false | Whether to use exact (not analyzed) matching or not (analyzed) |
-
load或read方式读取:
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span> import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SQLContext</span>._ val sqlContext = new SQLContext(sc) // options for Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.3</span> need to include the target path/resource val options13 = Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"path"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark/people"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pushdown"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"true"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.nodes"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"localhost"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.port"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"9200"</span>) // Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.3</span> style val spark13DF = sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.load</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.elasticsearch.spark.sql"</span>, options13) // options for Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span> - the path/resource is specified separately val options = Map(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"pushdown"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"true"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.nodes"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"localhost"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"es.port"</span> -> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"9200"</span>) // Spark <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span> style val spark14DF = sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.read</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.format</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.elasticsearch.spark.sql"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.options</span>(options)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.load</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spark/people"</span>) 查询name,age: spark14DF<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.select</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span>,<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"age"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>()<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.foreach</span>(println(_)) [zhang,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span>] [li,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>] [wang,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">40</span>] [li,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">30</span>] [du,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">101</span>] 注册临时表&查询name spark14DF<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.registerTempTable</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"people"</span>) val results = sqlContext<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sql</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT name FROM people"</span>) results<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.map</span>(t => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Name: "</span> + t(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.collect</span>()<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.foreach</span>(println) <span class="hljs-label" style="box-sizing: border-box;">Name:</span> zhang <span class="hljs-label" style="box-sizing: border-box;">Name:</span> li <span class="hljs-label" style="box-sizing: border-box;">Name:</span> wang <span class="hljs-label" style="box-sizing: border-box;">Name:</span> li <span class="hljs-label" style="box-sizing: border-box;">Name:</span> du</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li></ul>
-
读取elasticsearch数据并创建临时表myPeople
<code class="hljs sql has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sqlContext.sql( "<span class="hljs-operator" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CREATE</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TEMPORARY</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TABLE</span> myPeople <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + "</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">USING</span> org.elasticsearch.spark.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">sql</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + "</span>OPTIONS ( resource <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'spark/people'</span>, nodes <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'localhost:9200'</span>)<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" ) sqlContext.sql("</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">select</span> * <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> myPeople<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">").collect.foreach(println) [20,zhang,san] [100,li,bai] [40,wang,wu] [30,li,si] [101,du,fu] sqlContext.sql( "</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">CREATE</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TEMPORARY</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">TABLE</span> myPeople <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + "</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">USING</span> org.elasticsearch.spark.<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">sql</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" + "</span>OPTIONS ( resource <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'spark/people'</span>, nodes <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'localhost:9200'</span>,scroll_size <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'20'</span>)<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" ) 因为使用.会导致语法异常,应该用_风格代替它. 因此,在这个例子中es.scroll.size 变成 scroll_size(由于加载数据时候es可以删除)。注意这只工作在spark1.3/1.4的spark有更严格的解析器</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li></ul>
-
esDF方式读取elasticsearch数据:
<code class="hljs asciidoc has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org.apache.spark.sql.SQLContext import org.elasticsearch.spark.sql._ val sqlContext = new SQLContext(sc) val people = sqlContext.esDF("spark/people") <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// check the associated schema</span> println(people.schema.treeString) root <span class="hljs-code" style="box-sizing: border-box;"> |-- age: long (nullable = true)</span> <span class="hljs-code" style="box-sizing: border-box;"> |-- name: string (nullable = true)</span> <span class="hljs-code" style="box-sizing: border-box;"> |-- surname: string (nullable = true)</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// get only the wang</span> val wangs = sqlContext.esDF("spark/people","?q=wang" ) wangs.show() <span class="hljs-code" style="box-sizing: border-box;">+---+</span>----<span class="hljs-code" style="box-sizing: border-box;">+-------+</span> <span class="hljs-header" style="box-sizing: border-box;">|age|name|surname| +---+----+-------+</span> <span class="hljs-header" style="box-sizing: border-box;">| 40|wang| wu| +---+----+-------+</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>
参考文章:
https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-sql
作者:stark_summer
出处:http://blog.csdn.net/stark_summer/article/details/49743687