Spark修炼之道（进阶篇）——Spark入门到精通：第三节 Spark Intellij IDEA开发环境搭建

本节主要内容

1. Intellij IDEA 14.1.4开发环境配置
2. Spark应用程序开发

1. Intellij IDEA 14.1.4开发环境配置

Intellij IDEA 功能十分强大，能够开发JAVA、Scala等相关应用程序，在依赖管理 
智能提示等方面做到了极致，大家可以到:http://www.jetbrains.com/idea/download/下载，目前有两种：Ultimate Edition Free 30-day trial；Community Edition FREE。Ultimate版本是商业软件，需要付费，Community 版为免费版，足够平时日常开发需要。最新的版是 Intellij IDEA 14.1.4，但Intellij IDEA 没有自带scala开发插件，需要手功安装，但本人测试的时候发现，直接在Intellij IDEA中装很难装成功（个中原因大家懂得)，为此本人将带有Scala插件的Intellij IDEA已经打包好了，大家直接下载就可以进行Scala及后期的Spark应用程序开发，下载地址：链接：http://pan.baidu.com/s/1sjmS3jJ 密码：rcsy 
当然，上面提供的是Linux环境下的Intellij IDEA。

下载完成后，解压到/hadoopLearning目录，得到 
 
将其设置到环境变量 vim /etc/profile，添加红色下划线内容 

然后执行

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sparkmaster</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:/hadoopLearning</span><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># idea.sh </span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

启动Intellij IDEA，如下图所示（本机上已经创建过scala工程项目) 

（1）创建Scala项目

File->new->Project，如下图 
 
选择Scala 
 
然后next 
 
其中Project SDK指定安装的JDK，Scala SDK指定安装的Scala（这里使用的是IDEA自带的scala SDK)，这里将项目名称命令为SparkWordCount，然后finish 

在IDEA中开发应用程序时，常常需要通过一定的文件目录组织进行源码编写，例如源文件目录、测试源文件目录，下面演示在Intellij IDEA的src目录下创建main/scala源文件目录。 
直接按F4或右鍵点击工程文件 

再选择open module setting，打开项目配置，点击src目录，然后右键创建main/scala文件夹，再点击scala文件夹为sources，如下图所示 

### （2）导入Spark 1.5.0依赖包 
直接F4打开Project Structure，然后选择libraries 
 
点击上图中的+添加外部依赖包，选择”java”，然后再选择spark-assembly-1.5.0-hadoop2.4.0.jar 
 
成功后如下图 

至此Spark开发环境配置完成

2. Spark应用程序开发

(1) 本地方式执行Spark WordCount程序

在src/main/scala源文件目录中创建一个SparkWordCount 应用程序对象，编辑内容如下：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SparkContext</span>._
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span>.{SparkConf, SparkContext}

object SparkWordCount{
  def main(args: Array[String]) {
    //输入文件既可以是本地linux系统文件，也可以是其它来源文件，例如HDFS
    if (args<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.length</span> == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {
      System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.err</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.println</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Usage: SparkWordCount <inputfile>"</span>)
      System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.exit</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)
    }
    //以本地线程方式运行，可以指定线程个数，
    //如<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMaster</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local[2]"</span>)，两个线程执行
    //下面给出的是单线程执行
    val conf = new SparkConf()<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setAppName</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SparkWordCount"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMaster</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local"</span>)
    val sc = new SparkContext(conf)

    //wordcount操作，计算文件中包含Spark的行数
    val count=sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.textFile</span>(args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.filter</span>(line => line<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.contains</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Spark"</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.count</span>()
    //打印结果
     println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"count="</span>+count)
    sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.stop</span>()
  }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li></ul>

编译代码，直接Build->Make Project 
 
然后编程运行参数，Run->Edit Configurations 
 
Main Class输入：SparkWordCount 
Program arguments输入：/hadoopLearning/spark-1.5.0-bin-hadoop2.4/README.md 
如下图： 

完成后直接Run->Run或Alt+Shift+F10运行程序，执行结果如下图： 

(2) Spark集群上执行Spark WordCount程序

将SparkWordCount打包成Jar文件

将程序内容修改如下：

<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SparkContext</span>._
import org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span>.{SparkConf, SparkContext}

object SparkWordCount{
  def main(args: Array[String]) {
    //输入文件既可以是本地linux系统文件，也可以是其它来源文件，例如HDFS
    if (args<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.length</span> == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {
      System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.err</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.println</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Usage: SparkWordCount <inputfile> <outputfile>"</span>)
      System<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.exit</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)
    }
    //提交集群时，本地线程不起作用
    val conf = new SparkConf()<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setAppName</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SparkWordCount"</span>)<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.setMaster</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local"</span>)
    val sc = new SparkContext(conf)

    //rdd2为所有包含Spark的行
    val rdd2=sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.textFile</span>(args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>))<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.filter</span>(line => line<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.contains</span>(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Spark"</span>))
    //保存内容，在例子中是保存在HDFS上
    rdd2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.saveAsTextFile</span>(args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))
    sc<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.stop</span>()
  }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li></ul>

点击工程SparkWordCount，然后按F4打个Project Structure并选择Artifacts，如下图 
 
选择Jar->form modules with dependencies，如下图 
 
进入下面的界面 
在main class中，选择SparkWordCount，如下图 
 
点击确定后得到如下界面 

因为后期提交到集群上运行，因此相关jar包都存在，为减小jar包的体积，将spark-assembly-1.5.0-hadoop2.4.0.jar等jar包删除即可，如下图 
 
确定后，再点击Build->Build Artifacts 
 
生成后的jar文件保存在root@sparkmaster:~/IdeaProjects/SparkWordCount/out/artifacts/SparkWordCount_jar# 目录中，如下图： 

提交集群运行

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./spark-submit --master <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">spark:</span>/<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/sparkmaster:7077 --class SparkWordCount --executor-memory 1g /root</span><span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/IdeaProjects/</span><span class="hljs-constant" style="box-sizing: border-box;">SparkWordCount</span>/out/artifacts/<span class="hljs-constant" style="box-sizing: border-box;">SparkWordCount_jar</span>/<span class="hljs-constant" style="box-sizing: border-box;">SparkWordCount</span>.jar <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">hdfs:</span>/<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/ns1/</span><span class="hljs-constant" style="box-sizing: border-box;">README</span>.md <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">hdfs:</span>/<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/ns1/</span><span class="hljs-constant" style="box-sizing: border-box;">SparkWordCountResult</span>
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

将任务提交到集群： 

执行结果： 

HDFS文件已经生成了SparkWordCountResult 

使用

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sparkmaster</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:/hadoopLearning/spark-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-bin-hadoop2.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>/bin<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># hadoop dfs -ls /SparkWordCountResult</span>
root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sparkmaster</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:/hadoopLearning/spark-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.5</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-bin-hadoop2.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span>/bin<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># hadoop dfs -cat /SparkWordCountResult/part-00000</span>
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

查看目录内容，具体结果如下图所示：

转载：http://blog.csdn.net/lovehuangjiaju/article/details/48577281