spark在idea下的开发调试

之前写过在idea下开发Hadoop程序的文章，开发spark涉及到scala语言，配置上略有复杂，应该写篇博客记录一下所踩过的坑。

必要配置

在开发之前必须要先安装好spark，官网提供了详尽资料，在此就不多赘述

流程

1. 首先打开idea，File---new---project---maven,勾选Create from archetype，然后在下面选择org.jetbrains.idea.maven.model.MavenArchetpe-simple，这是利用maven提供的scala代码模板来创建项目，可以省去我们配置目录，仓库的步骤，然后选择next，填写gav等等。
2. 接下来我们打开项目，首先查看一下目录，应该是src/main/scala的目录，然后打开pom文件，模板已经帮我们配置了很多东西，我们只需要加上我们需要的依赖项等等。我们添加下Hadoop spark的依赖项

   <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
     <modelVersion>4.0.0</modelVersion>
     <groupId>cn.yanjun</groupId>
     <artifactId>sparkTest</artifactId>
     <version>1.0-SNAPSHOT</version>
     <inceptionYear>2008</inceptionYear>
     <properties>
       <scala.version>2.10.6</scala.version>
       <spark.version>2.0.0</spark.version>
       <hadoop.version>2.7.3</hadoop.version>
     </properties>
   
     <repositories>
       <repository>
         <id>scala-tools.org</id>
         <name>Scala-Tools Maven2 Repository</name>
         <url>http://scala-tools.org/repo-releases</url>
       </repository>
     </repositories>
   
     <pluginRepositories>
       <pluginRepository>
         <id>scala-tools.org</id>
         <name>Scala-Tools Maven2 Repository</name>
         <url>http://scala-tools.org/repo-releases</url>
       </pluginRepository>
     </pluginRepositories>
   
     <dependencies>
       <dependency>
         <groupId>org.scala-lang</groupId>
         <artifactId>scala-library</artifactId>
         <version>${scala.version}</version>
       </dependency>
       <dependency>
         <groupId>junit</groupId>
         <artifactId>junit</artifactId>
         <version>4.4</version>
         <scope>test</scope>
       </dependency>
       <dependency>
         <groupId>org.specs</groupId>
         <artifactId>specs</artifactId>
         <version>1.2.5</version>
         <scope>test</scope>
       </dependency>
   
       <dependency>
         <groupId>org.apache.spark</groupId>
         <artifactId>spark-core_2.10</artifactId>
         <version>${spark.version}</version>
       </dependency>
       <dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-client</artifactId>
         <version>${hadoop.version}</version>
       </dependency>
     </dependencies>
   
     <build>
       <sourceDirectory>src/main/scala</sourceDirectory>
       <testSourceDirectory>src/test/scala</testSourceDirectory>
       <plugins>
         <plugin>
           <groupId>org.scala-tools</groupId>
           <artifactId>maven-scala-plugin</artifactId>
           <executions>
             <execution>
               <goals>
                 <goal>compile</goal>
                 <goal>testCompile</goal>
               </goals>
             </execution>
           </executions>
           <configuration>
             <scalaVersion>${scala.version}</scalaVersion>
             <args>
               <arg>-target:jvm-1.5</arg>
             </args>
           </configuration>
         </plugin>
         <plugin>
           <groupId>org.apache.maven.plugins</groupId>
           <artifactId>maven-eclipse-plugin</artifactId>
           <configuration>
             <downloadSources>true</downloadSources>
             <buildcommands>
               <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
             </buildcommands>
             <additionalProjectnatures>
               <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
             </additionalProjectnatures>
             <classpathContainers>
               <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
               <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
             </classpathContainers>
           </configuration>
         </plugin>
       </plugins>
     </build>
     <reporting>
       <plugins>
         <plugin>
           <groupId>org.scala-tools</groupId>
           <artifactId>maven-scala-plugin</artifactId>
           <configuration>
             <scalaVersion>${scala.version}</scalaVersion>
           </configuration>
         </plugin>
       </plugins>
     </reporting>
   </project>
   

   
   
3. 接下来我们写一个demo测试一下功能

   package cn.yanjun
   
   import org.apache.spark.{SparkConf, SparkContext}
   
   /**
    * Hello world!
    *
    */
   object App  {
     def main(args: Array[String]): Unit = {
       val conf=new SparkConf().setMaster("local").setAppName("SPARK TEST");
       val sc=new SparkContext(conf);
       val file=sc.textFile("zookeeper.out");
       println(file.count());
     }
   }
   

   
   
4. 然后可以用mvn打包一下程序 

   mvn package

   可以用

   spark-submit --class cn.yanjun.App target/scala-1.0-SNAPSHOT.jar

   提交运行文件
   
   
5. 调试的话可以直接点运行的绿色按钮，选择application，main class，然后就可以选择调试和运行了。