【Spark学习】用maven构建spark项目

小白必看，用maven构建spark项目

首先，我们需要下载maven
maven下载地址

1.解压安装即可

2.进入conf目录，修改配置文件settings.xml文件，配置阿里云镜像

-<mirror>    
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>  
<name>Nexus aliyun</name>   
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

-<build>
-<plugins>
-<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
-<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build

3.添加完成后，我们就可以打开idea了

4.选择maven并点击next

5.设置GroupId和ArtifacId，并点击next

6.这里直接finish

7.点击file–>setting…

8.在左侧找到Build Tools–>Maven。并选择本地maven根目录，指定setting和repository

7.然后再箭头处添加依赖

        <properties>
            <scala.version>2.11.12</scala.version>
            <spark.version>2.4.4</spark.version>
            <hadoop.version>2.7.3</hadoop.version>
            <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        </properties>

        <repositories>
            <repository>
                <id>scala-tools.org</id>
                <name>Scala-Tools Maven2 Repository</name>
                <url>http://scala-tools.org/repo-releases</url>
            </repository>
        </repositories>

        <pluginRepositories>
            <pluginRepository>
                <id>scala-tools.org</id>
                <name>Scala-Tools Maven2 Repository</name>
                <url>http://scala-tools.org/repo-releases</url>
            </pluginRepository>
        </pluginRepositories>

        <dependencies>
            <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-library</artifactId>
                <version>${scala.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_2.11</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-hive_2.11</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-streaming_2.11</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>2.7.3</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-mllib_2.11</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <dependency>
                <groupId>mysql</groupId>
                <artifactId>mysql-connector-java</artifactId>
                <version>5.1.48</version>
            </dependency>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.11</version>
                <scope>test</scope>
            </dependency>
            <dependency>
                <groupId>org.specs</groupId>
                <artifactId>specs</artifactId>
                <version>1.2.5</version>
                <scope>test</scope>
            </dependency>
        </dependencies>

        <build>
            <sourceDirectory>src/main/scala</sourceDirectory>
            <testSourceDirectory>src/test/scala</testSourceDirectory>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.3</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                        </execution>
                    </executions>
                    <configuration>
                        <scalaVersion>${scala.version}</scalaVersion>
                        <args>
                            <arg>-target:jvm-1.8</arg>
                        </args>
                    </configuration>
                </plugin>
            </plugins>
        </build>

8.刚添加完依赖，本地仓库内可能会有很多包不存在（爆红），所以我们这里要导包。点击import changes idea开始自动导包

9.在main和test目录下分别创建Scala文件夹

10.创建完之后，scala文件夹是灰色的，我们要将他source一下

11.然后就可以了，test目录下也要创建一次

9.导完之后，会发现左侧已经添加很多依赖包了

10.这时我们就可以开始完成spark的项目了