Scala编写Spark的WorkCount

1 创建一个Maven项目

2 在pom.xml中添加依赖和插件

<!-- 定义了一些常量 -->
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <scala.version>2.12.10</scala.version>
        <spark.version>3.0.1</spark.version>
        <encoding>UTF-8</encoding>
    </properties>

    <dependencies>

        <!-- 导入scala的依赖 -->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <!-- 编译scala的插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!-- 编译java的插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- 打jar插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

3创建一个scala目录

创建一个scala目录，选择scala目录，右键，将目录转成源码包，或者点击maven的刷新按钮

4编写Spark程序

package com.doit.day01

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.rdd.RDD

object WordCount {
  def main(args: Array[String]): Unit = {

   /* //本地运行模式，使用如下代码
    //设置Hadoop用户，否则会抛没有写权限异常
    System.setProperty("HADOOP_USER_NAME","root")

    //设置为本地模式.setMaster("local[*]")
    val conf = new SparkConf().setAppName("WordCount").setMaster("local[*]")*/


    val conf = new SparkConf().setAppName("WordCount")
    val sc = new SparkContext(conf)

    //使用SparkContext创建RDD 
    //args(0):读取文件路径
    val lines: RDD[String] = sc.textFile(args(0))

    /*Transformation 开始*/
    //切分压平
    val words: RDD[String] = lines.flatMap(_.split(","))

    //将单词和1组合
    val wordAndOne: RDD[(String, Int)] = words.map((_, 1))

    //根据相同的单词进行聚合运算
    val reduced: RDD[(String, Int)] = wordAndOne.reduceByKey(_ + _)

    //排序
    val sorted = reduced.sortBy(_._2, false)
    /*Transformation 结束*/

    调用Action将计算结果保存到HDFS中
    sorted.saveAsTextFile(args(1))

    //释放资源
    sc.stop()
  }

}

5 使用maven打包

6提交任务

上传jar包到服务器，然后使用sparksubmit命令提交任务

/opt/apps/spark-3.0.1-bin-hadoop3.2/bin/spark-submit   --class com.doit.day01.WordCount   --master spark://linux01:7077   --executor-memory 1g   --total-executor-cores 4   /root/spark19-1.0-SNAPSHOT.jar   hdfs://linux01:8020/wordcount.txt hdfs://linux01:8020/data/out1

参数说明：

--class 指定运行的程序的main方法全类名

--master 指定master的地址和端口，协议为spark://，端口是RPC的通信端口

--executor-memory 指定每一个executor的使用的内存大小

--total-executor-cores指定整个application总共使用的cores

jar包路径  args0  args1

查看结果

[root@linux01 spark-3.0.1-bin-hadoop3.2]# hdfs dfs -cat /data/out1/part-00000

(e,6)
(a,5)

[root@linux01 spark-3.0.1-bin-hadoop3.2]# hdfs dfs -cat /data/out1/part-00001
(d,3)
(c,3)
(b,2)
(f,1)

全局排序是降序的

7 本地运行模式

运行main方法，设置main方法的参数