Spark WordCount编程示例

Spark Shell仅在测试和验证我们的程序时使用的较多，在生产环境中，通常会在IDEA中编制程序，然后打成Jar包，然后提交到集群，最常用的是创建一个Maven项目，利用Maven来管理Jar包的依赖。

3.1 编写程序

1）创建一个Maven项目WordCount

2）在项目WordCount上点击右键，Add Framework Support=》勾选scala

3）在main下创建scala文件夹，并右键Mark Directory as Sources Root=>在scala下创建包名为com.mischen

4）输入文件夹准备：在新建的WordCount项目名称上右键=》新建input文件夹=》在input文件夹上右键=》分别新建1.txt。每个文件里面准备一些word单词。

5）导入项目依赖，包括打包插件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mischen</groupId>
    <artifactId>SparkWordCount</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>
    </dependencies>

    <build>
        <finalName>WordCount</finalName>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.4.6</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.0.0</version>
        <configuration>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
        </configuration>
        <executions>
            <execution>
                <id>make-assembly</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
            </execution>
        </executions>
    </plugin>
</project>

6）创建伴生对象WordCount，编写代码

package com.mischen.wordcount

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

/*
 * @author mischen
 * @create_date 2021-06-06 08:25
 */
object WordCount {

  def main(args: Array[String]): Unit = {
    //1.创建SparkConf并设置App名称
    val conf = new SparkConf().setAppName("WC").setMaster("local[*]")

    //2.创建SparkContext，该对象是提交Spark App的入口
    val sc = new SparkContext(conf)

    //3.读取指定位置文件:hello atguigu atguigu
    val lineRdd: RDD[String] = sc.textFile("input/1.txt")

    //4.读取的一行一行的数据分解成一个一个的单词（扁平化）(hello)(atguigu)(atguigu)
    val wordRdd: RDD[String] = lineRdd.flatMap(line => line.split(" "))

    //5. 将数据转换结构：(hello,1)(atguigu,1)(atguigu,1)
    val wordToOneRdd: RDD[(String, Int)] = wordRdd.map(word => (word, 1))

    //6.将转换结构后的数据进行聚合处理 atguigu:1、1 =》1+1  (atguigu,2)
    val wordToSumRdd: RDD[(String, Int)] = wordToOneRdd.reduceByKey((v1, v2) => v1 + v2)

    //7.将统计结果采集到控制台打印
    val wordToCountArray: Array[(String, Int)] = wordToSumRdd.collect()
    wordToCountArray.foreach(println)

    //一行搞定
    //sc.textFile(args(0)).flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _).saveAsTextFile(args(1))

    //8.关闭连接
    sc.stop()
  }

}

Spark程序运行过程中会打印大量的执行日志，为了能够更好的查看程序的执行结果，可以在项目的resources目录中创建log4j.properties文件，并添加日志配置信息：

log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to ERROR. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=ERROR

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ERROR
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

7）程序运行结果

8）打包到集群测试

（1）点击package打包，然后，查看打完后的jar包

（2）将WordCount.jar上传到/opt/module/spark-yarn目录

（3）在HDFS上创建，存储输入文件的路径/input

[mischen@hadoop102 spark-yarn]$ hadoop fs -mkdir /input

（4）上传输入文件到/input路径

[mischen@hadoop102 spark-yarn]$ hadoop fs -put /opt/module/spark-local/input/1.txt /input

（5）执行任务

[mischen@hadoop102 spark-yarn]$ bin/spark-submit \

--class com.atguigu.spark.WordCount \

--master yarn \

WordCount.jar \

/input \

/output

注意：input和ouput都是HDFS上的集群路径。

（6）查询运行结果

[mischen@hadoop102 spark-yarn]$ hadoop fs -cat /output/*