创建maven工程
目录结构
首先需要引入所需要的spark等的依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.dndn</groupId>
<artifactId>ScalaDemo</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>war</packaging>
<name>ScalaDemo Maven Webapp</name>
<!-- FIXME change it to the project's website -->
<url>http://www.example.com</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.12</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.12</artifactId>
<version>1.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.12</artifactId>
<version>1.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.12</artifactId>
<version>2.0.0</version>
</dependency>
<!--connector dependency-->
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.12</artifactId>
<version>1.10.2</version>
</dependency>
<dependency>
<groupId>org.apache.bahir</groupId>
<artifactId>flink-connector-redis_2.11</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.12.14</version>
</dependency>
</dependencies>
<build>
<finalName>ScalaDemo</finalName>
<pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<version>3.1.0</version>
</plugin>
<!-- see http://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_war_packaging -->
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.1</version>
</plugin>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>3.2.2</version>
</plugin>
<plugin>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
</plugin>
<plugin>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
各题的分析
1、提供分析数据data.txt,该数据集包含了某大学计算机系的成绩,数据格式如下所示
Tom,DataBase,80
Tom,Algorithm,50
Tom,DataStructure,60
Jim,DataBase,90
Jim,Algorithm,60
Jim,DataStructure,80
……
请根据给定的实验数据,在scala中通过编程来计算以下内容:
(1)该系总共有多少学生;
(2)该系共开设了多少门课程;
(3)Tom同学的总成绩平均分是多少;
(4)求每名同学的选修的课程门数;
(5)该系DataBase课程共有多少人选修;
(6)各门课程的平均分是多少;
(7)使用累加器计算共有多少人选了DataBase这门课。
运行结果:
代码:
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
object scalaExam1 {
def main(args: Array[String]): Unit = {
//初始化spark
val sparkConf = new SparkConf().setAppName("scalaExam1").setMaster("local")
val sparkSession = SparkSession.builder().appName("scalaExam1").config(sparkConf).getOrCreate()
//读取数据
val rdd = sparkSession.sparkContext.textFile("data/data1/data.txt")
//分割只获取第一个数据 学生姓名
val name = rdd.map(_.split(",")(0))
//分割只获取第二个数据 课程
val subject = rdd.map(_.split(",")(1))
//获取学生数
//去重
val rdd2 = name.distinct()
//统计总行数
val count = rdd2.count()
//该系共开设来多少门课程
val rdd3 = subject.distinct()
//统计总数
val count1 = rdd3.count()
//输出结果 1 2(主要是取得某一列,对其去重后的总数)
println("该系总共有:" + count + "人")
println("该系共开设:" + count1 + "门")
// 3 Tom 同学的总成绩平均分是多少
// (思路:首先要过滤出名字为Tom的数据,然后分割出成绩化成如(Tom,(80,1))的形式,再聚合成如(Tom,(190,3))形式,
// 再利用mapvalues计算平均分 190/3 )
val tom = rdd.filter(_.split(",")(0) == "Tom")
val avgTom = tom.map(x => {
//以逗号分割,取值
val arr = x.split(",")
val name = arr(0)
val values = arr(2).toInt
(name, values)
//进行mapreduce操作
}).mapValues(v => {
(v, 1)
}).reduceByKey((curr, agg) => {
(curr._1 + agg._1, curr._2 + agg._2)
}).mapValues(v => {
(v._1 / v._2)
}).collect()
println("Tom的平均分为:" + avgTom(0)._2)
// (tom,xx)
//4 求每名同学的选修的课程门数
//(思路:首先对学生名字先去重,化成如(Tome,(80,1))形式,多个进行reduce处理,即处理成如(Tom,(190,3)),最后只取3即可)
rdd.map(x => {
val arr = x.split(",")
val name = arr(0)
val values = arr(2)
(name, values)
//进行mapreduce操作
}).mapValues(x => {
(x, 1)
}).reduceByKey((curr, agg) => {
("", curr._2 + agg._2)
}).mapValues(x => {
(x._2)
}).foreach(println)
//5 该系 DataBase 课程共有多少人选修
//(思路:直接过滤出课程是DataBase,统计数量)
val DataCount = rdd.filter(_.split(",")(1) == "DataBase").count()
println("该系 DataBase 课程共有:" + DataCount + "人选修")
//6 各门课程的平均分是多少、
rdd.map(x => {
val arr = x.split(",")
val subject = arr(1)
val values = arr(2).toInt
(subject, values)
}).mapValues(x => {
(x, 1)//对每一个值转换例如("DataStructure",(80,1))这种形式,key无操作
}).
reduceByKey((a, b) => {
//把相同key值合并,如("DataStructure",(60,1))和上面的
//合并成(80,1)(60,1)
(a._1 + b._1, a._2 + b._2)//例如结果为(DataStructure,(140,2))
}).
mapValues(x => {
(x._1 / x._2)
}).foreach(println)
//7 使用累加器计算共有多少人选了 DataBase 这门课
//创建计数器
val counter = sparkSession.sparkContext.longAccumulator("counter1")//定义long类型的累加器累加整型
rdd.map(x => {
val arr = x.split(",")
val subject = arr(1)
if (subject == "DataBase") {
counter.add(1)
}
}).collect()
println("该系 DataBase 课程共有:" + counter.value + "人选修")
}
}
2、编写应用程序实现数据去重
对于两个输入文件A和B,编写Scala应用程序,对两个文件进行合并,并剔除其中重复的内容,得到一个新文件C。本文给出三门课的成绩(A.txt、B.txt)下面是输入文件和输出文件的一个样例,供参考。
输入文件A的样例如下:
20200101 x
20200102 y
20200103 x
20200104 y
20200105 z
20200106 z
输入文件B的样例如下:
20200101 y
20200102 y
20200103 x
20200104 z
20200105 y
根据输入的文件A和B合并得到的输出文件C的样例如下:
20200101 x
20200101 y
20200102 y
20200103 x
20200104 y
20200104 z
20200105 y
20200105 z
20200106 z
运行结果:
代码:
import java.io.{File, FileOutputStream, OutputStreamWriter, PrintWriter}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
object scalaExam2 {
def main(args: Array[String]): Unit = {
//初始化spark
val sparkConf = new SparkConf().setAppName("scalaExam2").setMaster("local")
val sparkSession = SparkSession.builder().appName("scalaExam2").config(sparkConf).getOrCreate()
//读取数据
val rd = sparkSession.sparkContext.textFile("data/data2/A.txt")
val rdd = sparkSession.sparkContext.textFile("data/data2/B.txt")
//连接两个RDD 并且去重 转为Array格式
val rdd1 = rd.union(rdd).distinct().map(x => {
val arr = x.split(" ")
(arr(0), arr(1))
//排序,对学号从小到大排
}).sortBy(x => {
x._1 + " " + x._2
}).map(x => {
val one = x._1
val tow = x._2
//转为指定格式
one + " " + tow
}).collect()
//rdd1.foreach(println)
val write = new PrintWriter(new OutputStreamWriter(new FileOutputStream("data/out2/C.txt")))
for (i <- rdd1) {
write.write(i + "\n")
}
write.close()
}
}
3、编写应用程序实现求平均值问题
每个输入文件表示班级学生某个学科的成绩,每行内容由两个字段组成,第一个是学生名字,第二个是学生的成绩;编写Scala应用程序求出所有学生的平均成绩,并输出到一个新文件中。本文给出三门课的成绩(Algorithm.txt、Database.txt、Python.txt),下面是输入文件和输出文件的一个样例,供参考。
Algorithm成绩:
小明 92
小红 87
小新 82
小丽 90
Database成绩:
小明 95
小红 81
小新 89
小丽 85
Python成绩:
小明 82
小红 83
小新 94
小丽 91
平均成绩如下:
(小红,83.67)
(小新,88.33)
(小明,89.67)
(小丽,88.67)
运行结果:
代码:
import java.io.{FileOutputStream, OutputStreamWriter, PrintWriter}
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
object scalaExam3 {
def main(args: Array[String]): Unit = {
//编写独立应用程序实现求平均值问题
//初始化spark
val sparkConf = new SparkConf().setAppName("scalaExam3").setMaster("local")
val sparkSession = SparkSession.builder().appName("scalaExam3").config(sparkConf).getOrCreate()
//读取数据
val rd = sparkSession.sparkContext.textFile("data/data3/Algorithm.txt")
val rdd = sparkSession.sparkContext.textFile("data/data3/Database.txt")
val rdd1 = sparkSession.sparkContext.textFile("data/data3/Python.txt")
//连接 rdd和rdd1
val rdds = rd.union(rdd).union(rdd1)
.map(x => { //取值转为数组
val arr = x.split(" ")
val name = arr(0)
val values = arr(1).toDouble
//利用mapValues合成数据(小明,(92,1))
(name, values)
}).mapValues(x => {
(x, 1)
}).reduceByKey((x, y) => { //reduce后变成例如(小明,(269,3))
val a = x._1 + y._1
val b = x._2 + y._2
(a, b)
}).mapValues(v => { //进行把聚合的分数和科目数量求平均
val avg = v._1 / v._2
(avg.formatted("%.2f"))
}).collect()
val write = new PrintWriter(new OutputStreamWriter(new FileOutputStream("data/out3/out.txt")))
for (i <- rdds) {
write.write(i + "\n")
}
write.close()
}
}