Spark Streaming 入门
一.Spark Streaming 介绍
1.什么是Spark Streaming
- Spark Streaming 用于流式数据的处理, 什么是流式数据处理? 简单的讲就是数据源不断产生数据,我们周期性的将数据封装成RDD,然后将一个个RDD不断交给Spark Streaming处理,数据和水流一样.
- Spark Streaming 支持的数据源很多,比如Kafka , Flume,Twitter、ZeroMQ 和简单的 TCP 套接字等等
在Spark Streaming中是一个叫做DStream的类型表示本次收集到的数据(底层是RDD),所以总结来说DStream就是对RDD的一种封装,主要用于流式处理
2.Spark Streaming架构
![Spark Streaming的简单架构](https://i-blog.csdnimg.cn/blog_migrate/741a4780be92230a2030800d9533d109.png)
3.Spark Streaming的背压机制
二.Spark Streaming 之 DStream 入门
2.1 WordCount 案例实操
- ➢ 需求:使用 netcat 工具向 9999 端口不断的发送数据,通过 SparkStreaming 读取端口数据并
统计不同单词出现的次数 - maven项目引入依赖
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.12</artifactId>
<version>3.0.0</version>
</dependency>
- scala代码
object SparkStreaming_WordCount {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"),Seconds.apply(5))
val dStream: ReceiverInputDStream[String] = streamContext
.socketTextStream("node1",9999,StorageLevel.MEMORY_ONLY)
val wordToOne: DStream[(String, Int)] = dStream.flatMap(_.split(" ")).map((_, 1))
val wordToCount: DStream[(String, Int)] = wordToOne.reduceByKey(_+_)
wordToCount.print()
streamContext.start()
streamContext.awaitTermination()
}
}
三.DStream 创建
- 除了使用Socket作为数据源, 还可以使用其他方式,比如 RDD队列, 其他自定义数据源(Kafka,flume等)
1.使用RDD队列
object SparkStreaming_queueStream {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"),Seconds.apply(5))
val queue: mutable.Queue[RDD[Int]] = mutable.Queue[RDD[Int]]()
val dStream: InputDStream[Int] = streamContext.queueStream(queue,false)
dStream.map((_,1)).reduceByKey(_+_).print()
streamContext.start()
new Thread(()=>{
while(true) {
queue += streamContext.sparkContext.makeRDD(1 to 10)
Thread.sleep(1000)
}
}).start()
streamContext.awaitTermination()
}
}
2.自定义数据源
- 自定义数据源 需要一个继承Receiver的类
class MyReceiver extends Receiver[String](StorageLevel.MEMORY_ONLY) {
private var socket: Socket = _
private var isStop: Boolean = false
override def onStart(): Unit = {
new Thread(()=>{
socket = new Socket("node1", 9999)
val reader = new BufferedReader((new InputStreamReader(socket.getInputStream)))
var line: String = reader.readLine()
while (line != null && !isStop) {
println(line + "==============>" + (line == null))
store(line)
line = reader.readLine()
}
reader.close()
socket.close()
}).start()
}
override def onStop(): Unit = {
isStop = true
}
}
- 使用receiveStream传入Receiver对象 获取DStream
object SparkStreaming_Receiver {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"), Seconds.apply(5))
val dStream: ReceiverInputDStream[String] = streamContext.receiverStream(new MyReceiver())
dStream.print()
streamContext.start()
streamContext.awaitTermination()
}
}
3.Spark Streaming 读取Kafka数据源
- 读取Kafka数据源有两种不同的API
-
Receiver API: 需要一个专门的 Executor 去接收数据,然后发送给其他的 Executor 做计算,可能会有接收和计算速率不同引起内存溢出问题
-
Derect API: 是由计算的 Executor 来主动消费 Kafka 的数据,速度由自身控制。
- Kafka 0-8 Receiver 模式 : 老版本
- Kafka 0-8 Direct 模式: 老版本
- Kafka 0-10 Direct 模式
-
引入依赖
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
<version>2.4.5</version>
</dependency>
-
代码实现
object DirectAPI {
def main(args: Array[String]): Unit = {
val sparkConf: SparkConf = new
SparkConf().setAppName("ReceiverWordCount").setMaster("local[*]")
val ssc = new StreamingContext(sparkConf, Seconds(3))
val kafkaPara: Map[String, Object] = Map[String, Object](
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG ->
"linux1:9092,linux2:9092,linux3:9092",
ConsumerConfig.GROUP_ID_CONFIG -> "testTopic",
"key.deserializer" ->
"org.apache.kafka.common.serialization.StringDeserializer",
"value.deserializer" ->
"org.apache.kafka.common.serialization.StringDeserializer"
)
val kafkaDStream: InputDStream[ConsumerRecord[String, String]] =
KafkaUtils.createDirectStream[String, String](ssc,
LocationStrategies.PreferConsistent,
ConsumerStrategies.Subscribe[String, String](Set("testTopic"), kafkaPara))
val valueDStream: DStream[String] = kafkaDStream.map(record => record.value())
valueDStream.flatMap(_.split(" "))
.map((_, 1))
.reduceByKey(_ + _)
.print()
ssc.start()
ssc.awaitTermination()
}
}
- 查看消费进度
-
bin/kafka-consumer-groups.sh --describe --bootstrap-server linux1:9092 --group testTopic
四.DStream 的转换
- DStream的操作API和RDD类似,分为转换API和输出API,还有一些转换,状态,窗口(Window)相关的API
1.无状态转换
- 所谓的无状态转换是指调用转换方法,仅仅作用于当前批次的RDD上,结果与上一次RDD数据无关
-
例如: reduceByKey 只会统计当前批次的数据,而不是包含上一批次的数据
- 注意: 针对DStream的KV类型的API操作,需要导入 import StreamingContext._
- Transform API
object SparkStreaming_transform {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"), Seconds.apply(5))
val dStream: ReceiverInputDStream[String] = streamContext.socketTextStream("node0", 9999)
val wordCount: DStream[(String, Int)] = dStream.transform(
(rdd: RDD[String]) => {
rdd.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _)
}
)
wordCount.print()
streamContext.start()
streamContext.awaitTermination()
}
}
输出结果:
Time: 1679194645000 ms
-------------------------------------------
(hello,2)
(world,2)
-------------------------------------------
Time: 1679194650000 ms 每次只统计当前批次的结果,不会合并上一次的结果,我们称为无状态的API
-------------------------------------------
(aaa,1)
(bb,1)
(cc,1)
(aa,2)
- join API
- 两个DStream流之间的 join 需要两个流的批次大小一致(这里的一致是指相同的周期),这样才能做到同时触发计算。计算过程就是对当前批次的两个流中各自的 RDD 进行 join,与两个 RDD 的 join 效果相同。
object SparkStreaming_join {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"), Seconds.apply(5))
val dStream1: ReceiverInputDStream[String] = streamContext.socketTextStream("node0", 9999)
val dStream2: ReceiverInputDStream[String] = streamContext.socketTextStream("node1", 9999)
val wordCount1: DStream[(String, Int)] = dStream1.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
val wordCount2: DStream[(String, Int)] = dStream2.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
val joinWordCount: DStream[(String, (Int, Int))] = wordCount1.join(wordCount2)
joinWordCount.print()
streamContext.start()
streamContext.awaitTermination()
}
}
2.有状态转换
- 与有状态转换相反, 统计本批次数据时,可以把结果和上次批次数据归并到一起
- updateStateByKey API
-
可以统计本次结果并合并上一次结果
object SparkStreaming_UpdateStateByKey {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"), Seconds.apply(5))
streamContext.checkpoint("checkpoint")
val dStream: ReceiverInputDStream[String] = streamContext.socketTextStream("node0",9999)
val wordOne: DStream[(String, Int)] = dStream.flatMap(_.split(" ")).map((_,1))
val wordCount: DStream[(String, Int)] = wordOne.updateStateByKey(
(seq: Seq[Int], option: Option[Int]) => {
Option(seq.sum + option.getOrElse(0))
}
)
wordCount.print()
streamContext.start()
streamContext.awaitTermination()
}
}
- WindowOperations 窗口操作API
-
1.reduceByKeyAndWindow( func : ( V , V ) => V , 窗口周期 , 执行周期 )
object SparkStreaming_window {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount"), Seconds.apply(5))
val dStream: ReceiverInputDStream[String] = streamContext.socketTextStream("node0", 9999)
val wordOne: DStream[(String, Int)] = dStream.flatMap(_.split(" ")).map((_, 1))
val wordCount: DStream[(String, Int)] = wordOne.reduceByKeyAndWindow(
(a: Int, b: Int) => {
a + b
}, Seconds(10), Seconds(5))
wordCount.print()
streamContext.start()
streamContext.awaitTermination()
}
}
-
2.reduceByKeyAndWindow( funIn : ( V , V ) => V , funOut : ( V , V ) => V , 窗口周期 , 执行周期 )
object SparkStreaming_window_inv {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount_Window_InV"), Seconds.apply(5))
streamContext.checkpoint("checkpoint")
val queue: mutable.Queue[RDD[Char]] = mutable.Queue[RDD[Char]]()
val dStream: InputDStream[Char] = streamContext.queueStream(queue, false)
val numOne: DStream[(Char, Int)] = dStream.map((_,1))
val numCount: DStream[(Char,Int)] = numOne.reduceByKeyAndWindow(
(x, y) => {
x + y
},
(x, y) => {
x - y
}, Seconds(40), Seconds(10))
numCount.print()
streamContext.start()
new Thread(() => {
for (i <- 1 to 10) {
queue += streamContext.sparkContext.makeRDD(List('a','a','a'))
Thread.sleep(5000)
}
}).start()
streamContext.awaitTermination()
}
}
-
3.window( 窗口周期 , 执行周期 ) : DStream 注意::window后调用的聚合方法是无状态API 和 xxxAndWindow不一样(功能类似但是性能较低,并且不用到checkpoint)
object SparkStreaming_window {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().setMaster("local[*]")
.setAppName("WordCount_window"), Seconds.apply(5))
val dStream: ReceiverInputDStream[String] = streamContext.socketTextStream("node0", 9999)
val wordToOne: DStream[(String, Int)] = dStream.flatMap(_.split(" ")).map((_, 1))
val windowDStream: DStream[(String, Int)] = wordToOne.window(Seconds(20), Seconds(5))
val wordToCount: DStream[(String, Int)] = windowDStream.reduceByKey(
(x, y) => {
println(x + "============>" + y)
x + y
})
wordToCount.print()
streamContext.start()
streamContext.awaitTermination()
}
}
-
4.其他方法不举例: reduceByWindow 值聚合 countByWindow 统计数量 countByValueAndWindow 不同值个数统计,相当于写好的wordcount groupByKeyAndWindow 窗口内的聚合操作
五.DStream 输出
- DStream 与 RDD 类似 如果没有调用输出方法(RDD称为Action算子),那么整个context不会启动并且程序会抛出异常,提示: No output operations registered, so nothing to execute
- DStream的输出目的也有很多
- ➢ print():在Driver节点上打印,默认是print(10) 一次打印10个多余省略
- ➢ saveAsTextFiles(prefix, [suffix]):以 text 文件形式存储这个 DStream 的内容。每一批次的存
储文件名基于参数中的 prefix 和 suffix。”prefix-Time_IN_MS[.suffix]”。 - ➢ saveAsObjectFiles(prefix, [suffix]):以 Java 对象序列化的方式将 Stream 中的数据保存为
SequenceFiles . 每一批次的存储文件名基于参数中的为"prefix-TIME_IN_MS[.suffix]". Python
中目前不可用。 - ➢ saveAsHadoopFiles(prefix, [suffix]):将 Stream 中的数据保存为 Hadoop files. 每一批次的存
储文件名基于参数中的为"prefix-TIME_IN_MS[.suffix]"。Python API 中目前不可用。 - ➢ foreachRDD(func):这是最通用的输出操作,即将函数 func 用于产生于 stream 的每一个
RDD。其中参数传入的函数 func 应该实现将每一个 RDD 中数据推送到外部系统,如将
RDD 存入文件或者通过网络将其写入数据库。
六 优雅关闭
- Spark Streaming流程序比较特殊,所以不能直接执行kill -9 这种暴力方式停掉,如果使用这种方式停程序,那么就有可能丢失数据或者重复消费数据
- 有一种方案就是开启一个线程, 间隔检查第三方系统的标记(Redis,ZK,Hadoop),标记存在则调用streamContext.stop(true,true) 停止底层的SparkContext 同时 优雅的停止Streaming
object SparkStreaming_Close {
def main(args: Array[String]): Unit = {
val streamContext: StreamingContext = StreamingContext
.getActiveOrCreate("./checkPointDir", createStreamingContext)
streamContext.start()
new Thread(new StreamingMonitor(streamContext)).start()
streamContext.awaitTermination()
}
def createStreamingContext(): StreamingContext = {
val streamContext: StreamingContext = new StreamingContext(
new SparkConf().set("spark.streaming.stopGracefullyOnShutdown", "true").setMaster("local[*]")
.setAppName("WordCount"), Seconds.apply(5))
streamContext.checkpoint("./checkPointDir")
val dStream: ReceiverInputDStream[String] = streamContext.socketTextStream("node0", 9999)
val wordToOne: DStream[(String, Int)] = dStream.flatMap(_.split(" ")).map((_, 1))
val wordToCount: DStream[(String, Int)] = wordToOne.updateStateByKey((seq: Seq[Int], option: Option[Int]) => {
Option(seq.sum + option.getOrElse(0))
})
wordToCount.print()
streamContext
}
class StreamingMonitor(streamContext: StreamingContext) extends Runnable {
val SPARK_STREAMING_STOP_FLAG_ON_REDIS_KEY = "spark_streaming_key_stop_"
var loop_flag = true
override def run(): Unit = {
while (loop_flag) {
try {
Thread.sleep(5000)
} catch {
case e: Exception => e.printStackTrace()
}
if (RedisUtils.keyExist(SPARK_STREAMING_STOP_FLAG_ON_REDIS_KEY)) {
if (streamContext.getState() == StreamingContextState.ACTIVE) {
streamContext.stop(stopSparkContext = true, stopGracefully = true)
}
loop_flag = false
}
}
}
}
}