Spark-Streaming之window滑动窗口应用,Spark Streaming提供了滑动窗口操作的支持,从而让我们可以对一个滑动窗口内的数据执行计算操作。每次掉落在窗口内的RDD的数据,会被聚合起来执行计算操作,然后生成的RDD,会作为window DStream的一个RDD。
网官图中所示,就是对每三秒钟的数据执行一次滑动窗口计算,这3秒内的3个RDD会被聚合起来进行处理,然后过了两秒钟,又会对最近三秒内的数据执行滑动窗口计算。所以每个滑动窗口操作,都必须指定两个参数,窗口长度以及滑动间隔,而且这两个参数值都必须是batch间隔的整数倍。
Spark Streaming对滑动窗口的支持,是比Storm更加完善和强大的。
Spark-Streaming对滑动窗口支持的转换操作:
热点搜索词滑动统计,每隔10秒钟,统计最近60秒钟的搜索词的搜索频次,并打印出排名最靠前的3个搜索词以及出现次数
scala版本:
- package com.spark.streaming
-
- import org.apache.spark.streaming.Seconds
- import org.apache.spark.streaming.StreamingContext
- import org.apache.spark.SparkConf
-
-
-
-
- object WindowHotWordS {
- def main(args: Array[String]): Unit = {
- val conf = new SparkConf().setAppName("WindowHotWordS").setMaster("local[2]")
-
-
- val ssc = new StreamingContext(conf, Seconds(5))
-
- val searchLogsDStream = ssc.socketTextStream("spark1", 9999)
-
- val searchWordsDStream = searchLogsDStream.map { searchLog => searchLog.split(" ")(1) }
-
- val searchWordPairDStream = searchWordsDStream.map { searchWord => (searchWord, 1) }
-
-
-
-
-
-
-
-
-
- val searchWordCountsDStream = searchWordPairDStream.reduceByKeyAndWindow((v1: Int, v2: Int) => v1 + v2, Seconds(60), Seconds(10))
-
-
- val finalDStream = searchWordCountsDStream.transform(searchWordCountsRDD => {
- val countSearchWordsRDD = searchWordCountsRDD.map(tuple => (tuple._2, tuple._1))
- val sortedCountSearchWordsRDD = countSearchWordsRDD.sortByKey(false)
- val sortedSearchWordCountsRDD = sortedCountSearchWordsRDD.map(tuple => (tuple._1, tuple._2))
- val top3SearchWordCounts = sortedSearchWordCountsRDD.take(3)
-
- for (tuple <- top3SearchWordCounts) {
- println("result : " + tuple)
- }
-
- searchWordCountsRDD
- })
-
- finalDStream.print()
-
- ssc.start()
- ssc.awaitTermination()
- }
- }