Spark分析（七）Spark Streaming运行流程详解（2）

最新推荐文章于 2023-03-20 14:11:36 发布

SDU_可颂

最新推荐文章于 2023-03-20 14:11:36 发布

阅读量1k

点赞数

分类专栏：项目实训（2021年）文章标签： spark

本文链接：https://blog.csdn.net/m0_55939339/article/details/121385369

版权

2021SC@SDUSC

前言

上一篇博客分析了StreamingContext的部分代码，本篇将继续分析DStream.print

DStream.print

下图为DStream的输出操作print的流程

DStream的输出操作print的流程

DStream的输出操作print的流程

先来看看DStream.print是如何把ForeachDStream对象注册到DStreamGraph中的。

// DStream.print
/**
 * Print the first ten elements of each RDD generated in this DStream. This is an output
 * operator, so this DStream will be registered as an output stream and there
 * materialized
 */
 
def print():Unit = ssc.withScope {
   
	print(10)
}

其中的print调用了另一个重载的print

// DStream.rpint
/**
 * Print the first num elements of each RDD generated in this DStream. This is an output
 * operator, so this DStream will be registered as an output stream and there 
 * materialized
 */

def print(num: Int): Unit = ssc.withScope {
   
	def foreachFunc: (RDD[T], Time) => Unit = {
   
		(rdd: RDD[T], time: Time) => {
   
			val firstNum = rdd.take（num + 1)
			// scalastyle:off println
			println("-----------------------")
			println("Time: " + time)
			println("-----------------------")
			firstNum.take(num).foreach(println)
			if (firstNum.length > num) println("...")
			println(</