在了解SparkStreaming之前,先来弄清几个变量:
//生成SparkJob,使用JobGenerator来生成Spark Job,然后在一个线程池运行这些Job
private[streaming] val scheduler = new JobScheduler(this)
//
private[streaming] val waiter = new ContextWaiter
private[streaming] val progressListener = new StreamingJobProgressListener(this)
创建DStreamGraph,可以类比RDD,DStreamGraph类似于RDD dag
private[streaming] val graph: DStreamGraph = {
if (isCheckpointPresent) {
_cp.graph.setContext(this)
_cp.graph.restoreCheckpointData()
_cp.graph
} else {
require(_batchDur != null, "Batch duration for StreamingContext cannot be null")
val newGraph = new DStreamGraph()
newGraph.setBatchDuration(_batchDur)
newGraph
}
}
如果使用checkpoint,那么就从checkpoint中恢复graph,否则新建一个,DStreamGraph用来动态的创建DStream DAG ,
DStreamGraph有两个重要成员:
private val inputStreams = new ArrayBuffer[InputDStream[_]]()
private val outputStreams = new ArrayBuffer[DStream[_]]()
inputStreams集合用来记录所有的输入数据流
outputStreams集合用来记录所有的DStream