SparkStreaming源码阅读

在了解SparkStreaming之前,先来弄清几个变量:

  //生成SparkJob,使用JobGenerator来生成Spark Job,然后在一个线程池运行这些Job
  private[streaming] val scheduler = new JobScheduler(this)

  //
  private[streaming] val waiter = new ContextWaiter

  private[streaming] val progressListener = new StreamingJobProgressListener(this)

 

 

创建DStreamGraph,可以类比RDD,DStreamGraph类似于RDD dag

private[streaming] val graph: DStreamGraph = {
    if (isCheckpointPresent) {
      _cp.graph.setContext(this)
      _cp.graph.restoreCheckpointData()
      _cp.graph
    } else {
      require(_batchDur != null, "Batch duration for StreamingContext cannot be null")
      val newGraph = new DStreamGraph()
      newGraph.setBatchDuration(_batchDur)
      newGraph
    }
  }

    如果使用checkpoint,那么就从checkpoint中恢复graph，否则新建一个,DStreamGraph用来动态的创建DStream DAG ,

DStreamGraph有两个重要成员：

  private val inputStreams = new ArrayBuffer[InputDStream[_]]()
  private val outputStreams = new ArrayBuffer[DStream[_]]()

inputStreams集合用来记录所有的输入数据流

outputStreams集合用来记录所有的DStream