spark的异步消息总线LiveListenBus

LiveListenBus是spark提供的异步消息总线。

private val queues = new CopyOnWriteArrayList[AsyncEventQueue]()

@volatile private[scheduler] var queuedEvents = new mutable.ListBuffer[SparkListenerEvent]()

在LiveListenBus中，通过线程安全的CopyOnWriteArrayList来保存AsyncEventQueue，AsyncEventQueue为异步的消息队列，具体的监听器listener保存在其中。

queuedEvents为保存当消息总校还没有启动完毕，但是消息时间已经发布到消息总线时的情况，这时候的消息会直接保存在这个ListBuffer中，直到启动后被发布到消息总线中。

def start(sc: SparkContext, metricsSystem: MetricsSystem): Unit = synchronized {
  if (!started.compareAndSet(false, true)) {
    throw new IllegalStateException("LiveListenerBus already started.")
  }

  this.sparkContext = sc
  queues.asScala.foreach { q =>
    q.start(sc)
    queuedEvents.foreach(q.post)
  }
  queuedEvents = null
  metricsSystem.registerSource(metrics)
}

看到其start()方法，当其start()方法调用，代表该异步消息总线开始工作，在开始的过程中，一次调用所有queues中的AsyncEventQueue的start()方法，表示各个异步消息队列开始工作，并将上文提到分queuedEvents容器中的所有存放的消息时间发布到各个启动的异步队列中并在发布完毕所有存量事件之后，将原本的queuedEvents置为null，代表该异步消息总线已经中的所有异步消息队列已经全部启动。

def post(event: SparkListenerEvent): Unit = {
  if (stopped.get()) {
    return
  }

  metrics.numEventsPosted.inc()

  // If the event buffer is null, it means the bus has been started and we can avoid
  // synchronization and post events directly to the queues. This should be the most
  // common case during the life of the bus.
  if (queuedEvents == null) {
    postToQueues(event)
    return
  }

  // Otherwise, need to synchronize to check whether the bus is started, to make sure the thread
  // calling start() picks up the new event.
  synchronized {
    if (!started.get()) {
      queuedEvents += event
      return
    }
  }

  // If the bus was already started when the check above was made, just post directly to the
  // queues.
  postToQueues(event)
}

private def postToQueues(event: SparkListenerEvent): Unit = {
  val it = queues.iterator()
  while (it.hasNext()) {
    it.next().post(event)
  }
}

当消息总线中存在事件需要被发布，那么需要调用post()方法进行调用，如果queuedEvents为null，说明该消息总线中的所有异步消息队列已经全部启动，那么直接通过postToqueues()方法将事件依次通过各个消息队列的psot()方法进行发布，否则存放在queuedEvents中，等待各个队列启动后进行发布。

 

AsyncEventQueue的核心是下面这个dispatchThread。

private val dispatchThread = new Thread(s"spark-listener-group-$name") {
  setDaemon(true)
  override def run(): Unit = Utils.tryOrStopSparkContext(sc) {
    dispatch()
  }
}

private[scheduler] def start(sc: SparkContext): Unit = {
  if (started.compareAndSet(false, true)) {
    this.sc = sc
    dispatchThread.start()
  } else {
    throw new IllegalStateException(s"$name already started!")
  }
}

在AsyncEventQueue的构造过程中，会定义该线程，并设置其为守护线程，其run()方法为直接调用dispatch()方法。AsyncEventQueue的start()方法也是直接启动该线程。

if (stopped.get()) {
  return
}

eventCount.incrementAndGet()
if (eventQueue.offer(event)) {
  return
}

当消息总校调用了AsyncEventQueue的post()方法时，是直接将其放入到队列当中eventQueue中，等待之前的dispatchThread线程获取并处理。

private def dispatch(): Unit = LiveListenerBus.withinListenerThread.withValue(true) {
  var next: SparkListenerEvent = eventQueue.take()
  while (next != POISON_PILL) {
    val ctx = processingTime.time()
    try {
      super.postToAll(next)
    } finally {
      ctx.stop()
    }
    eventCount.decrementAndGet()
    next = eventQueue.take()
  }
  eventCount.decrementAndGet()
}

在dispatch线程中，会不断从队列中获取发布的时间，并通过postToAll()发布到其中的监听器listener中。

postToAll()的具体逻辑实现在ListenBus这一特征中，在这里会调用该特征中所有具体的监听器去对该事件进行通知，在这里会变成同步调用。

具体的方法处理，可以参考SparkListenBus的doPostEvent()方法。

protected override def doPostEvent(
    listener: SparkListenerInterface,
    event: SparkListenerEvent): Unit = {
  event match {
    case stageSubmitted: SparkListenerStageSubmitted =>
      listener.onStageSubmitted(stageSubmitted)
    case stageCompleted: SparkListenerStageCompleted =>
      listener.onStageCompleted(stageCompleted)
    case jobStart: SparkListenerJobStart =>
      listener.onJobStart(jobStart)
    case jobEnd: SparkListenerJobEnd =>
      listener.onJobEnd(jobEnd)
    case taskStart: SparkListenerTaskStart =>
      listener.onTaskStart(taskStart)
    case taskGettingResult: SparkListenerTaskGettingResult =>
      listener.onTaskGettingResult(taskGettingResult)
    case taskEnd: SparkListenerTaskEnd =>
      listener.onTaskEnd(taskEnd)
    case environmentUpdate: SparkListenerEnvironmentUpdate =>
      listener.onEnvironmentUpdate(environmentUpdate)
    case blockManagerAdded: SparkListenerBlockManagerAdded =>
      listener.onBlockManagerAdded(blockManagerAdded)
    case blockManagerRemoved: SparkListenerBlockManagerRemoved =>
      listener.onBlockManagerRemoved(blockManagerRemoved)
    case unpersistRDD: SparkListenerUnpersistRDD =>
      listener.onUnpersistRDD(unpersistRDD)
    case applicationStart: SparkListenerApplicationStart =>
      listener.onApplicationStart(applicationStart)
    case applicationEnd: SparkListenerApplicationEnd =>
      listener.onApplicationEnd(applicationEnd)
    case metricsUpdate: SparkListenerExecutorMetricsUpdate =>
      listener.onExecutorMetricsUpdate(metricsUpdate)
    case executorAdded: SparkListenerExecutorAdded =>
      listener.onExecutorAdded(executorAdded)
    case executorRemoved: SparkListenerExecutorRemoved =>
      listener.onExecutorRemoved(executorRemoved)
    case executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage =>
      listener.onExecutorBlacklistedForStage(executorBlacklistedForStage)
    case nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage =>
      listener.onNodeBlacklistedForStage(nodeBlacklistedForStage)
    case executorBlacklisted: SparkListenerExecutorBlacklisted =>
      listener.onExecutorBlacklisted(executorBlacklisted)
    case executorUnblacklisted: SparkListenerExecutorUnblacklisted =>
      listener.onExecutorUnblacklisted(executorUnblacklisted)
    case nodeBlacklisted: SparkListenerNodeBlacklisted =>
      listener.onNodeBlacklisted(nodeBlacklisted)
    case nodeUnblacklisted: SparkListenerNodeUnblacklisted =>
      listener.onNodeUnblacklisted(nodeUnblacklisted)
    case blockUpdated: SparkListenerBlockUpdated =>
      listener.onBlockUpdated(blockUpdated)
    case speculativeTaskSubmitted: SparkListenerSpeculativeTaskSubmitted =>
      listener.onSpeculativeTaskSubmitted(speculativeTaskSubmitted)
    case _ => listener.onOtherEvent(event)
  }
}

将会根据发布的时间类型，调用监听器listener对应的方法达成对消息的反应。