Kafka 延迟操作模块（三）：DelayedOperation 与 DelayedOperationPurgatory

本文链接：https://blog.csdn.net/lianggx3/article/details/108892247

DelayedOperation

DelayedOperation 是延时任务的抽象，它实现了 TimerTask 特质，

abstract class DelayedOperation(override val delayMs: Long, // DelayedOperation 类是一个抽象类，它的构造函数中只需要传入一个超时时间即可。
                                                            // 这个超时时间通常是客户端发出请求的超时时间
                                lockOpt: Option[Lock] = None)
  extends TimerTask with Logging {
  // 标识该延迟操作是否已经完成
  private val completed = new AtomicBoolean(false)
  // 防止多个线程同时检查操作是否可完成时发生锁竞争导致操作最终超时
  // 确保拿到锁的线程有机会再次检查条件是否已经满足。
  private val tryCompletePending = new AtomicBoolean(false)
  // Visible for testing
  private[server] val lock: Lock = lockOpt.getOrElse(new ReentrantLock)
}

并定义了以下方法：

forceComplete：强制完成延迟操作，不管它是否满足完成条件。每当操作满足完成条件或已经过期了，就需要调用该方法完成该操作。
isCompleted：检查延迟操作是否已经完成。源码使用这个方法来决定后续如何处理该操作。比如如果操作已经完成了，那么通常需要取消该操作。
onExpiration：强制完成之后执行的过期逻辑回调方法。只有真正完成操作的那个线程才有资格调用这个方法。
onComplete：完成延迟操作所需的处理逻辑。这个方法只会在 forceComplete 方法中被调用。
tryComplete：尝试完成延迟操作的顶层方法，内部会调用 forceComplete 方法。
maybeTryComplete：线程安全版本的 tryComplete 方法。这个方法其实是社区后来才加入的，不过已经慢慢地取代了 tryComplete，现在外部代码调用的都是这个方法了。
run：调用延迟操作超时后的过期逻辑，也就是组合调用 forceComplete + onExpiration。

abstract class DelayedOperation(override val delayMs: Long, // DelayedOperation 类是一个抽象类，它的构造函数中只需要传入一个超时时间即可。
                                                            // 这个超时时间通常是客户端发出请求的超时时间
                                lockOpt: Option[Lock] = None)
  extends TimerTask with Logging {
  // 标识该延迟操作是否已经完成
  private val completed = new AtomicBoolean(false)
  // 防止多个线程同时检查操作是否可完成时发生锁竞争导致操作最终超时
  // 确保拿到锁的线程有机会再次检查条件是否已经满足。
  private val tryCompletePending = new AtomicBoolean(false)
  // Visible for testing
  private[server] val lock: Lock = lockOpt.getOrElse(new ReentrantLock)

  /*
   * Force completing the delayed operation, if not already completed.
   * This function can be triggered when
   *
   * 1. The operation has been verified to be completable inside tryComplete()
   * 2. The operation has expired and hence needs to be completed right now
   *
   * Return true iff the operation is completed by the caller: note that
   * concurrent threads can try to complete the same operation, but only
   * the first thread will succeed in completing the operation and return
   * true, others will still return false
   */
  // 强制完成延迟操作，不管它是否满足完成条件。
  // 每当操作满足完成条件或已经过期了，就需要调用该方法完成该操作。
  def forceComplete(): Boolean = {
    if (completed.compareAndSet(false, true)) {
      // cancel the timeout timer
      // 将当前延时任务从时间轮中移除
      cancel()
      // 立即触发执行延时任务
      onComplete()
      true
    } else {
      false
    }
  }

  /**
   * Check if the delayed operation is already completed
   */
  // 检查延迟操作是否已经完成。源码使用这个方法来决定后续如何处理该操作。比如如果操作已经完成了，那么通常需要取消该操作。
  def isCompleted: Boolean = completed.get()

  /**
   * Call-back to execute when a delayed operation gets expired and hence forced to complete.
   */
  // 强制完成之后执行的过期逻辑回调方法。只有真正完成操作的那个线程才有资格调用这个方法。
  def onExpiration(): Unit

  /**
   * Process for completing an operation; This function needs to be defined
   * in subclasses and will be called exactly once in forceComplete()
   */
  //.完成延迟操作所需的处理逻辑。这个方法只会在 forceComplete 方法中被调用。
  def onComplete(): Unit

  /**
   * Try to complete the delayed operation by first checking if the operation
   * can be completed by now. If yes execute the completion logic by calling
   * forceComplete() and return true iff forceComplete returns true; otherwise return false
   *
   * This function needs to be defined in subclasses
   */
  // 尝试完成延迟操作的顶层方法，内部会调用 forceComplete 方法。
  def tryComplete(): Boolean

  /*
   * run() method defines a task that is executed on timeout
   */
  // 调用延迟操作超时后的过期逻辑，也就是组合调用 forceComplete + onExpiration。
  override def run(): Unit = {
    if (forceComplete())
      onExpiration()
  }
}

单独把 maybeTryComplete 拿出来讲讲。加入 tryCompletePending 字段目的，就是确保拿到锁的线程有机会再次检查条件是否已经满足。

  // 线程安全版本的 tryComplete 方法。
  private[server] def maybeTryComplete(): Boolean = {
    var retry = false // 是否需要重试
    var done = false // 延迟操作是否已完成
    do {
      if (lock.tryLock()) { // 尝试获取锁对象
        try {
          tryCompletePending.set(false)
          done = tryComplete()
        } finally {
          lock.unlock()
        }
        // While we were holding the lock, another thread may have invoked `maybeTryComplete` and set
        // `tryCompletePending`. In this case we should retry.
        // 运行到这里的线程持有锁，其他线程只能运行else分支的代码
        // 如果其他线程将maybeTryComplete设置为true，那么retry=true
        // 这就相当于其他线程给了本线程重试的机会
        retry = tryCompletePending.get()
      } else {
        // Another thread is holding the lock. If `tryCompletePending` is already set and this thread failed to
        // acquire the lock, then the thread that is holding the lock is guaranteed to see the flag and retry.
        // Otherwise, we should set the flag and retry on this thread since the thread holding the lock may have
        // released the lock and returned by the time the flag is set.
        // 运行到这里的线程没有拿到锁
        // 设置tryCompletePending=true给持有锁的线程一个重试的机会
        retry = !tryCompletePending.getAndSet(true)
      }
    } while (!isCompleted && retry)
    done
  }

DelayedOperation

DelayedOperationPurgatory 类提供了对 DelayedOperation 进行管理的功能。

final class DelayedOperationPurgatory[T <: DelayedOperation](purgatoryName: String,
                                                             timeoutTimer: Timer, // 定时器
                                                             brokerId: Int = 0, // 所在 broker 节点 ID
                                                             purgeInterval: Int = 1000, // 控制删除线程移除 Bucket 中的过期延迟请求的频率，在绝大部分情况下，都是 1 秒一次。
                                                             reaperEnabled: Boolean = true, // 是否启用后台指针推进器
                                                             timerEnabled: Boolean = true)
        extends Logging with KafkaMetricsGroup {
  /* a list of operation watching keys */
  private class WatcherList {
    // 定义一组按照Key分组的Watchers对象
    /** 用于管理 DelayedOperation，其中 key 是 Watcher 中的 DelayedOperation 集合所关心的对象 */
    val watchersByKey = new Pool[Any, Watchers](Some((key: Any) => new Watchers(key)))
    /** watchersForKey 读写锁 */
    val watchersLock = new ReentrantLock()

    /*
     * Return all the current watcher lists,
     * note that the returned watchers may be removed from the list by other threads
     */
    // 返回所有Watchers对象
    def allWatchers = {
      watchersByKey.values
    }
  }

  private val watcherLists = Array.fill[WatcherList](DelayedOperationPurgatory.Shards)(new WatcherList)
  private def watcherList(key: Any): WatcherList = {
    watcherLists(Math.abs(key.hashCode() % watcherLists.length))
  }

  // the number of estimated total operations in the purgatory
  /** 记录当前 DelayedOperationPurgatory 中延时任务的个数 */
  private[this] val estimatedTotalOperations = new AtomicInteger(0)

  /* background thread expiring operations that have timed out */
  /**
    * 主要具备 2 个作用：
    *  1. 推进时间指针
    *  2. 定期清理 watchersForKey 中已经完成的延时任务
    */
  private val expirationReaper = new ExpiredOperationReaper()
}

DelayedOperationPurgatory 类的两个重要方法：tryCompleteElseWatch 和 checkAndComplete 方法。

tryCompleteElseWatch

tryCompleteElseWatch的作用是检查操作是否能够完成，

   def tryCompleteElseWatch(operation: T, watchKeys: Seq[Any]): Boolean = {
    assert(watchKeys.nonEmpty, "The watch key list can't be empty")

    // The cost of tryComplete() is typically proportional to the number of keys. Calling
    // tryComplete() for each key is going to be expensive if there are many keys. Instead,
    // we do the check in the following way. Call tryComplete(). If the operation is not completed,
    // we just add the operation to all keys. Then we call tryComplete() again. At this time, if
    // the operation is still not completed, we are guaranteed that it won't miss any future triggering
    // event since the operation is already on the watcher list for all keys. This does mean that
    // if the operation is completed (by another thread) between the two tryComplete() calls, the
    // operation is unnecessarily added for watch. However, this is a less severe issue since the
    // expire reaper will clean it up periodically.

    // At this point the only thread that can attempt this operation is this current thread
    // Hence it is safe to tryComplete() without a lock
    // 尝试完成延迟请求
    var isCompletedByMe = operation.tryComplete()
    // 如果该延迟请求是由本线程完成的，直接返回true即可
    // 如果返回结果是 true，就说明执行 tryCompleteElseWatch 方法的线程正常地完成了该延迟请求，
    // 也就不需要再添加到 WatcherList 了，直接返回 true 就行了。
    if (isCompletedByMe)
      return true

    var watchCreated = false
    for(key <- watchKeys) {
      // If the operation is already completed, stop adding it to the rest of the watcher list.
      // 再次查看请求的完成状态，如果已经完成，就说明是被其他线程完成的，返回false
      if (operation.isCompleted)
        return false
      // 否则，将该operation加入到Key所在的WatcherList
      watchForOperation(key, operation)
      // 设置watchCreated标记，表明该任务已经被加入到WatcherList中，等待后续完成。
      if (!watchCreated) {
        watchCreated = true
        estimatedTotalOperations.incrementAndGet()
      }
    }
    // 再次尝试完成该延迟请求
    isCompletedByMe = operation.maybeTryComplete()
    if (isCompletedByMe)
      return true

    // if it cannot be completed by now and hence is watched, add to the expire queue also
    // 如果依然不能完成此请求，将其加入到过期队列
    if (!operation.isCompleted) {
      if (timerEnabled)
        timeoutTimer.add(operation)
      if (operation.isCompleted) {
        // cancel the timer task
        operation.cancel()
      }
    }

    false
  }

checkAndComplete

检查给定 Key 所在的 WatcherList 中的延迟请求是否满足完成条件，如果是的话，则结束掉它们。

  def checkAndComplete(key: Any): Int = {
    // 获取给定Key的WatcherList
    val wl = watcherList(key)
    // 获取WatcherList中Key对应的Watchers对象实例
    val watchers = inLock(wl.watchersLock) { wl.watchersByKey.get(key) }
    // 尝试完成满足完成条件的延迟请求并返回成功完成的请求数
    val numCompleted = if (watchers == null)
      0
    else
      watchers.tryCompleteWatched()
    debug(s"Request key $key unblocked $numCompleted $purgatoryName operations")
    numCompleted
  }