DelayedOperationPurgatory机制（六）：DelayedFetch之fetchMessages

DelayedFetch是FetchRequest对应的延迟操作，它的原理与DelayedProduce类似。消费者发送fetchRequest有KafkaApis.handleFetchRequest方法处理，他会调用FeplicaManager.fetcherMessages()方法从响应的log里读取消息，并生成delayedFetch添加到delayedFetchPurgatory中处理，delayedFetchPurgatory是ReplicaManager中的字段，它是专门用来处理DelayedFetch的DelayedOperationPurgatory对象。

  def fetchMessages(timeout: Long,
                    replicaId: Int,
                    fetchMinBytes: Int,
                    fetchInfo: immutable.Map[TopicAndPartition, PartitionFetchInfo],
                    responseCallback: Map[TopicAndPartition, FetchResponsePartitionData] => Unit) {
    // 只有Follower副本才有replicaId，而消费者是没有的，消费者的replicaId始终是-1
    val isFromFollower = replicaId >= 0
    val fetchOnlyFromLeader: Boolean = replicaId != Request.DebuggingConsumerId
    val fetchOnlyCommitted: Boolean = ! Request.isValidBrokerId(replicaId)

    // read from local logs
    // 从log中读取
    val logReadResults = readFromLocalLog(fetchOnlyFromLeader, fetchOnlyCommitted, fetchInfo)

    // if the fetch comes from the follower,
    // update its corresponding log end offset
    // 检测FetchRequest是否来自Follower副本
    if(Request.isValidBrokerId(replicaId))
      //updateFollowerLogReadResults主要用来处理来自Follwer副本的FetchRequest请求：
      // 1. 在leader中维护了follower副本的各种状态，这里会更新对应Follower副本的状态。
      // 2. 检测是否需要对ISR集合进行扩张如果ISR集合发生变更，就把新的ISR集合记录保存到zookeeper中
      // 3. 检测是否后移leader的HW
      // 4. 检测delayedProducePurgatory中相关key对应的DelayedProduce，满足条件则将其执行完成
      updateFollowerLogReadResults(replicaId, logReadResults)

    // check if this fetch request can be satisfied right away
    // 统计从log读取消息的字节总数
    val bytesReadable = logReadResults.values.map(_.info.messageSet.sizeInBytes).sum
    // 统计从log读取消息中是否有异常
    val errorReadingData = logReadResults.values.foldLeft(false) ((errorIncurred, readResult) =>
      errorIncurred || (readResult.errorCode != Errors.NONE.code))

    // respond immediately if 1) fetch request does not want to wait
    //                        2) fetch request does not require any data
    //                        3) has enough data to respond
    //                        4) some error happens while reading data
    // 是否可以返回Fetchresponse，满足以下条件之一：
    // 1. FetchRequest中timeout<=0，则消费者不希望等待
    // 2. FetchRequest没有指定要读取的分区
    // 3. 已经读取了足够的数据，即bytesReadable>=fetchMinBytes
    // 4. 读取数据中有异常
    if(timeout <= 0 || fetchInfo.size <= 0 || bytesReadable >= fetchMinBytes || errorReadingData) {
      val fetchPartitionData = logReadResults.mapValues(result =>
        FetchResponsePartitionData(result.errorCode, result.hw, result.info.messageSet))
        // 直接调用回调函数，发送FetchResponse
      responseCallback(fetchPartitionData)
    } else {
      // construct the fetch results from the read results
      val fetchPartitionStatus = logReadResults.map { case (topicAndPartition, result) =>
        (topicAndPartition, FetchPartitionStatus(result.info.fetchOffsetMetadata, fetchInfo.get(topicAndPartition).get))
      }
      val fetchMetadata = FetchMetadata(fetchMinBytes, fetchOnlyFromLeader, fetchOnlyCommitted, isFromFollower, fetchPartitionStatus)
      // 创建delayFetch对象
      val delayedFetch = new DelayedFetch(timeout, fetchMetadata, this, responseCallback)

      // create a list of (topic, partition) pairs to use as keys for this delayed fetch operation
      // 创建delayedFetchKeys
      val delayedFetchKeys = fetchPartitionStatus.keys.map(new TopicPartitionOperationKey(_)).toSeq

      // try to complete the request immediately, otherwise put it into the purgatory;
      // this is because while the delayed fetch operation is being created, new requests
      // may arrive and hence make this operation completable.
      // 尝试完成DelayedFetch，否者将DelayFetch添加到delayedFetchPurgatory中
      delayedFetchPurgatory.tryCompleteElseWatch(delayedFetch, delayedFetchKeys)
    }
  }

在处理ProducerRequest的过程中会向Log中添加数据，可能会后移leader副本的leo，Follower副本就可以读取到足量的数据，所以会尝试完成DelayedFetch；在处理来自Follower副本的FetchRequest中，可能会后移HW，所以会尝试完成DelayedProduce。