kafka源码解析之九ReplicaManager

最新推荐文章于 2024-07-17 17:40:59 发布

亮亮-AC米兰

最新推荐文章于 2024-07-17 17:40:59 发布

阅读量4.5k

点赞数 1

分类专栏： kafka kafka源码解析文章标签： kafka 源码

本文链接：https://blog.csdn.net/wl044090432/article/details/51035614

版权

本文深入解析Kafka的ReplicaManager，探讨其在管理副本角色中的核心功能，包括处理LeaderAndISRCommand、StopReplicaCommand以及maybeShrinkIsr的流程，以确保AR和ISR的有效维护。

摘要由CSDN通过智能技术生成

首先解释下2个名词：

AR(assignreplicas):分配副本 ISR(in-sync replicas)：在同步中的副本，即下图：

Partition {                              
  topic                   : string           //topic名称
  partition_id            : int          //partition id
  leader                  : Replica           // 这个分区的leader副本，是isr中的其中一个
  ISR                     : Set[Replica]      // 正在同步中的副本集合
  AR                      : Set[Replica]      // 这个分区的所有副本分配集合，一个broker上有至多一个分区副本
  LeaderAndISRVersionInZK : long    // version id of the LeaderAndISR path; used for conditionally update the LeaderAndISR path in ZK
}
Replica {                                // 一个分区副本信息
  broker_id               : int
  partition               : Partition    //分区信息
  log                     : Log          //本地日志与副本关联信息
  hw                      : long         //最后被commit的message的offset信息
  leo                     : long         // 日志结尾offset
  isLeader                : Boolean      //是否为该副本的leader
}

接下来来看ReplicaManager的主要作用，它的角色定位是负责接收controller的command以完成replica的管理工作，command主要有两种， LeaderAndISRCommand和StopReplicaCommand。因此主要完成三件事：

1）接受LeaderAndISRCommand命令 2）接受StopReplicaCommand命令 3）开启定时线程maybeShrinkIsr

，以便发现那些已经没有进行同步的复本

9.1 LeaderAndISRCommand处理流程

当KafkaServer接受到LeaderAndIsrRequest指令时，会调用ReplicaManager的becomeLeaderOrFollower函数

def becomeLeaderOrFollower(leaderAndISRRequest: LeaderAndIsrRequest,
                           offsetManager: OffsetManager): (collection.Map[(String, Int), Short], Short) = {
  leaderAndISRRequest.partitionStateInfos.foreach { case ((topic, partition), stateInfo) =>
    stateChangeLogger.trace("Broker %d received LeaderAndIsr request %s correlation id %d from controller %d epoch %d for partition [%s,%d]"
                              .format(localBrokerId, stateInfo, leaderAndISRRequest.correlationId,
                                      leaderAndISRRequest.controllerId, leaderAndISRRequest.controllerEpoch, topic, partition))
  }
  replicaStateChangeLock synchronized {
    val responseMap = new collection.mutable.HashMap[(String, Int), Short]
    if(leaderAndISRRequest.controllerEpoch < controllerEpoch) { // 检查requset epoch
      leaderAndISRRequest.partitionStateInfos.foreach { case ((topic, partition), stateInfo) =>
      stateChangeLogger.warn(("Broker %d ignoring LeaderAndIsr request from controller %d with correlation id %d since " +
        "its controller epoch %d is old. Latest known controller epoch is %d").format(localBrokerId, leaderAndISRRequest.controllerId,
        leaderAndISRRequest.correlationId, leaderAndISRRequest.controllerEpoch, controllerEpoch))
      }
      (responseMap, ErrorMapping.StaleControllerEpochCode)
    } else {
      val controllerId = leaderAndISRRequest.controllerId
      val correlationId = leaderAndISRRequest.correlationId
      controllerEpoch = leaderAndISRRequest.controllerEpoch

      // First check partition's leader epoch
      // 前面只是检查了request的epoch，但是还要检查其中的每个partitionStateInfo中的leader epoch
      val partitionState = new HashMap[Partition, PartitionStateInfo]()
      leaderAndISRRequest.partitionStateInfos.foreach{ case ((topic, partitionId), partitionStateInfo) =>
        val partition = getOrCreatePartition(topic, partitionId)
        val partitionLeaderEpoch = partition.getLeaderEpoch()
        // If the leader epoch is valid record the epoch of the controller that made the leadership decision.
        // This is useful while updating the isr to maintain the decision maker controller's epoch in the zookeeper path
        // local的partitionLeaderEpoch要小于request中的leaderEpoch，否则就是过时的request
        if (partitionLeaderEpoch < partitionStateInfo.leaderIsrAndControllerEpoch.leaderAndIsr.leaderEpoch) {
          // 判断该partition是否被assigned给当前的broker
          if(partitionStateInfo.allReplicas.contains(config.brokerId))
          // 只将被分配到当前broker的partition放入partitionState，其中partition是当前的状况，partitionStateInfo是request中最新的状况
            partitionState.put(partition, partitionStateInfo)
          else {
            stateChangeLogger.warn(("Broker %d ignoring LeaderAndIsr