Kafka的Replica分配策略之二 Replica变为0了怎么办

这一篇文章准备讨论当kafka集群的broker发生变化,诸如broker崩溃,退出时,kafka集群会如何分配该broker上的Replica和Partition.

在讨论这个问题之前,需要先搞清kafka集群中,leader与follower的分工.可以看我写的这篇文章 Kafka的leader选举过程

在之前介绍kafka的选举过程时,提到成功选举出的leader会向zookeeper注册各种监视其中

replicaStateMachine.registerListeners()    //"/brokers/ids" 重点,监视所有的follower的加入,离开集群的行为

这一句注册了对/brokers/ids的监视,跟进这句命令

private def registerBrokerChangeListener() = {
  zkUtils.zkClient.subscribeChildChanges(ZkUtils.BrokerIdsPath, brokerChangeListener)
}

来到了这里,ZkUtils.BrokerIdsPath就是/brokers/ids这个路径,那由此可以得知,重点就在brokerChangeListener上.这个listener被定义在kafka.controller包的ReplicaStateMachine.scala下.

class BrokerChangeListener() extends IZkChildListener with Logging {
  this.logIdent = "[BrokerChangeListener on Controller " + controller.config.brokerId + "]: "
  def handleChildChange(parentPath : String, currentBrokerList : java.util.List[String]) {
    info("Broker change listener fired for path %s with children %s".format(parentPath, currentBrokerList.sorted.mkString(",")))
    inLock(controllerContext.controllerLock) {
      if (hasStarted.get) {
        ControllerStats.leaderElectionTimer.time {
          try {
            val curBrokers = currentBrokerList.map(_.toInt).toSet.flatMap(zkUtils.getBrokerInfo)
            val curBrokerIds = curBrokers.map(_.id)
            val liveOrShuttingDownBrokerIds = controllerContext.liveOrShuttingDownBrokerIds
            val newBrokerIds = curBrokerIds -- liveOrShuttingDownBrokerIds
            val deadBrokerIds = liveOrShuttingDownBrokerIds -- curBrokerIds
            val newBrokers = curBrokers.filter(broker => newBrokerIds(broker.id))
            //上面几句很好理解,筛选出新加入的broker,与退出的broker
            controllerContext.liveBrokers = curBrokers
            val newBrokerIdsSorted = newBrokerIds.toSeq.sorted
            val deadBrokerIdsSorted = deadBrokerIds.toSeq.sorted
            val liveBrokerIdsSorted = curBrokerIds.toSeq.sorted
            info("Newly added brokers: %s, deleted brokers: %s, all live brokers: %s"
              .format(newBrokerIdsSorted.mkString(","), deadBrokerIdsSorted.mkString(","), liveBrokerIdsSorted.mkString(",")))
            newBrokers.foreach(controllerContext.controllerChannelManager.addBroker)
            deadBrokerIds.foreach(controllerContext.controllerChannelManager.removeBroker)
            //上面两句维护存有broker信息的map
            if(newBrokerIds.nonEmpty)
              controller.onBrokerStartup(newBrokerIdsSorted)
            if(deadBrokerIds.nonEmpty)
              //这一句是重点,如何处理fail的broker
              controller.onBrokerFailure(deadBrokerIdsSorted)
          } catch {
            case e: Throwable => error("Error while handling broker changes", e)
          }
        }
      }
    }
  }
}

可以看到,针对broker变化这一情况.Kafka controller从znode节点的变化,推测出了新加入与新离开的节点.对于离开的节点调用了onBrokerFailure函数.

继续跟进这里只截取了部分onBrokerFailure的源码,一段一段来分析.

def onBrokerFailure(deadBrokers: Seq[Int]){
.....
....
val deadBrokersSet = deadBrokers.toSet
// trigger OfflinePartition state for all partitions whose current leader is one amongst the dead brokers
//筛选出dead的broker中所有担任partition leader