Kafka的Repilica分配策略之一

最新推荐文章于 2024-07-12 10:30:01 发布

Rapper金馆长

最新推荐文章于 2024-07-12 10:30:01 发布

阅读量892

点赞数 3

分类专栏： kafka 文章标签： kafka 源码

本文链接：https://blog.csdn.net/qq_21519863/article/details/78562669

版权

kafka 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

近来对kafka的源码很有兴趣.

说起kafka真的是极大地方便了我的工作,每次用起ta都心怀感激.好用不说,还相当稳定.

爱屋及乌,我决心一探kafka的究竟.

对我来说最感兴趣的莫过于这几个个问题:

1.在创建topic的时候,kafka如何分配partition以及replica所在的位置.

2.要是一个broker down了,那它的replica该怎么重新分配.

3.如果一个broker因为2成为了一个topic的新replica,那么他没有之前的那些message该怎么办?需要从其他broker拉过来吗,如果要拉,那么数据量太大的会不会对网络造成负载?

4.kafka的Ack机制.当producer发来一条消息,是经由leader转发,还是可以直接放到replica所在的broker上.当一个broker存下了消息,就发ack给produce,r还是等大多数/所有replica存下了消息再发ack给producer.

这篇博客先讨论第一个问题.

当客户端发起create topic请求时,在broker上会有这样的调用栈.

KafkaApis.handleCreateTopicsRequest()->adminManager.createTopics()->AdminUtils.assignReplicasToBrokers()

真正关于assignReplicas就在assignReplicasToBrokers这个函数中完成.

现来看AdminUtils这个类中,作者在源码上的注释

* There are 3 goals of replica assignment:
*
* 1. Spread the replicas evenly among brokers.
* 2. For partitions assigned to a particular broker, their other replicas are spread over the other brokers.
* 3. If all brokers have rack information, assign the replicas for each partition to different racks if possible
*
* To achieve this goal for replica assignment without considering racks, we:
* 1. Assign the first replica of each partition by round-robin, starting from a random position in the broker list.
* 2. Assign the remaining replicas of each partition with an increasing shift.

重点是下面两条!

1.从broker-list中选定一个随机的位置,从这个位置开始,将每一个partition的第一个replica依次赋予brokerList中的broker.

比如现在有broker0~4,同时该topic有10个partition,随机选定的起始位置是broker0,那么就从broker0开始依次赋予partition,当partition超过了broker的数目时,再回到一开始选定的broker开始分配,就是如下效果.

* broker-0  broker-1  broker-2  broker-3  broker-4
* p0        p1        p2        p3        p4       (1st replica)
* p5        p6        p7        p8        p9       (1st replica)

2.当分配好了第一个replica之后,剩下的replica以第一个replica所在的broker为基准,依次赋予之后的broker

比如partition0的replica0给了broker2,那么partion0的replica1与replica2依次赋予broker3和broker4

* broker-0  broker-1  broker-2  broker-3  broker-4
* p0        p1        p2        p3        p4       (1st replica)
* p5        p6        p7        p8        p9       (1st replica)
* p4        p0        p1        p2        p3       (2nd replica)
* p8        p9        p5        p6        p7       (2nd replica)
* p3        p4        p0        p1        p2       (3nd replica)
* p7        p8        p9        p5        p6       (3nd replica)

那么brokerList从哪里来?

这就有关于kafka与zookeeper的协作了,kafka问zookeeper要自己所在kafka集群的brokerList.

谁问zookeeper要brokerList?

kafka集群会通过zookeeper选出一个集群中的leader,由这个leader与zookeeper交互.选举或者加入集群成为follower在一个broker初始化的时候完成.

下面上一些源码,均摘自AdminUtils类.在这篇文章中,我暂且不讨论考虑racker的情况,并去掉了一些错误检查的部分.

private def assignReplicasToBrokersRackUnaware(nPartitions: Int,
                                               replicationFactor: Int,
                                               brokerList: Seq[Int],
                                               fixedStartIndex: Int,
                                               startPartitionId: Int): Map[Int, Seq[Int]] = {
  val ret = mutable.Map[Int, Seq[Int]]()
  val brokerArray = brokerList.toArray
  val startIndex = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(brokerArray.length)
  var currentPartitionId = math.max(0, startPartitionId)
  var nextReplicaShift = if (fixedStartIndex >= 0) fixedStartIndex else rand.nextInt(brokerArray.length)
  for (_ <- 0 until nPartitions) {
    if (currentPartitionId > 0 && (currentPartitionId % brokerArray.length == 0))
      nextReplicaShift += 1
    val firstReplicaIndex = (currentPartitionId + startIndex) % brokerArray.length
    val replicaBuffer = mutable.ArrayBuffer(brokerArray(firstReplicaIndex))
    for (j <- 0 until replicationFactor - 1)
      replicaBuffer += brokerArray(replicaIndex(firstReplicaIndex, nextReplicaShift, j, brokerArray.length))
    ret.put(currentPartitionId, replicaBuffer)
    currentPartitionId += 1
  }
  ret
}

首先关注一下这个方法的返回值,这是一个map的返回值.key是partition的ID,而value是这个partition所在的brokerList.

每个partition的第一个replica分配方式同我上文所说的相同,而这个partiion剩下的replica分配方式与我上文所说的实现有稍微的不同,就是加入了nextReplicashift.总之就是经过了某些的运算,replica并不是在broker之间依次分配下去的.而是间隔了nextReplicaShift个broker分配的.

private def replicaIndex(firstReplicaIndex: Int, secondReplicaShift: Int, replicaIndex: Int, nBrokers: Int): Int = {
  val shift = 1 + (secondReplicaShift + replicaIndex) % (nBrokers - 1)
  (firstReplicaIndex + shift) % nBrokers
}

可以看到的,每两个replica之间所间隔的broker数目取决与1.nextReplicaShit的大小 2.该replica是该partition的第几个replica

最后的shif在不考虑shift超出了broker数目的情况下t为1+nextReplicaShift+replicaindex

至于为什么要这么做,我认为是尽量将备份放在间隔远的机器上.来提高容灾备份的能力吧.