kafka-优先副本的选举

kafka-优先副本的选举

《深入理解kafk 核心设计与实践》

分区使多副本 机制来提升可靠性，但只有leader副本对外提供读写 服务，而follower副本 只负责内部进行消息的同步

创建 topic-prefer , 3个分区，3个副本:

bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --create --replication-factor 3 --partitions 3  --topic topic-prefer
Created topic topic-prefer.
bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --topic topic-prefer --describe
Topic: topic-prefer     TopicId: -YsiKnHnTsu-8zfQW6-njg PartitionCount: 3       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: topic-prefer     Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 3,1,2
        Topic: topic-prefer     Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: topic-prefer     Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1

每个分区均匀地分布在三个 broker节点上

优先副本

某个时间， broker 只有两个节点了:

        Topic: topic-prefer     Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,3,1
bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --topic topic-prefer --describe
Topic: topic-prefer     TopicId: -YsiKnHnTsu-8zfQW6-njg PartitionCount: 3       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: topic-prefer     Partition: 0    Leader: 1       Replicas: 3,1,2 Isr: 1,2
        Topic: topic-prefer     Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2
        Topic: topic-prefer     Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,1

可以发现， 原来分区0的 leader 节点为3，现在变成了1，使得负载失衡

为了能够有效的治理负载失衡的情况， kafka 引入了 优先副本( preferred replica)机制

所谓优先副本指在AR集合列表中的第一个副本，如 partion_0 的 AR 集[3,1,2], 那么分区0 的优先副本为3

所谓的优先副本选举是指通过一定的方式促使优先副本选举为leader副本，以些来促使集群的负载均衡，这一行为也可以称为分区平衡

如何进行优先副本选举

• kafka 提供分区自动 平衡功能，broker 端对应的参数是auto.leader.rebalance.enable, 默认为true
• 通过 kafka-preferred-replica-elction.sh 手动选举

重启 broker3节点

执行重新选举:

bash-5.1# kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type preferred  --all-topic-partitions
bash-5.1# kafka-topics.sh --bootstrap-server localhost:9092 --topic topic-prefer --describe
Topic: topic-prefer     TopicId: -YsiKnHnTsu-8zfQW6-njg PartitionCount: 3       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: topic-prefer     Partition: 0    Leader: 3       Replicas: 3,1,2 Isr: 1,2,3
        Topic: topic-prefer     Partition: 1    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: topic-prefer     Partition: 2    Leader: 2       Replicas: 2,3,1 Isr: 2,1,3

可以发现已恢复正常

指定分区进行选举

{
  "partitions": [
    {
      "topic": "topic-prefer",
      "partition": 1
    },
    {
      "topic": "topic-prefer",
      "partition": 2
    }
  ]
}

保存为 prefer_election.json

bash-5.1# kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type preferred  --path-to-json-file prefer_election.json 
Valid replica already elected for partitions 
bash-5.1#