kafka-分区管理

1.分区优先副本选举

当分区的leader分配不均衡时,例如3个broker,0,1,2,有一个topic有3个分区p0,p1,p2,当所有的分区leader都在一个broker上时,就是分区分配不均衡

./kafka-preferred-replica-election.sh --zookeeper 127.0.0.1:2181/kafka-plaintext-poc

结果太多,不展示

可以通过指定topic和分区的方式,对特定topic和分区进行优先副本选举

2.分区重分配

当有broker宕机或增减broker时,及时某些分区的副本数不够,或者负载不均衡,kafka不会主动补充副本或迁移副本,需要手动进行副本重分配

分为三个步骤

(1) 创建一个需要迁移的主题清单JSON格式

{
    "topics":[
        {
            "topic":"test-topic"
        }
    ],
    "version":1
}

(2)根据主题清单和broker节点清单生成重分配方案

./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181/kafka-plaintext-poc --generate --topics-to-move-json-file reassign.json --broker-list 0,2

 

Current partition replica assignment
{"version":1,"partitions":[{"topic":"jc-kafka1-topic7","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"jc-kafka1-topic7","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"jc-kafka1-topic7","partition":0,"replicas":[0],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"jc-kafka1-topic7","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"jc-kafka1-topic7","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"jc-kafka1-topic7","partition":0,"replicas":[2],"log_dirs":["any"]}]}

将下面的方案保存到project.json

(3)执行重分配动作

./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181/kafka-plaintext-poc --execute --reassignment-json-file project.json

或者自定义重分配方案,直接执行重分配动作

3.复制限流

通过kafka-config.sh和kafka-reassign-partitions.sh脚本可以限制follower和leader的复制速度

参数为follower.replication.throttled.rate和leader.replication.throttled.rate

当参数配置为broker配置则为broker级别的限流

当参数配置为topic配置则为topic级别的限流