Kafka集群新增节点后数据重分配的步骤

kafka版本2.2.1

目录

新增节点的步骤

数据迁移原理

数据迁移工具介绍

示例：

创建要迁移的topic配置文件

生成重新分配计划

执行数据迁移

检查重新分配的分区状态

---

新增节点的步骤

将其他节点的server.properties配置文件拷贝后修改以下参数

broker.id
log.dirs
zookeeper.connect

数据迁移原理

1. 只有新增的Topic才会将数据分布在新节点上，如果要将现有数据也分配到新节点，需要将Topic中的数据迁移到新节点上。
2. 数据迁移过程是手动启动的，但是是完全自动化的。Kafka会将新节点添加为要迁移的分区的追随者，并允许其完全复制该分区中的现有数据。新节点完全复制此分区的内容并加入同步副本后，现有副本之一将删除其分区的数据。

数据迁移工具介绍

分区重新分配工具可用于在代理之间移动分区。理想的分区分配将确保所有代理之间的数据负载和分区大小均匀。分区重新分配工具没有能力自动研究Kafka群集中的数据分布，并四处移动分区以实现均匀的负载分布。因此，必须弄清楚应该移动哪些主题或分区。

分区重新分配工具可以在3种模式下运行：

• --generate：在此模式下，给定主题列表和代理列表，该工具会生成分区与副本重新分配的计划，以将指定主题的所有分区在所有节点上重新分配。在给定主题和目标代理的列表的情况下，此选项仅提供了一种方便的方式来生成分区重新分配计划。
• --execute：在此模式下，该工具将根据用户提供的重新分配计划启动分区的重新分配。（使用--reassignment-json-file选项）。这可以是管理员手工制作的自定义重新分配计划，也可以使用--generate选项提供
• --verify：在此模式下，该工具会验证上一次--execute期间列出的所有分区的重新分配状态。状态可以是成功完成，失败或进行中

 

示例：

现有5个节点的broker_id为1,2,3,4,5；新增节点broker_id为6

Topic：test 有6个分区,5个副本

 

创建要迁移的topic配置文件

topics-to-move.json

{"topics": [{"topic": "test"}],"version":1}

 

生成重新分配计划

kafka-reassign-partitions --bootstrap-server localhost:9092 --zookeeper zookeeper-001:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5,6" --generate

以上命令将会产生以下内容，将 Proposed partition reassignment configuration 下的内容保存为test-reassign.json文件

Current partition replica assignment

{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[5,4,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":5,"replicas":[5,2,3,4,1],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":1,"replicas":[1,5,2,3,4],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":4,"replicas":[4,5,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":3,"replicas":[3,4,5,1,2],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":2,"replicas":[2,1,3,4,5],"log_dirs":["any","any","any","any","any"]}]}



Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"test","partition":4,"replicas":[5,1,2,3,4],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":1,"replicas":[2,4,5,6,1],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":3,"replicas":[4,6,1,2,3],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":0,"replicas":[1,3,4,5,6],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":5,"replicas":[6,2,3,4,5],"log_dirs":["any","any","any","any","any"]},{"topic":"test","partition":2,"replicas":[3,5,6,1,2],"log_dirs":["any","any","any","any","any"]}]}

 

执行数据迁移

kafka-reassign-partitions --bootstrap-server localhost:9092 --zookeeper zookeeper-001:2181 --reassignment-json-file test-reassign.json --execute

 

检查重新分配的分区状态

kafka-reassign-partitions --bootstrap-server localhost:9092 --zookeeper zookeeper-001:2181 --reassignment-json-file test-reassign.json --verify