kafka 扩容之后迁移topic

最新推荐文章于 2024-06-26 09:15:00 发布

b10l07

最新推荐文章于 2024-06-26 09:15:00 发布

阅读量1k

点赞数

文章标签：大数据 json 开发工具

原文链接：https://my.oschina.net/remainsu/blog/1816010

版权

2019独角兽企业重金招聘Python工程师标准>>>

1. 准备移动

这里假设要移动的topic名称为：topicA、topicB

vim topics-to-move.json

{"topics": [{"topic": "topicA"},
            {"topic": "topicB"}],
"version":1
}

将上述 topic 移动到 broker 3,4,5上，用 generate 命令生成 partition 分配 json 串，如下：

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 --topics-to-move-json-file topics-to-move.json --broker-list "3,4,5" --generate

将看到如下：

Current partition replica assignment
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[0,2]},
{"topic":"topicA","partition":1,"replicas":[1,0]},
{"topic":"topicB","partition":1,"replicas":[0,1]},
{"topic":"topicA","partition":0,"replicas":[2,1]},
{"topic":"topicB","partition":0,"replicas":[2,0]},
{"topic":"topicB","partition":2,"replicas":[1,2]}]}

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

2. 执行移动

将 Proposed partition reassignment configuration 下方的数据写入 expand-cluster-reassignment.json

vim expand-cluster-reassignment.json

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

用 execute 命令开始正式执行，将会把上述2个topic按 expand-cluster-reassignment 中的设置进行分配，该过程可能会影响其它topic的读取（延时）

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --execute

3. 查看状态

查询执行状态

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --verify

主要有几种状态

Reassignment of partition [topicA,4] is still in progress # 转移中
Reassignment of partition [topicB,2] completed successfully # 转移结束

如果你看到如下方的ERROR信息，没关系，不要关闭节点，过会再看看就正常了

ERROR: Assigned replicas (0,5,1,2,3,4) don't match the list of replicas for reassignment (5,3,4) for partition [topicA,3]
Reassignment of partition [topicA,3] failed

转载于:https://my.oschina.net/remainsu/blog/1816010