kafka集群扩容(Topic迁移)

文章新地址

文章开始前推荐使用 kafka-manager 工具,可以实时查看kafka同步状态,broker实时读写流量,topic分区情况等信息,提供重选最优leader等操作。

kafka的集群扩容实际上就是把 topic 的 partition 移动到新加的集群上。

生成 topic 移动 json文件有两种方式:

  1. 通过 --topics-to-move-json-file--broker-list 批量生成新的topic分区信息,然后根据该信息执行转移操作。

  2. 手动写要移动的topic信息,更灵活,但是在大量 topic 和 partition 的情况下非常繁琐并且容易出错。

1. 启动新节点

将原节点上的 kafka 目录通过 scp 命令拷贝到新节点,只需要修改配置文件中的 broker_id 和 ip 地址,然后依次启动 kafka 服务。

2. 移动topic

移动 topicA , topicB

sudo vim topics-to-move.json

{"topics": [{"topic": "topicA"},
            {"topic": "topicB"}],
"version":1
}

将上述 topic 移动到 broker 3,4,5上,用 generate 命令生成 partition 分配 json 串。

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--topics-to-move-json-file topics-to-move.json --broker-list "3,4,5" --generate 

将看到如下:

Current partition replica assignment
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[0,2]},
{"topic":"topicA","partition":1,"replicas":[1,0]},
{"topic":"topicB","partition":1,"replicas":[0,1]},
{"topic":"topicA","partition":0,"replicas":[2,1]},
{"topic":"topicB","partition":0,"replicas":[2,0]},
{"topic":"topicB","partition":2,"replicas":[1,2]}]}

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

Proposed partition reassignment configuration 下方的数据写入 expand-cluster-reassignment.json

上文说的第二种方式其实就是手动写该文件,而不用命令生成的。

vim expand-cluster-reassignment.json

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

execute 命令开始正式执行,将会把上述2个topic按 expand-cluster-reassignment 中的设置进行分配,该过程可能会影响其它topic的读取(延时)

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --execute

执行状态查询

查询执行状态

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --verify

主要有几种状态

Reassignment of partition [topicA,4] is still in progress # 转移中
Reassignment of partition [topicB,2] completed successfully # 转移结束

如果你看到一大堆的 ERROR 信息:

ERROR: Assigned replicas (0,5,1,2,3,4) don't match the list of replicas for reassignment (5,3,4) for partition [topicA,3]
Reassignment of partition [topicA,3] failed

不要方,不要关闭节点,过会再看看就正常了。

发布了129 篇原创文章 · 获赞 180 · 访问量 70万+
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: Age of Ai 设计师: meimeiellie

分享到微信朋友圈

×

扫一扫,手机浏览