kafka集群扩容(Topic迁移)

文章新地址

文章开始前推荐使用 kafka-manager 工具，可以实时查看kafka同步状态，broker实时读写流量，topic分区情况等信息，提供重选最优leader等操作。

kafka的集群扩容实际上就是把 topic 的 partition 移动到新加的集群上。

生成 topic 移动 json文件有两种方式:

1. 通过 --topics-to-move-json-file 和 --broker-list 批量生成新的topic分区信息，然后根据该信息执行转移操作。

2. 手动写要移动的topic信息，更灵活，但是在大量 topic 和 partition 的情况下非常繁琐并且容易出错。

1. 启动新节点

将原节点上的 kafka 目录通过 scp 命令拷贝到新节点，只需要修改配置文件中的 broker_id 和 ip 地址，然后依次启动 kafka 服务。

2. 移动topic

移动 topicA , topicB

sudo vim topics-to-move.json

{"topics": [{"topic": "topicA"},
            {"topic": "topicB"}],
"version":1
}

将上述 topic 移动到 broker 3,4,5上，用 generate 命令生成 partition 分配 json 串。

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--topics-to-move-json-file topics-to-move.json --broker-list "3,4,5" --generate 

将看到如下：

Current partition replica assignment
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[0,2]},
{"topic":"topicA","partition":1,"replicas":[1,0]},
{"topic":"topicB","partition":1,"replicas":[0,1]},
{"topic":"topicA","partition":0,"replicas":[2,1]},
{"topic":"topicB","partition":0,"replicas":[2,0]},
{"topic":"topicB","partition":2,"replicas":[1,2]}]}

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

将 Proposed partition reassignment configuration 下方的数据写入 expand-cluster-reassignment.json

上文说的第二种方式其实就是手动写该文件，而不用命令生成的。

vim expand-cluster-reassignment.json

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

用 execute 命令开始正式执行，将会把上述2个topic按 expand-cluster-reassignment 中的设置进行分配，该过程可能会影响其它topic的读取（延时）

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --execute

执行状态查询

查询执行状态

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --verify

主要有几种状态

Reassignment of partition [topicA,4] is still in progress # 转移中
Reassignment of partition [topicB,2] completed successfully # 转移结束

如果你看到一大堆的 ERROR 信息：

ERROR: Assigned replicas (0,5,1,2,3,4) don't match the list of replicas for reassignment (5,3,4) for partition [topicA,3]
Reassignment of partition [topicA,3] failed

不要方，不要关闭节点，过会再看看就正常了。