kafka 执行topic迁移到新建的分区(partition)

创建新的topic
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test1
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test2
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test3
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test4
查看所有topic列表
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
增加topic分区数(越来越大,没找到删除)
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic test --partitions 4

1.创建迁移topic.json文件
{
"topics": [
{ "topic": "test5"},
{ "topic": "test6"},
{ "topic": "test7"},
{ "topic": "test8"},
{ "topic": "test9"}
],
"version": 1
}
2.迁移命令

# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --topics-to-move-json-file topic.json --broker-list "2,3" --generate

[2018-04-02 16:50:49,031] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)

打印出来的是分配规则
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[3]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test8","partition":0,"replicas":[3]},{"topic":"test5","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[3]}]}

注释: Current partition replica assignment表示当前的消息存储状况 。
Proposed partition reassignment configuration表示迁移后的消息存储状况(复制生成topicExec.json文件,用于分区命令)


3.生成topicExec.json文件,把分配规则复制进去
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
4.执行命令,开始分区重新分配
# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -execute
执行结果:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[0]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[0]},{"topic":"test5","partition":0,"replicas":[3]},{"topic":"test8","partition":0,"replicas":[1]}]}

5.验证分配结果 completed successfully表示成功
kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -verify
执行结果:
Status of partition reassignment:
Reassignment of partition [test7,0] completed successfully
Reassignment of partition [test9,2] completed successfully
Reassignment of partition [test6,0] completed successfully
Reassignment of partition [test5,3] completed successfully
Reassignment of partition [test8,1] completed successfully





  • 1
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值