创建新的topic
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test1
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test2
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test3
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test4
查看所有topic列表
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
增加topic分区数(越来越大,没找到删除)
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic test --partitions 4
1.创建迁移topic.json文件
{
"topics": [
{ "topic": "test5"},
{ "topic": "test6"},
{ "topic": "test7"},
{ "topic": "test8"},
{ "topic": "test9"}
],
"version": 1
}
2.迁移命令
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[3]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test8","partition":0,"replicas":[3]},{"topic":"test5","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[3]}]}
注释: Current partition replica assignment表示当前的消息存储状况 。
Proposed partition reassignment configuration表示迁移后的消息存储状况(复制生成topicExec.json文件,用于分区命令)。
3.生成topicExec.json文件,把分配规则复制进去
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
4.执行命令,开始分区重新分配
# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -execute
执行结果:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[0]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[0]},{"topic":"test5","partition":0,"replicas":[3]},{"topic":"test8","partition":0,"replicas":[1]}]}
5.验证分配结果 completed successfully表示成功
kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -verify
执行结果:
Status of partition reassignment:
Reassignment of partition [test7,0] completed successfully
Reassignment of partition [test9,2] completed successfully
Reassignment of partition [test6,0] completed successfully
Reassignment of partition [test5,3] completed successfully
Reassignment of partition [test8,1] completed successfully
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test1
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test2
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test3
kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test4
查看所有topic列表
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
增加topic分区数(越来越大,没找到删除)
./kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic test --partitions 4
1.创建迁移topic.json文件
{
"topics": [
{ "topic": "test5"},
{ "topic": "test6"},
{ "topic": "test7"},
{ "topic": "test8"},
{ "topic": "test9"}
],
"version": 1
}
2.迁移命令
# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --topics-to-move-json-file topic.json --broker-list "2,3" --generate
[2018-04-02 16:50:49,031] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
打印出来的是分配规则Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[3]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test8","partition":0,"replicas":[3]},{"topic":"test5","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[3]}]}
注释: Current partition replica assignment表示当前的消息存储状况 。
Proposed partition reassignment configuration表示迁移后的消息存储状况(复制生成topicExec.json文件,用于分区命令)。
3.生成topicExec.json文件,把分配规则复制进去
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
4.执行命令,开始分区重新分配
# ./kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -execute
执行结果:
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[1001]},{"topic":"test9","partition":0,"replicas":[1001]},{"topic":"test8","partition":0,"replicas":[1001]},{"topic":"test5","partition":0,"replicas":[1001]},{"topic":"test6","partition":0,"replicas":[1001]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"test7","partition":0,"replicas":[0]},{"topic":"test9","partition":0,"replicas":[2]},{"topic":"test6","partition":0,"replicas":[0]},{"topic":"test5","partition":0,"replicas":[3]},{"topic":"test8","partition":0,"replicas":[1]}]}
5.验证分配结果 completed successfully表示成功
kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file topicExec.json -verify
执行结果:
Status of partition reassignment:
Reassignment of partition [test7,0] completed successfully
Reassignment of partition [test9,2] completed successfully
Reassignment of partition [test6,0] completed successfully
Reassignment of partition [test5,3] completed successfully
Reassignment of partition [test8,1] completed successfully