Kafka修改分区、副本数量

一、概述

无需重启服务

二、修改分区

• 方式一：直接用软件重新分区

  Kafka Manager：

• 方式二：命令行分区

  –zookeeper已经过时，使用–bootstrap-server

  # 查看分区
  bin/kafka-topics.sh --describe --bootstrap-server 172.25.18.45:9092,172.25.18.45:9093,172.25.18.45:9094 --topic mc-topic
  
  # 修改分区
  bin/kafka-topics.sh --alter --bootstrap-server 172.25.18.45:9092,172.25.18.45:9093,172.25.18.45:9094 --topic mc-topic --partitions 3
  

三、修改副本

1.指定Topic

vi reassign_topic.json
{"topics":
  [{"topic":"__consumer_offsets"}],
  "version": 1
}

2.生成分配计划

–generate ： 生成分配计划，不执行

bin/kafka-reassign-partitions.sh --bootstrap-server 172.25.18.45:9092,172.25.18.45:9093,172.25.18.45:9094 --topics-to-move-json-file reassign_topic.json --broker-list "0,1,2" --generate
# 输出：
Current partition replica assignment # 当前分区配置
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[1,2,0]}]}
...
Proposed partition reassignment configuration # 建议分区配置
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[0,2,1]}]}
...

3.修改分配计划

复制建议配置，去掉,"log_dirs":["any","any","any"]

vi replcas.json

内容：

{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":1,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":2,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":3,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":4,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":5,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":6,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":7,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":8,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":9,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":10,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":11,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":12,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":13,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":14,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":15,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":16,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":17,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":18,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":19,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":20,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":21,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":22,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":23,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":24,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":25,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":26,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":27,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":28,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":29,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":30,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":31,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":32,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":33,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":34,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":35,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":36,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":37,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":38,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":39,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":40,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":41,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":42,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":43,"replicas":[0,2,1]},{"topic":"__consumer_offsets","partition":44,"replicas":[1,0,2]},{"topic":"__consumer_offsets","partition":45,"replicas":[2,0,1]},{"topic":"__consumer_offsets","partition":46,"replicas":[0,1,2]},{"topic":"__consumer_offsets","partition":47,"replicas":[1,2,0]},{"topic":"__consumer_offsets","partition":48,"replicas":[2,1,0]},{"topic":"__consumer_offsets","partition":49,"replicas":[0,2,1]}]}

4.执行分配计划

bin/kafka-reassign-partitions.sh --bootstrap-server 172.25.18.45:9092,172.25.18.45:9093,172.25.18.45:9094 --reassignment-json-file replcas.json --execute
# Successfully started partition reassignments for __consumer_offsets-0,...

5.查看分配进度

bin/kafka-reassign-partitions.sh --bootstrap-server 172.25.18.45:9092,172.25.18.45:9093,172.25.18.45:9094 --reassignment-json-file replcas.json --verify

6.查看新的副本

bin/kafka-topics.sh --bootstrap-server 172.25.18.45:9092,172.25.18.45:9093,172.25.18.45:9094 --topic __consumer_offsets --describe