Kafka3.x修改主题的副本数和负载节点数

Kafka3.x修改主题的副本数和负载节点数

修改分区数

移动至目录"/opt/module/kafka/bin"下，创建名称为"first"的主题，并指定分区为1，副本为2：

kafka-topics.sh --bootstrap-server hadoop101:9092 --create --partitions 1 --replication-factor 2 --topic first

修改分区数（注意：分区数只能增加，不能减少）：

kafka-topics.sh --bootstrap-server hadoop101:9092 --alter --topic first --partitions 3

修改副本数

现在，需要修改每个分区的副本数为3，在目录"/opt/module/kafka"目录下，创建配置文件"increase-replication-factor.json"。需确定修改的主题名称和对应分区副本所在的broker.id：

{
	"version":1,
	"partitions":[
		{"topic":"first","partition":0,"replicas":[0,1,2]},
		{"topic":"first","partition":1,"replicas":[1,2,0]},
		{"topic":"first","partition":2,"replicas":[2,0,1]}
	]
}

执行命令修改副本数：

bin/kafka-reassign-partitions.sh --bootstrap-server 192.168.37.101:9092  --reassignment-json-file increase-replication-factor.json --execute

验证副本存储计划：

bin/kafka-reassign-partitions.sh --bootstrap-server 192.168.37.101:9092  --reassignment-json-file increase-replication-factor.json --verify

打开PrettyZoo查看主题first，分区各分区情况，进一步确定节点信息。

/kafka/brokers/topics/first/partitions/*/state:

{“controller_epoch”:4,“leader”:1,“version”:1,“leader_epoch”:5,“isr”:[1,0,2]}

{“controller_epoch”:4,“leader”:2,“version”:1,“leader_epoch”:5,“isr”:[2,1,0]}

{“controller_epoch”:4,“leader”:2,“version”:1,“leader_epoch”:5,“isr”:[2,0,1]}

可以看到，副本个数已修改为3。

服役新节点

首先，克隆一台虚拟机hadoop104，过程可以参考博客：Linux虚拟机的克隆。

然后，修改/opt/module/kafka/config/server.properties的broker.id=3，添加hadoop104:2181至zookeeper.connect。

修改主机映射：

vim /etc/hosts

192.168.37.104 hadoop104

删除hadoop104中kafka/logs和kafka/datas目录下的内容：

rm -rf datas/* logs/*

重新启动hadoop101，hadoop102，hadoop103集群：

zk.sh start

kf.sh start

再单独启动hadoop105的Kafka服务，目录"/opt/module/kafka/bin"：

kafka-server-start.sh -daemon ../config/properties

此时，hadoop104还未负载主题，创建一个指明主题的"topics-to-move.json"：

{
	"topics": [
		{"topic": "first"}
	],
	"version": 1
}

生成负载均衡计划：

bin/kafka-reassign-partitions.sh --bootstrap-server hadoop101:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate

{“version”:1,“partitions”:[{“topic”:“first”,“partition”:0,“replicas”:[3,0,1],“log_dirs”:[“any”,“any”,“any”]},{“topic”:“first”,“partition”:1,“replicas”:[0,1,2],“log_dirs”:[“any”,“any”,“any”]},{“topic”:“first”,“partition”:2,“replicas”:[1,2,3],“log_dirs”:[“any”,“any”,“any”]}]}

创建副本存储计划"increase-replication-factor.json"，再kafka目录下：

vim increase-replication-factor.json

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[3,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,3],"log_dirs":["any","any","any"]}]}

执行副本存储计划：

bin/kafka-reassign-partitions.sh --bootstrap-server hadoop101:9092 --reassignment-json-file increase-replication-factor.json --execute

验证副本计划：

bin/kafka-reassign-partitions.sh --bootstrap-server hadoop101:9092 --reassignment-json-file increase-replication-factor.json --execute

或者：

bin/kafka-topics.sh --bootstrap-server hadoop101:9092 --topic first --describe

Topic: first TopicId: WiWtw7QmSP28Le4JCy8uKw PartitionCount: 3 ReplicationFactor: 3 Configs: segment.bytes=1073741824
Topic: first Partition: 0 Leader: 3 Replicas: 3,0,1 Isr: 1,0,3
Topic: first Partition: 1 Leader: 2 Replicas: 0,1,2 Isr: 2,0,1
Topic: first Partition: 2 Leader: 2 Replicas: 1,2,3 Isr: 2,1,3

可以看出，broker.id=3的节点参与了负载。

退役旧节点

创建执行计划：

bin/kafka-reassign-partitions.sh --bootstrap-server hadoop101:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2" --generate

{“version”:1,“partitions”:[{“topic”:“first”,“partition”:0,“replicas”:[2,0,1],“log_dirs”:[“any”,“any”,“any”]},{“topic”:“first”,“partition”:1,“replicas”:[0,1,2],“log_dirs”:[“any”,“any”,“any”]},{“topic”:“first”,“partition”:2,“replicas”:[1,2,0],“log_dirs”:[“any”,“any”,“any”]}]}

创建副本存储计划：

vim increase-replication-factor.json

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,0],"log_dirs":["any","any","any"]}]}

执行副本存储计划：

bin/kafka-reassign-partitions.sh --bootstrap-server hadoop101:9092 --reassignment-json-file increase-replication-factor.json --execute

查看主题为first的详细信息：

bin/kafka-topics.sh --bootstrap-server hadoop101:9092 --topic first --describe

Topic: first TopicId: WiWtw7QmSP28Le4JCy8uKw PartitionCount: 3 ReplicationFactor: 3 Configs: segment.bytes=1073741824
Topic: first Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 1,0,2
Topic: first Partition: 1 Leader: 2 Replicas: 0,1,2 Isr: 2,0,1
Topic: first Partition: 2 Leader: 2 Replicas: 1,2,0 Isr: 2,1,0

可以看出id为3的节点，即hadoop104已经退役。

停止hadoop105的kafka服务即可：

bin/kafka-server-stop.sh