Kafka节点服役和退役

一、服役新节点

1. 新节点配置

• 基础配置

vim /etc/sysconfig/network-scripts/ifcfg-ens33

BOOTPROTO=static
IPADDR=192.168.10.105
GATEWAY=192.168.10.2
DNS1=192.168.10.2

vim /etc/hostname

hadoop105

• 重启hadoop105
• 修改hadoop105中kafak的broker.id为3
• 删除hadoop105中kafka下的datas和logs
• 启动hadoop102、hadoop103、hadoop104上的kafka集群
• 单独启动hadoop105中的kafka

2. 执行负载均衡操作

• 创建一个要均衡的主题

vim topics-to-move.json

{
    "topics": [
        {
            "topic": "first"
        }
    ],
    "version": 1
}

• 生成一个负载均衡的计划

bin/kafka-reassign-partitions.sh \
--bootstrap-server hadoop102:9092 \
--topics-to-move-json-file topics-to-move.json \
--broker-list "0,1,2,3" \
--generate

• 创建副本存储计划（所有副本存储在broker0、broker1、broker2、broker3中）

vim increase-replication-factor.json

就是上一步生成的内容

{
    "version":1,
    "partitions":[
        {
            "topic":"first",
            "partition":0,
            "replic as":[2,3,0],
            "log_dirs":["any","any","any"]
        },
        {
            "topic":"first",
            "partition":1,
            "replicas":[3,0,1],
            "log_dirs":["any","any","any"]
        },
        {
            "topic":"first",
            "partition":2,
            "replicas":[0,1,2],
            "log_dirs":["any","any","any"]
        }
    ]
}

• 执行副本存储计划

bin/kafka-reassign-partitions.sh \
--bootstrap-server hadoop102:9092 \
--reassignment-json-file increase-replication-factor.json \
--execute

• 验证副本存储计划

bin/kafka-reassign-partitions.hs \
--bootstrap-server hadoop102:9092 \
--reassignment-json-file increase-replication-factor.json \
--verify

二、退役旧节点

1. 执行负载均衡操作

先按照退役一台节点，生成执行计划，然后按照服役时操作流程执行负载均衡

• 创建一个要均衡的主题

vim topics-to-move.json

{
    "topics": [
        {"topic": "first"}
    ],
    "version": 1
}

• 创建执行计划

bin/kafka-reassign-partitions.sh \
--bootstrap-server hadoop102:9092 \
--topics-to-move-json-file topics-to-move.json \
--broker-list "0,1,2" \
--generate

• 创建副本存储计划(所有副本存储在broker0、broker1、broker2中)

vim /increase-replication-factor.json

就是上一步生成的数据

{
    "version":1,
    "partitions":[
        {
            "topic":"first",
            "partition":0,
            "replic as":[2,0,1],
            "log_dirs":["any","any","any"]
        },
        {
            "topic":"first",
            "partition":1,
            "replicas":[0,1,2],
            "log_dirs":["any","any","any"]
        },
        {
            "topic":"first",
            "partition":2,
            "replicas":[1,2,0],
            "log_dirs":["any","any","any"]
        }
    ]
}

• 执行副本存储计划

bin/kafak-reassign-partitions.sh \
--bootstrap-server hadoop102:9092 \
--reassignment-json-file increase-replication-factor.json \
--execute

• 验证副本存储计划

bin/kafka-reassign-partitions.sh \
--bootstrap-server hadoop102:9092 \
--reassignment-json-file increase-replication-factor.json \
--verify

• 执行停止命令

# 在hadoop105上执行停止命令即可
bin/kafka-server-stop.sh