【让数据排队的kafka | 01】Kafka集群单点故障？3步教你如何不重启服务解决

🗺️博客地图

📍前言

📍一、查（查看kafka集群信息）

📍二、改（修改consumer_offset配置文件）

📍三、执（执行Json文件）

📍四、测（模拟故障，测试服务）

---

前言

kafka从0.11版本开始，__consumer_offsets这个topic是存放消费者偏移量的，但是该主题默认配置副本数量只有1，容易造成单点故障，我们可以动态修改（无需重启服务）副本因子，提高kafka的可靠性。

一、查（查看kafka集群信息）

若执行命令不成功，可以尝试手打命令

#需要进入kafka对应目录下，使用kakfa自带脚本命令执行操作

cd /usr/local/kafka/bin

或者直接执行

/usr/local/kafka/bin/kafka-topics.sh

#查看集群信息

/usr/local/kafka/bin/kafka-topics.sh --zookeeper 192.168.0.17:2181 --topic __consumer_offsets --describe

可以看到集群__consumer_offsets主题默认配置副本数量只有1个，那么要解决单点故障问题，需要将主题副本数做下调整：

二、改（修改consumer_offset配置文件）

#将json文件保存至linux中，并取名为consumer_offset.json

vim consumer_offset.json

{
    "version": 1, 
    "partitions": [
        {
            "topic": "__consumer_offsets", 
            "partition": 0, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 1, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 2, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 3, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 4, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 5, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 6, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 7, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 8, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 9, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 10, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 11, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 12, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 13, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 14, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 15, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 16, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 17, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 18, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 19, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 20, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 21, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 22, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 23, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 24, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 25, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 26, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 27, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 28, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 29, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 30, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 31, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 32, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 33, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 34, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 35, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 36, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 37, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 38, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 39, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 40, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 41, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 42, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 43, 
            "replicas": [1, 3, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 44, 
            "replicas": [2, 1, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 45, 
            "replicas": [2, 3, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 46, 
            "replicas": [3, 1, 2]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 47, 
            "replicas": [3, 2, 1]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 48, 
            "replicas": [1, 2, 3]
        },
        {
            "topic": "__consumer_offsets", 
            "partition": 49, 
            "replicas": [1, 3, 2]
        }
    ]
}

三、执（执行Json文件）

#执行Json文件

注：如果consumer_offset.json文件不在当前目录下，需要在命令中指定路径，例如：./kafka-reassign-partitions.sh --zookeeper 192.168.0.17:2181 --reassignment-json-file /root/consumer_offset.json --execute

/usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper 192.168.0.17:2181 --reassignment-json-file consumer_offset.json --execute

执行成功

#验证执行计划结果

/usr/local/kafka/bin/kafka-reassign-partitions.sh --zookeeper 192.168.0.17:2181 --reassignment-json-file consumer_offset.json --verify

执行成功

#再次查看集群信息副本数量

/usr/local/kafka/bin/kafka-topics.sh --zookeeper 192.168.0.17:2181 --topic __consumer_offsets --describe

可以看到副本数变为3个

四、测（模拟故障，测试服务）

关闭一台或者两台kafka服务，测试产品是否可用