Kafka消费者组与分区分配的实战解析

t0_54manong

于 2025-02-14 01:17:39 发布

阅读量544

点赞数 23

文章标签： kafka 分布式个人开发

本文链接：https://blog.csdn.net/m0_74337424/article/details/145623553

版权

在分布式消息系统中，Apache Kafka凭借其强大的可扩展性和容错能力，成为了众多开发者和企业的首选。其中，消费者组（Consumer Group）和分区（Partition）的机制是Kafka实现高吞吐量和负载均衡的关键。本文将通过实例代码，深入解析Kafka消费者组与分区分配的原理。
Kafka消费者组与分区分配的基本原理
在Kafka中，一个消费者组由多个消费者实例组成，用于实现系统的可扩展性和容错能力。如果所有消费者实例都属于同一个消费者组，那么消息记录将被有效地负载均衡分配到各个消费者实例上。当多个消费者订阅同一个主题（Topic）且属于同一个消费者组时，每个消费者将从不同的分区接收消息。
实例解析：分区在消费者组中的分配
为了更好地理解Kafka中主题分区是如何在消费者组中分配的，我们可以通过以下实例进行分析。
创建主题
首先，我们需要创建一个包含三个分区的主题。以下是使用Kafka Admin API创建主题的代码：
java复制
package com.logicbig.example;

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import java.util.Collections;
import java.util.Properties;
import java.util.stream.Collectors;

public class TopicCreator {
public static void main(String[] args) throws Exception {
createTopic(“example-topic”, 3);
}

private static void createTopic(String topicName, int numPartitions) throws Exception {
    Properties config = new Properties();
    config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, ExampleConfig.BROKERS);
    AdminClient admin = AdminClient.create(config);

    boolean alreadyExists = admin.listTopics().names().get().stream()
            .anyMatch(existingTopicName -> existingTopicName.equals(topicName));

    if (alreadyExists) {
        System.out.printf("topic already exists: %s%n", topicName);
    } else {
        System.out.printf("creating topic: %s%n", topicName);
        NewTopic newTopic = new NewTopic(topicName, numPartitions, (short) 1);
        admin.createTopics(Collections.singleton(newTopic)).all().get();
    }

    System.out.println("-- describing topic --");
    admin.describeTopics(Collections.singleton(topicName)).all().get()
            .forEach((topic, desc) -> {
                System.out.println("Topic: " + topic);
                System.out.printf("Partitions: %s, partition ids: %s%n", desc.partitions().size(),
                        desc.partitions().stream().map(p -> Integer.toString(p.partition()))
                                .collect(Collectors.joining(",")));
            });

    admin.close();
}

}
运行上述代码后，输出结果如下：
复制
creating topic: example-topic
– describing topic –
Topic: example-topic
Partitions: 3, partition ids: 0,1,2
发布与消费消息
接下来，我们将通过代码实现消息的发布和消费。以下是完整的代码示例：
java复制
package com.logicbig.example;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import java.time.Duration;
import java.util.Arrays;
import java.util.Collections;
import java.util.Properties;
import java.util.TreeSet;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

public class ConsumerGroupExample {
private final static int PARTITION_COUNT = 3;
private final static String TOPIC_NAME = “example-topic”;
private final static int MSG_COUNT = 4;
private static int totalMsgToSend;
private static AtomicInteger msg_received_counter = new AtomicInteger(0);

public static void run(int consumerCount, String[] consumerGroups) throws Exception {
    int distinctGroups = new TreeSet<>(Arrays.asList(consumerGroups)).size();
    totalMsgToSend = MSG_COUNT * PARTITION_COUNT * distinctGroups;

    ExecutorService executorService = Executors.newFixedThreadPool(consumerCount + 1);
    for (int i = 0; i < consumerCount; i++) {
        String consumerId = Integer.toString(i + 1);
        int finalI = i;
        executorService.execute(() -> startConsumer(consumerId, consumerGroups[finalI]));
    }

    executorService.execute(ConsumerGroupExample::sendMessages);
    executorService.shutdown();
    executorService.awaitTermination(10, TimeUnit.MINUTES);
}

private static void startConsumer(String consumerId, String consumerGroup) {
    System.out.printf("starting consumer: %s, group: %s%n", consumerId, consumerGroup);
    Properties consumerProps = ExampleConfig.getConsumerProps(consumerGroup);
    KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);
    consumer.subscribe(Collections.singleton(TOPIC_NAME));

    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(2));
        for (ConsumerRecord<String, String> record : records) {
            msg_received_counter.incrementAndGet();
            System.out.printf("consumer id:%s, partition id= %s, key = %s, value = %s, offset = %s%n",
                    consumerId, record.partition(), record.key(), record.value(), record.offset());
        }
        consumer.commitSync();
        if (msg_received_counter.get() == totalMsgToSend) {
            break;
        }
    }
}

private static void sendMessages() {
    Properties producerProps = ExampleConfig.getProducerProps();
    KafkaProducer producer = new KafkaProducer<>(producerProps);
    int key = 0;

    for (int i = 0; i < MSG_COUNT; i++) {
        for (int partitionId = 0; partitionId < PARTITION_COUNT; partitionId++) {
            String value = "message-" + i;
            key++;
            System.out.printf("Sending message topic: %s, key: %s, value: %s, partition id: %s%n",
                    TOPIC_NAME, key, value, partitionId);
            producer.send(new ProducerRecord<>(TOPIC_NAME, partitionId, Integer.toString(key), value));
        }
    }
}

}
不同消费者数量下的分区分配

启动3个消费者
当启动3个消费者且它们都属于同一个消费者组时，每个消费者将被分配到一个分区。以下是运行代码的输出结果：
复制
starting consumer: 1, group: test-consumer-group
starting consumer: 2, group: test-consumer-group
starting consumer: 3, group: test-consumer-group
Sending message topic: example-topic, key: 1, value: message-0, partition id: 0
…
consumer id:2, partition id= 2, key = 3, value = message-0, offset = 4
consumer id:1, partition id= 1, key = 2, value = message-0, offset = 4
consumer id:3, partition id= 0, key = 1, value = message-0, offset = 4
…
从输出结果可以看出，消费者1、2、3分别被分配到分区1、2和0。
启动2个消费者
当启动2个消费者时，一个消费者会被分配到一个分区，另一个消费者会被分配到两个分区。以下是运行代码的输出结果：
复制
starting consumer: 1, group: test-consumer-group
starting consumer: 2, group: test-consumer-group
…
consumer id:2, partition id= 2, key = 3, value = message-0, offset = 8
consumer id:2, partition id= 2, key = 6, value = message-1, offset = 9
consumer id:2, partition id= 2, key = 9, value = message-2, offset = 10
consumer id:2, partition id= 2, key = 12, value = message-3, offset = 11
consumer id:1, partition id= 0, key = 1, value = message-0, offset = 8
consumer id:1, partition id= 0, key = 4, value = message-1, offset = 9
consumer id:1, partition id= 0, key = 7, value = message-2, offset = 10
consumer id:1, partition id= 0, key = 10, value = message-3, offset = 11
consumer id:1, partition id= 1, key = 2, value = message-0, offset = 8
…
从输出结果可以看出，消费者1被分配到分区0和1，消费者2被分配到分区2。
启动1个消费者
当启动1个消费者时，该消费者将被分配到所有分区。以下是运行代码的输出结果：
复制
starting consumer: 1, group: test-consumer-group
…
consumer id:1, partition id= 2, key = 3, value = message-0, offset = 12
consumer id:1, partition id= 2, key = 6, value = message-1