Kafka 9：分区再均衡

最新推荐文章于 2022-03-12 22:59:44 发布

Fisher3652

最新推荐文章于 2022-03-12 22:59:44 发布

阅读量700

点赞数

分类专栏： Kafka 文章标签： kafka

本文链接：https://blog.csdn.net/qq_40977118/article/details/107435441

版权

Kafka 专栏收录该内容

13 篇文章 2 订阅

订阅专栏

在这里插入图片描述

1.再均衡1表示，新增一个消费者3；再均衡2表示，去掉一个消费者3；每次再均衡都会从数据库读取最新的偏移量开始消费

在这里插入图片描述

2.再均衡监听器

消费者在退出和进行分区再均衡之前,会做一些清理工作比如，提交偏移量、关闭文件句柄、数据库连接等。
在为消费者分配新分区或移除旧分区时,可以通过消费者 API 执行一些应用程序代码，在调用 subscribe()方法时传进去一个 ConsumerRebalancelistener 实例就可以了。
ConsumerRebalancelistener 有两个需要实现的方法。

public void onPartitionsRevoked( Collection< TopicPartition> partitions)方法会在再均衡开始之前和消费者停止读取消息之后被调用。如果在这里提交偏移量，下一个接管分区的消费者就知道该从哪里开始读取了
public void onPartitionsAssigned( Collection< TopicPartition> partitions)方法会在重新分配分区之后和消费者开始读取消息之前被调用。
具体使用，我们先创建一个 3 分区的主题rebalance-topic

3.从特定偏移量处开始记录

如果想从分区的起始位置开始读取消息,或者直接跳到分区的末尾开始读取消息,可以使 seekToBeginning(Collection tp)和 seekToEnd( Collectiontp)这两个方法。
如果需要从特定的偏移量处开始读取消息。使用Kafka 提供的查找特定偏移量的 API。它有很多用途,比如向后回退几个消息或者向前跳过几个消息(对时间比较敏感的应用程序在处理滞后的情况下希望能够向前跳过若干个消息)。在使用 Kafka 以外的系统来存储偏移量时,它将给我们带来更大的惊喜–让消息的业务处理和偏移量的提交变得一致
场景:应用程序从 Kaka 读取事件(可能是网站的用户点击事件流),对它们进行处理(可能是使用自动程序清理点击操作并添加会话信息), 然后把结果保存到数据库。假设我们真的不想丢失任何数据,也不想在数据库里多次保存相同的结果。
我们可能会，毎处理一条记录就提交一次偏移量。尽管如此,在记录被保存到数据库之后以及偏移量被提交之前,应用程序仍然有可能发生崩溃,导致重复处理数据,数据库里就会出现重复记录。
如果保存记录和偏移量可以在一个原子操作里完成,就可以避免出现上述情况。记录和偏移量要么都被成功提交,要么都不提交。如果记录是保存在数据库里而偏移量是提交到Kafka上,那么就无法实现原子操作不过,如果在同一个事务里把记录和偏移量都写到数据库里会怎样呢?那么我们就会知道记录和偏移量要么都成功提交,要么都没有,然后重新处理记录。
现在的问题是:如果偏移量是保存在数据库里而不是 Kafka 里,那么消费者在得到新分区时怎么知道该从哪里开始读取?这个时候可以使用 seek()方法。在消费者启动或分配到新分区时,可以使用 seck()方法查找保存在数据库里的偏移量。我们可以使用使用 Consumer Rebalancelistener 和 seek()方法确保我们是从数据库里保存的偏移量所指定的位置开始处理消息的。

4.添加常量rebalance-topic

在这里插入图片描述

package org.example.config;

public class BusiConst {
    public static final String HELLO_TOPIC  = "hello-kafka";
    public static final String CONCURRENT_TOPIC  = "concurrent-test";
    public static final int CONCURRENT_PARTITION_COUNT = 2;
    public static final String SELF_PARTITION  = "self-partition";
    public static final String CONSUMER_COMMIT_TOPIC = "simple";
    public static final String REBALANCE_TOPIC = "rebalance-topic";
}

5.创建生产者RebalanceProducer，生产50条消息

package org.example.rebalance;

import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;
import org.example.config.BusiConst;

import java.util.Properties;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 * 多线程下使用生产者
 **/
public class RebalanceProducer {
    private static final int MSG_SIZE = 50;
    private static ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
    private static CountDownLatch countDownLatch = new CountDownLatch(MSG_SIZE);

    private static class ProducerWorker implements Runnable {
        private ProducerRecord<String, String> record;
        private KafkaProducer<String, String> producer;

        public ProducerWorker(ProducerRecord<String, String> record, KafkaProducer<String, String> producer) {
            this.record = record;
            this.producer = producer;
        }

        @Override
        public void run() {
            final String id = Thread.currentThread().getId() + "-" + System.identityHashCode(producer);
            try {
                producer.send(record, new Callback() {
                    @Override
                    public void onCompletion(RecordMetadata recordMetadata, Exception e) {
                        if (e != null) {
                            e.printStackTrace();
                        }
                        if (recordMetadata != null) {
                            System.out.println("偏移量：" + recordMetadata.offset() + ";分区:" + recordMetadata.partition());
                        }
                    }
                });
                System.out.printf(id + "数据[" + record + "]已发出");
                countDownLatch.countDown();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put("bootstrap.servers", "192.168.42.111:9092");
        properties.put("key.serializer", StringSerializer.class);
        properties.put("value.serializer", StringSerializer.class);
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);
        try {
            for (int i = 0; i < MSG_SIZE; i++) {
                ProducerRecord<String, String> record = new ProducerRecord<String, String>(BusiConst.REBALANCE_TOPIC, null,
                        System.currentTimeMillis(), String.valueOf(i), "Fisher" + i);
                executorService.submit(new ProducerWorker(record, producer));
                Thread.sleep(600);
            }
            countDownLatch.await();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            producer.close();
            executorService.shutdown();
        }
    }
}

6.创建再均衡监听器HandlerRebalance，实现ConsumerRebalanceListener接口

package org.example.rebalance;

import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;

import java.util.Collection;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
/**
 * 再均衡监听器
 **/
public class HandlerRebalance implements ConsumerRebalanceListener {
    //模拟一个保存分区偏移量的数据库表
    public final static ConcurrentHashMap<TopicPartition, Long> partitionOffsetMap = new ConcurrentHashMap<>();
    private final Map<TopicPartition, OffsetAndMetadata> currOffset;
    private final KafkaConsumer<String, String> consumer;

    public HandlerRebalance(Map<TopicPartition, OffsetAndMetadata> currOffset, KafkaConsumer<String, String> consumer) {
        this.currOffset = currOffset;
        this.consumer = consumer;
    }

    //分区在均衡之前
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> collection) {
        final String id = String.valueOf(Thread.currentThread().getId());
        System.out.println(id + "-Partition参数值为" + collection);
        System.out.println(id + "服务器准备分区再均衡，提交偏移量。当前偏移量为：" + currOffset);
        //我们可以不使用consumer.commitSync(currOffsets);
        //提交偏移量到kafka,由我们自己维护*/
        //开始事务
        //偏移量写入数据库
        System.out.println("分区偏移量表中：" + partitionOffsetMap);
        for (TopicPartition topicPartition : collection) {
            partitionOffsetMap.put(topicPartition, currOffset.get(topicPartition).offset());
        }
        consumer.commitSync(currOffset);
        //提交业务数和偏移量入库  tr.commit
    }

    //分区再均衡完成之后
    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> collection) {
        final String id = String.valueOf(Thread.currentThread().getId());
        System.out.println(id + "-再均衡完成，Partition参数为：" + collection);
        System.out.println("分区偏移量表中：" + partitionOffsetMap);
        for (TopicPartition topicPartition : collection) {
            System.out.println(id + "-topicPartition:" + topicPartition);
            //模拟从数据库中取得上次的偏移量
            Long offset = partitionOffsetMap.get(topicPartition);
            if (offset == null) continue;
            //从特定偏移量处开始记录（从指定分区中的指定偏移量开始消费）
            //这样就可以确保分区在均衡中的记录不错乱
            consumer.seek(topicPartition, partitionOffsetMap.get(topicPartition));
        }
    }
}

7.创建ConsumerWorker实现Runnable接口

package org.example.rebalance;

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.example.config.BusiConst;

import java.time.Duration;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;

public class ConsumerWork implements Runnable {
    private final KafkaConsumer<String, String> consumer;
    //todo 用来保存每个消费者当前读取分区的偏移量，解决分区再均衡的关键
    private final Map<TopicPartition, OffsetAndMetadata> currOffsets;
    private final boolean isStop;
    //todo 事务类可以送入（tr）
//    private final Transaction tr;事务类实例
    public ConsumerWork(boolean isStop) {
        //消费者配置
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.42.111:9092");
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, "rebalanceconsumer");
        //todo 取消自动提交
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        this.isStop = isStop;
        this.consumer = new KafkaConsumer<String, String>(properties);
        //todo 偏移量
        this.currOffsets = new HashMap<>();
        //todo 消费者订阅时加入再均衡监听器（HandlerRebalance）
        consumer.subscribe(Collections.singletonList(BusiConst.REBALANCE_TOPIC), new HandlerRebalance(currOffsets, consumer));
    }

    @Override
    public void run() {
        final String id = String.valueOf(Thread.currentThread().getId());
        int count = 0;
        TopicPartition topicPartition = null;
        long offset = 0;
        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
                //todo 业务处理
                //开始事务 tr.begin
                for (ConsumerRecord<String, String> record : records) {
                    System.out.println(id + "处理主题：" + record.topic() + ";分区：" + record.partition() +
                            ";偏移量：" + record.offset() + ";key:" + record.key() + ";value:" + record.value());
                    topicPartition = new TopicPartition(record.topic(), record.partition());
                    offset = record.offset() + 1;
                    //消费者消费时把偏移量提交到统一HashMap
                    currOffsets.put(topicPartition, new OffsetAndMetadata(offset, "no"));
                    count++;
                    //执行业务sql
                }
                //todo 同时将业务和偏移量入库
                if (currOffsets.size() > 0) {
                    for (TopicPartition topicPartitionkey : currOffsets.keySet()) {
                        HandlerRebalance.partitionOffsetMap.put(topicPartitionkey, currOffsets.get(topicPartitionkey).offset());
                    }
                }
                //todo 提交事务
                //todo 如果stop参数为true，这个消费者消费到第5个时自动关闭
                if (isStop && count >= 5) {
                    System.out.println(id + "-将关闭，当前偏移量为：" + currOffsets);
                    consumer.commitSync();
                    break;
                }
                consumer.commitSync();
            }
        } finally {
            consumer.close();
        }
    }
}

8.创建消费者RebalanceConsumer

package org.example.rebalance;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
 * 设置了再均衡监听器的消费者
 **/
public class RebalanceConsumer {
    //使用线程池，使用2个线程（2个消费者）
    private static ExecutorService executorService = Executors.newFixedThreadPool(2);

    public static void main(String[] args) throws InterruptedException {
        //todo 1.起2个消费者，消费3个分区
        for (int i = 0; i < 2; i++) {
            executorService.submit(new ConsumerWork(false));
        }
        Thread.sleep(500);
        //用来被停止观察消费情况
        //todo 2.再起一个消费者，消费5条数据之后关闭
        //todo 这里会触发2次分区再均衡，一次是新消费者加入，另一次是这个消费者消费5条数据之后离开
        new Thread(new ConsumerWork(true)).start();
    }
}

9.先启动消费者，再启动生产者

生产者打印

13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=0, value=Fisher0, timestamp=1595077712653)]已发出偏移量：0;分区:2
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=1, value=Fisher1, timestamp=1595077713259)]已发出偏移量：0;分区:0
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=2, value=Fisher2, timestamp=1595077713859)]已发出偏移量：1;分区:2
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=3, value=Fisher3, timestamp=1595077714460)]已发出偏移量：2;分区:2
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=4, value=Fisher4, timestamp=1595077715060)]已发出偏移量：0;分区:1
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=5, value=Fisher5, timestamp=1595077715661)]已发出偏移量：1;分区:0
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=6, value=Fisher6, timestamp=1595077716262)]已发出偏移量：1;分区:1
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=7, value=Fisher7, timestamp=1595077716862)]已发出偏移量：2;分区:0
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=8, value=Fisher8, timestamp=1595077717462)]已发出偏移量：3;分区:0
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=9, value=Fisher9, timestamp=1595077718062)]已发出偏移量：3;分区:2
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=10, value=Fisher10, timestamp=1595077718663)]已发出偏移量：2;分区:1
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=11, value=Fisher11, timestamp=1595077719263)]已发出偏移量：4;分区:0
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=12, value=Fisher12, timestamp=1595077719864)]已发出偏移量：3;分区:1
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=13, value=Fisher13, timestamp=1595077720464)]已发出偏移量：4;分区:1
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=14, value=Fisher14, timestamp=1595077721064)]已发出偏移量：5;分区:1
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=15, value=Fisher15, timestamp=1595077721665)]已发出偏移量：5;分区:0
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=16, value=Fisher16, timestamp=1595077722265)]已发出偏移量：4;分区:2
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=17, value=Fisher17, timestamp=1595077722865)]已发出偏移量：6;分区:0
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=18, value=Fisher18, timestamp=1595077723466)]已发出偏移量：6;分区:1
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=19, value=Fisher19, timestamp=1595077724066)]已发出偏移量：7;分区:1
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=20, value=Fisher20, timestamp=1595077724666)]已发出偏移量：8;分区:1
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=21, value=Fisher21, timestamp=1595077725267)]已发出偏移量：7;分区:0
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=22, value=Fisher22, timestamp=1595077725867)]已发出偏移量：8;分区:0
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=23, value=Fisher23, timestamp=1595077726468)]已发出偏移量：9;分区:0
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=24, value=Fisher24, timestamp=1595077727068)]已发出偏移量：9;分区:1
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=25, value=Fisher25, timestamp=1595077727669)]已发出偏移量：10;分区:0
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=26, value=Fisher26, timestamp=1595077728269)]已发出偏移量：10;分区:1
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=27, value=Fisher27, timestamp=1595077728869)]已发出偏移量：11;分区:0
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=28, value=Fisher28, timestamp=1595077729470)]已发出偏移量：12;分区:0
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=29, value=Fisher29, timestamp=1595077730071)]已发出偏移量：5;分区:2
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=30, value=Fisher30, timestamp=1595077730671)]已发出偏移量：13;分区:0
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=31, value=Fisher31, timestamp=1595077731272)]已发出偏移量：11;分区:1
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=32, value=Fisher32, timestamp=1595077731872)]已发出偏移量：6;分区:2
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=33, value=Fisher33, timestamp=1595077732472)]已发出偏移量：14;分区:0
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=34, value=Fisher34, timestamp=1595077733073)]已发出偏移量：15;分区:0
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=35, value=Fisher35, timestamp=1595077733673)]已发出偏移量：12;分区:1
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=36, value=Fisher36, timestamp=1595077734274)]已发出偏移量：7;分区:2
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=37, value=Fisher37, timestamp=1595077734874)]已发出偏移量：16;分区:0
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=38, value=Fisher38, timestamp=1595077735475)]已发出偏移量：13;分区:1
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=39, value=Fisher39, timestamp=1595077736075)]已发出偏移量：14;分区:1
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=40, value=Fisher40, timestamp=1595077736676)]已发出偏移量：8;分区:2
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=41, value=Fisher41, timestamp=1595077737277)]已发出偏移量：9;分区:2
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=42, value=Fisher42, timestamp=1595077737877)]已发出偏移量：15;分区:1
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=43, value=Fisher43, timestamp=1595077738477)]已发出偏移量：17;分区:0
15-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=44, value=Fisher44, timestamp=1595077739078)]已发出偏移量：18;分区:0
16-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=45, value=Fisher45, timestamp=1595077739679)]已发出偏移量：19;分区:0
17-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=46, value=Fisher46, timestamp=1595077740279)]已发出偏移量：16;分区:1
18-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=47, value=Fisher47, timestamp=1595077740880)]已发出偏移量：20;分区:0
13-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=48, value=Fisher48, timestamp=1595077741480)]已发出偏移量：21;分区:0
14-608597659数据[ProducerRecord(topic=rebalance-topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=49, value=Fisher49, timestamp=1595077742081)]已发出偏移量：10;分区:2

Process finished with exit code 0

10.查看消费者打印

12-Partition参数值为[]
12服务器准备分区再均衡，提交偏移量。当前偏移量为：{}
分区偏移量表中：{}
13-Partition参数值为[]
13服务器准备分区再均衡，提交偏移量。当前偏移量为：{}
分区偏移量表中：{}
16-Partition参数值为[]
16服务器准备分区再均衡，提交偏移量。当前偏移量为：{}
分区偏移量表中：{}
13-再均衡完成，Partition参数为：[rebalance-topic-1]
分区偏移量表中：{}
12-再均衡完成，Partition参数为：[rebalance-topic-0]
分区偏移量表中：{}
12-topicPartition:rebalance-topic-0
16-再均衡完成，Partition参数为：[rebalance-topic-2]
分区偏移量表中：{}
16-topicPartition:rebalance-topic-2
13-topicPartition:rebalance-topic-1
16处理主题：rebalance-topic;分区：2;偏移量：0;key:0;value:Fisher0
12处理主题：rebalance-topic;分区：0;偏移量：0;key:1;value:Fisher1
16处理主题：rebalance-topic;分区：2;偏移量：1;key:2;value:Fisher2
16处理主题：rebalance-topic;分区：2;偏移量：2;key:3;value:Fisher3
13处理主题：rebalance-topic;分区：1;偏移量：0;key:4;value:Fisher4
12处理主题：rebalance-topic;分区：0;偏移量：1;key:5;value:Fisher5
13处理主题：rebalance-topic;分区：1;偏移量：1;key:6;value:Fisher6
12处理主题：rebalance-topic;分区：0;偏移量：2;key:7;value:Fisher7
12处理主题：rebalance-topic;分区：0;偏移量：3;key:8;value:Fisher8
16处理主题：rebalance-topic;分区：2;偏移量：3;key:9;value:Fisher9
13处理主题：rebalance-topic;分区：1;偏移量：2;key:10;value:Fisher10
12处理主题：rebalance-topic;分区：0;偏移量：4;key:11;value:Fisher11
13处理主题：rebalance-topic;分区：1;偏移量：3;key:12;value:Fisher12
13处理主题：rebalance-topic;分区：1;偏移量：4;key:13;value:Fisher13
13处理主题：rebalance-topic;分区：1;偏移量：5;key:14;value:Fisher14
12处理主题：rebalance-topic;分区：0;偏移量：5;key:15;value:Fisher15
16处理主题：rebalance-topic;分区：2;偏移量：4;key:16;value:Fisher16
16-将关闭，当前偏移量为：{rebalance-topic-2=OffsetAndMetadata{offset=5, leaderEpoch=null, metadata='no'}}
12处理主题：rebalance-topic;分区：0;偏移量：6;key:17;value:Fisher17
13处理主题：rebalance-topic;分区：1;偏移量：6;key:18;value:Fisher18
13处理主题：rebalance-topic;分区：1;偏移量：7;key:19;value:Fisher19
13-Partition参数值为[rebalance-topic-1]
13服务器准备分区再均衡，提交偏移量。当前偏移量为：{rebalance-topic-1=OffsetAndMetadata{offset=8, leaderEpoch=null, metadata='no'}}
分区偏移量表中：{rebalance-topic-0=7, rebalance-topic-1=8, rebalance-topic-2=5}
12-Partition参数值为[rebalance-topic-0]
12服务器准备分区再均衡，提交偏移量。当前偏移量为：{rebalance-topic-0=OffsetAndMetadata{offset=7, leaderEpoch=null, metadata='no'}}
分区偏移量表中：{rebalance-topic-0=7, rebalance-topic-1=8, rebalance-topic-2=5}
13-再均衡完成，Partition参数为：[rebalance-topic-2]
分区偏移量表中：{rebalance-topic-0=7, rebalance-topic-1=8, rebalance-topic-2=5}
13-topicPartition:rebalance-topic-2
12-再均衡完成，Partition参数为：[rebalance-topic-0, rebalance-topic-1]
分区偏移量表中：{rebalance-topic-0=7, rebalance-topic-1=8, rebalance-topic-2=5}
12-topicPartition:rebalance-topic-0
12-topicPartition:rebalance-topic-1
12处理主题：rebalance-topic;分区：1;偏移量：8;key:20;value:Fisher20
12处理主题：rebalance-topic;分区：0;偏移量：7;key:21;value:Fisher21
12处理主题：rebalance-topic;分区：0;偏移量：8;key:22;value:Fisher22
12处理主题：rebalance-topic;分区：0;偏移量：9;key:23;value:Fisher23
12处理主题：rebalance-topic;分区：1;偏移量：9;key:24;value:Fisher24
12处理主题：rebalance-topic;分区：0;偏移量：10;key:25;value:Fisher25
12处理主题：rebalance-topic;分区：1;偏移量：10;key:26;value:Fisher26
12处理主题：rebalance-topic;分区：0;偏移量：11;key:27;value:Fisher27
12处理主题：rebalance-topic;分区：0;偏移量：12;key:28;value:Fisher28
13处理主题：rebalance-topic;分区：2;偏移量：5;key:29;value:Fisher29
12处理主题：rebalance-topic;分区：0;偏移量：13;key:30;value:Fisher30
12处理主题：rebalance-topic;分区：1;偏移量：11;key:31;value:Fisher31
13处理主题：rebalance-topic;分区：2;偏移量：6;key:32;value:Fisher32
12处理主题：rebalance-topic;分区：0;偏移量：14;key:33;value:Fisher33
12处理主题：rebalance-topic;分区：0;偏移量：15;key:34;value:Fisher34
12处理主题：rebalance-topic;分区：1;偏移量：12;key:35;value:Fisher35
13处理主题：rebalance-topic;分区：2;偏移量：7;key:36;value:Fisher36
12处理主题：rebalance-topic;分区：0;偏移量：16;key:37;value:Fisher37
12处理主题：rebalance-topic;分区：1;偏移量：13;key:38;value:Fisher38
12处理主题：rebalance-topic;分区：1;偏移量：14;key:39;value:Fisher39
13处理主题：rebalance-topic;分区：2;偏移量：8;key:40;value:Fisher40
13处理主题：rebalance-topic;分区：2;偏移量：9;key:41;value:Fisher41
12处理主题：rebalance-topic;分区：1;偏移量：15;key:42;value:Fisher42
12处理主题：rebalance-topic;分区：0;偏移量：17;key:43;value:Fisher43
12处理主题：rebalance-topic;分区：0;偏移量：18;key:44;value:Fisher44
12处理主题：rebalance-topic;分区：0;偏移量：19;key:45;value:Fisher45
12处理主题：rebalance-topic;分区：1;偏移量：16;key:46;value:Fisher46
12处理主题：rebalance-topic;分区：0;偏移量：20;key:47;value:Fisher47
12处理主题：rebalance-topic;分区：0;偏移量：21;key:48;value:Fisher48
13处理主题：rebalance-topic;分区：2;偏移量：10;key:49;value:Fisher49

消费者在没有启动生产者的时候，已经开始发生分区再均衡，结果为（消费1对应线程12与分区0，消费者2对应线程13与分区1，消费者3对应线程16与分区2）
生产者生产消息之后，消费者开始消费者，直到消费者3消费了5条消息停止之后，发生第2次分区再均衡，此时分区2的偏移量为5
此时消费者1和消费者2重新分配分区，分区0的偏移量为7，分区1的偏移量为8
再均衡完成，消费者1消费分区0和1，消费者2消费分区2
消费者1消费分区1从偏移量8开始消费，消费分区0从偏移量7开始消费；消费者2消费分区2从偏移量5开始消费

Fisher3652

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
2
评论
Kafka 9：分区再均衡

1.再均衡1表示，新增一个消费者3；再均衡2表示，去掉一个消费者3；每次再均衡都会从数据库读取最新的偏移量开始消费2.再均衡监听器消费者在退出和进行分区再均衡之前,会做一些清理工作比如，提交偏移量、关闭文件句柄、数据库连接等。在为消费者分配新分区或移除旧分区时,可以通过消费者 API 执行一些应用程序代码，在调用 subscribe()方法时传进去一个 ConsumerRebalancelistener 实例就可以了。ConsumerRebalancelistener 有两个需要实现的方法。
复制链接

扫一扫