RocketMQ 消费者运行原理，Consumer 集群消费、广播消费

最新推荐文章于 2024-05-09 16:34:39 发布

小道仙97

最新推荐文章于 2024-05-09 16:34:39 发布

阅读量1.1k

点赞数

分类专栏：源码学习 # RocketMQ 文章标签： java-rocketmq rocketmq java consumer 消费者

本文链接：https://blog.csdn.net/tomwildboar/article/details/130786777

版权

源码学习同时被 2 个专栏收录

15 篇文章 0 订阅

订阅专栏

RocketMQ

9 篇文章 1 订阅

订阅专栏

B站 https://www.bilibili.com/video/BV1zm4y1872b
在线学习文档 https://d9bp4nr5ye.feishu.cn/wiki/wikcnqTsEVMD74nV9W6IsmVHuxe

上次我们整体的看了一下RocketMQ Consumer 的消费过程 RocketMQ之 Consumer，消费者消费原理解析，今天再来聚焦看一下 Consumer 是如何进行集群消费和广播消费的。

先来说结论

消费者注册的时候已经告知服务器自己是的消费模型（集群/广播）
消费者去拉取数据的时候由服务器判断是否可以拉到消息

再来看问题

消费者如何注册进去的呢？
第一个请求是如何来到的呢？
rebalanceService 是如何做负载的？
不同的消费者最终执行任务的线程池是不是一个？

一、开始

没看过上一篇的小伙伴先看看之前的，这篇是上一篇的进阶版 Consumer，消费者消费原理解析

每一个使用@RocketMQMessageListener 注解修饰的类，都会根据配置信息生成一个 DefaultRocketMQListenerContainer 消费开始就是从这个容器的 start() 方法开始。

二、消费者是如何注册到nameserver

结论：通过心跳注册并实时更新的

mQClientFactory.start() 中会开启一系列的定时任务，其中有一个就是定时任务

MQClientInstance.java

public void start() throws MQClientException {

    synchronized (this) {
        switch (this.serviceState) {
            case CREATE_JUST:
                // ... 
                
                // 开启定时任务
                this.startScheduledTask();
                
                // ...
                break;
            case START_FAILED:
                throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
            default:
                break;
        }
    }
}

这里面的定时任务很多，更新nameserver地址、更新topic、更新offset 等

private void startScheduledTask() {
   
    // ... 
    
    // 定时发送心跳更新消费者信息
    this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

        @Override
        public void run() {
            try {
                MQClientInstance.this.cleanOfflineBroker();
                MQClientInstance.this.sendHeartbeatToAllBrokerWithLock();
            } catch (Exception e) {
                log.error("ScheduledTask sendHeartbeatToAllBroker exception", e);
            }
        }
    }, 1000, this.clientConfig.getHeartbeatBrokerInterval(), TimeUnit.MILLISECONDS);

    // ...
}

简化版发送心跳操作

public void sendHeartbeatToAllBrokerWithLock() {
    this.sendHeartbeatToAllBroker();
}



private void sendHeartbeatToAllBroker() {
    // 获取发送心跳的信息
    final HeartbeatData heartbeatData = this.prepareHeartbeatData();
    

    if (!this.brokerAddrTable.isEmpty()) {
        // ... 
        // 发送心跳
        int version = this.mQClientAPIImpl.sendHearbeat(addr, heartbeatData, clientConfig.getMqClientApiTimeout());
        // ...
    }
}


private HeartbeatData prepareHeartbeatData() {
    HeartbeatData heartbeatData = new HeartbeatData();

    // clientID
    heartbeatData.setClientID(this.clientId);

   
    // 每个消费者都会被注册到 consumerTable 里面，它是一个 ConcurrentMap
    for (Map.Entry<String, MQConsumerInner> entry : this.consumerTable.entrySet()) {
        MQConsumerInner impl = entry.getValue();
        if (impl != null) {
            // 获取消费者的信息
            ConsumerData consumerData = new ConsumerData();
            consumerData.setGroupName(impl.groupName());
            consumerData.setConsumeType(impl.consumeType());
            consumerData.setMessageModel(impl.messageModel());
            consumerData.setConsumeFromWhere(impl.consumeFromWhere());
            consumerData.getSubscriptionDataSet().addAll(impl.subscriptions());
            consumerData.setUnitMode(impl.isUnitMode());

            heartbeatData.getConsumerDataSet().add(consumerData);
        }
    }

    // Producer
    for (Map.Entry<String/* group */, MQProducerInner> entry : this.producerTable.entrySet()) {
        MQProducerInner impl = entry.getValue();
        if (impl != null) {
            ProducerData producerData = new ProducerData();
            producerData.setGroupName(entry.getKey());

            heartbeatData.getProducerDataSet().add(producerData);
        }
    }

    return heartbeatData;
}

DefaultMQPushConsumerImpl.java

consumerTable 的数据来源

public synchronized void start() throws MQClientException {
    switch (this.serviceState) {
        case CREATE_JUST:
          
            // ... 
            // 注册消费者
            boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
        
            // ... 
            this.serviceState = ServiceState.RUNNING;
            break;
        case RUNNING:
        case START_FAILED:
        case SHUTDOWN_ALREADY:
            throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
                + this.serviceState
                + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                null);
        default:
            break;
    }
}

private final ConcurrentMap<String/* group */, MQConsumerInner> consumerTable = new ConcurrentHashMap<String, MQConsumerInner>();

public synchronized boolean registerConsumer(final String group, final MQConsumerInner consumer) {
    if (null == group || null == consumer) {
        return false;
    }
    
    // putIfAbsent 方法如果设置值成功就返回 true，如果key已经存在了就返回当前 key对应的值
    MQConsumerInner prev = this.consumerTable.putIfAbsent(group, consumer);
    if (prev != null) {
        log.warn("the consumer group[" + group + "] exist already.");
        return false;
    }

    return true;
}

三、第一个请求是如何来到的呢

上一篇文章我们提到：每一个消费者都会开始一个死循环，一直从队列取数据进行消费，我们也知道每次任务完成都会把当前的这个请求存入队列构成一个循环请求，这样就会有个问题：第一次请求是怎么来的呢，其实是来自负载均衡。

其实它是来自负载，它的负载就是：我们知道 topic是逻辑上面的分类，队列才是存储数据的实质，负载就是切换不同的队列去消费数据。（只有集群消费才有）

MQClientInstance.java

public void start() throws MQClientException {

    synchronized (this) {
        switch (this.serviceState) {
            case CREATE_JUST:
                // ...
                
                // 开启负载均衡
                this.rebalanceService.start();
                
                // ...
                break;
            case START_FAILED:
                throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
            default:
                break;
        }
    }
}

RebalanceService.java run()

这里开启了一个死循环，只要线程不停止就会一直执行

@Override
public void run() {
    log.info(this.getServiceName() + " service started");

    while (!this.isStopped()) {
        this.waitForRunning(waitInterval);
        this.mqClientFactory.doRebalance();
    }

    log.info(this.getServiceName() + " service end");
}

MQClientInstance.java

上面我们已经知道每一个使用注解的类都会被注册成一个 DefaultMQPushConsumerImpl implements MQConsumerInner

public void doRebalance() {
    for (Map.Entry<String, MQConsumerInner> entry : this.consumerTable.entrySet()) {
        MQConsumerInner impl = entry.getValue();
        if (impl != null) {
            try {
                impl.doRebalance();
            } catch (Throwable e) {
                log.error("doRebalance exception", e);
            }
        }
    }
}

DefaultMQPushConsumerImpl.java

@Override
public void doRebalance() {
    if (!this.pause) {
        this.rebalanceImpl.doRebalance(this.isConsumeOrderly());
    }
}

RebalanceImpl.java
public void doRebalance(final boolean isOrder) {
    Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
    if (subTable != null) {
        for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
            final String topic = entry.getKey();
            try {
                this.rebalanceByTopic(topic, isOrder);
            } catch (Throwable e) {
                if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                    log.warn("rebalanceByTopic Exception", e);
                }
            }
        }
    }

    this.truncateMessageQueueNotMyTopic();
}

这里我们可以看到，不管是集群消费，还是广播消费都会，获取当前 topic对应的队列信息，然后进行投递到队列中（集群消费的分配策略复杂一些，这里先注视掉，下面再做解释）

private void rebalanceByTopic(final String topic, final boolean isOrder) {
    switch (messageModel) {
        case BROADCASTING: {
            Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
            if (mqSet != null) {
                boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
                if (changed) {
                    this.messageQueueChanged(topic, mqSet, mqSet);
                    log.info("messageQueueChanged {} {} {} {}",
                        consumerGroup,
                        topic,
                        mqSet,
                        mqSet);
                }
            } else {
                log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
            }
            break;
        }
        case CLUSTERING: {
             Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
             // ...

            boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
            if (changed) {
                log.info(
                    "rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}",
                    strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
                    allocateResultSet.size(), allocateResultSet);
                this.messageQueueChanged(topic, mqSet, allocateResultSet);
            }
          
            break;
        }
        default:
            break;
    }
}

这个方法会把每一个队列（MessageQueue）都组装成一个请求（PullRequest）

private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
    final boolean isOrder) {
    boolean changed = false;

    // ...
    
    List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
    for (MessageQueue mq : mqSet) {
        if (!this.processQueueTable.containsKey(mq)) {
            if (isOrder && !this.lock(mq)) {
                log.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
                continue;
            }

            this.removeDirtyOffset(mq);
            ProcessQueue pq = new ProcessQueue();

            long nextOffset = -1L;
            try {
                nextOffset = this.computePullFromWhereWithException(mq);
            } catch (Exception e) {
                log.info("doRebalance, {}, compute offset failed, {}", consumerGroup, mq);
                continue;
            }

            if (nextOffset >= 0) {
                ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                if (pre != null) {
                    log.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
                } else {
                    log.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                    PullRequest pullRequest = new PullRequest();
                    pullRequest.setConsumerGroup(consumerGroup);
                    pullRequest.setNextOffset(nextOffset);
                    pullRequest.setMessageQueue(mq);
                    pullRequest.setProcessQueue(pq);
                    pullRequestList.add(pullRequest);
                    changed = true;
                }
            } else {
                log.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
            }
        }
    }

    this.dispatchPullRequest(pullRequestList);

    return changed;
}

调用这个方法把请求（PullRequest），添加到队列中去，由之前的死循环去拉取请求处理

@Override
public void dispatchPullRequest(List<PullRequest> pullRequestList) {
    for (PullRequest pullRequest : pullRequestList) {
        // 之前我们已经看过，这个 方法就会把任务添加到队列中去了
        this.defaultMQPushConsumerImpl.executePullRequestImmediately(pullRequest);
        log.info("doRebalance, {}, add a new pull request {}", consumerGroup, pullRequest);
    }
}

四、rebalanceService 是如何做负载的

首先一个 topic 的消息，会投递到多个消息队列（这里我们假设是A、B、C三个队列），所谓的负载就是消费者按照某种策略分别从三（N）个列队依次消费。

上面第一个请求中，我们已经知道负载的入口了，我们直接来看负载的核心方法
org.apache.rocketmq.client.impl.consumer.RebalanceImpl#rebalanceByTopic

private void rebalanceByTopic(final String topic, final boolean isOrder) {
    switch (messageModel) {
        case BROADCASTING: {
             // ... 广播负载
             
        }
        case CLUSTERING: {
            // ... 集群负载
        
        }
    }
}

广播消费

case BROADCASTING: {
    // 获取当前topic的队列信息
    Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
    if (mqSet != null) {
        boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
        if (changed) {
            this.messageQueueChanged(topic, mqSet, mqSet);
            log.info("messageQueueChanged {} {} {} {}",
                consumerGroup,
                topic,
                mqSet,
                mqSet);
        }
    } else {
        log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
    }
    break;
}

广播消费每个消费者是需要消费topic里面的每一个消息，所以就不存在什么负载了。

updateProcessQueueTableInRebalance

updateProcessQueueTableInRebalance 方法是用来对负载均衡结果进行处理的，这个方法是通用的。因为广播消费不存在负载均衡，所以直接处理结果。

private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet,
    final boolean isOrder) {
    boolean changed = false;
    // ... 
    
    List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
    for (MessageQueue mq : mqSet) {
        // 判断当前正在进行的队列中，没有此队列
        if (!this.processQueueTable.containsKey(mq)) {
            if (isOrder && !this.lock(mq)) {
                log.warn("doRebalance, {}, add a new mq failed, {}, because lock failed", consumerGroup, mq);
                continue;
            }
            
            // 删除异常的队列
            this.removeDirtyOffset(mq);
            ProcessQueue pq = new ProcessQueue();

            long nextOffset = -1L;
            try {
                nextOffset = this.computePullFromWhereWithException(mq);
            } catch (Exception e) {
                log.info("doRebalance, {}, compute offset failed, {}", consumerGroup, mq);
                continue;
            }

            if (nextOffset >= 0) {
                ProcessQueue pre = this.processQueueTable.putIfAbsent(mq, pq);
                if (pre != null) {
                    log.info("doRebalance, {}, mq already exists, {}", consumerGroup, mq);
                } else {
                    // 组装消息请求，之前说到会有一个死循环不停的从队列中获取请求信息
                    log.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                    PullRequest pullRequest = new PullRequest();
                    pullRequest.setConsumerGroup(consumerGroup);
                    pullRequest.setNextOffset(nextOffset);
                    pullRequest.setMessageQueue(mq);
                    pullRequest.setProcessQueue(pq);
                    pullRequestList.add(pullRequest);
                    changed = true;
                }
            } else {
                log.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
            }
        }
    }

    // 这个方法就是把这些个请求信息放到之前所说的队列中去
    this.dispatchPullRequest(pullRequestList);

    return changed;
}

集群消费

集群消费因为每个消息只保证给消费组中的某个消费者消费，所以才需要负载均衡。

集群消费的负载均衡就是拿到全部的队列，和当前topic、consumerGroup下的全部消费者，再按照某种策略进行分发。

case CLUSTERING: {
    // 获取当前topic下的全部队列
    Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
    
    // 获取当前topic和consumerGroup 下的全部消费者（会发起请求去服务端获取）
    List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
    
    // ... 省略参数校验
      
    if (mqSet != null && cidAll != null) {
        List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
        mqAll.addAll(mqSet);

        Collections.sort(mqAll);
        Collections.sort(cidAll);
        
        // 获取负载均衡的策略 默认是【AllocateMessageQueueAveragely】平均哈希队列算法
        AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
        List<MessageQueue> allocateResult = null;
        try {
            allocateResult = strategy.allocate(
                this.consumerGroup,
                this.mQClientFactory.getClientId(),
                mqAll,
                cidAll);
        } catch (Throwable e) {
            log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
                e);
            return;
        }

        Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
        if (allocateResult != null) {
            allocateResultSet.addAll(allocateResult);
        }
        
        // 把策略返回的结果进行实践处理【上面已经说过了】
        boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
        if (changed) {
            log.info(
                "rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}",
                strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
                allocateResultSet.size(), allocateResultSet);
            this.messageQueueChanged(topic, mqSet, allocateResultSet);
        }
    }
    break;
}

AllocateMessageQueueAveragely

想必你也和我一样好奇这个默认策略是怎么来的？？？

默认策略是从当前对象中拿到的，当前对象是 RebalanceImpl AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;

再往上看持有 RebalanceImpl 对象的是 DefaultMQPushConsumerImpl 对象

@Override
public void doRebalance() {
    if (!this.pause) {
        this.rebalanceImpl.doRebalance(this.isConsumeOrderly());
    }
}

所以我们需要找到在创建DefaultMQPushConsumerImpl 对象的时候，它的创建在最开始解析 @RocketMQMessageListener的时候，可以看看上篇文章，这里只是把最终结果展示出来

private void initRocketMQPushConsumer() throws MQClientException {
    // ...
    
    // 不管是下面哪种创建方式负载均衡策略的默认值都是 【AllocateMessageQueueAveragely】
    if (Objects.nonNull(rpcHook)) {
        consumer = new DefaultMQPushConsumer(consumerGroup, rpcHook, new AllocateMessageQueueAveragely(),
            enableMsgTrace, this.applicationContext.getEnvironment().
            resolveRequiredPlaceholders(this.rocketMQMessageListener.customizedTraceTopic()));
        consumer.setVipChannelEnabled(false);
    } else {
        log.debug("Access-key or secret-key not configure in " + this + ".");
        consumer = new DefaultMQPushConsumer(consumerGroup, enableMsgTrace,
            this.applicationContext.getEnvironment().
                resolveRequiredPlaceholders(this.rocketMQMessageListener.customizedTraceTopic()));
    }
    
    // ...
}

五、不同的消费者最终执行任务的线程池是不是一个

之所以产生这个疑问，是在想如果每个消费者都创建一个自己的线程池，那线程池不是很多么？（其实线程不多的话怎么做到高性能呢）

DefaultMQPushConsumerImpl#start
创建线程池的起始点是 start 方法，代码片段如下：

public synchronized void start() throws MQClientException {

    // ...
    
    if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
        this.consumeOrderly = true;
        this.consumeMessageService =
            new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
    } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
        this.consumeOrderly = false;
        this.consumeMessageService =
            new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
    }
    this.consumeMessageService.start();
    
    // ... 
}

并发消费走的是 ConsumeMessageConcurrentlyService

public ConsumeMessageConcurrentlyService(DefaultMQPushConsumerImpl defaultMQPushConsumerImpl,
    MessageListenerConcurrently messageListener) {
    // ...
    // 总是new创建的线程池
    this.consumeExecutor = new ThreadPoolExecutor(
        this.defaultMQPushConsumer.getConsumeThreadMin(),
        this.defaultMQPushConsumer.getConsumeThreadMax(),
        1000 * 60,
        TimeUnit.MILLISECONDS,
        this.consumeRequestQueue,
        new ThreadFactoryImpl(consumeThreadPrefix));

   // ...
}