RocketMQ消息消费一：消息拉取

概述

前面讲了消息在broker端的存储，下面讲消息在消费者端的消费。简单介绍几个消费相关的概念。

集群消费&广播消费

是消费者两种不同的消费模式，广播消费模式下，一个消息会被所有的机器消费。集群消费模式下，一个消息只会被同一个集群消费一次，这里的集群由ConsumerGroup相同的机器组成。

ConsumerGroup

消费者归属于一个ConsumerGroup，相同ConsumerGroup的消费者组成一个集群，一个消息只被同一个集群的机器消费一次，起到负载均衡的作用。队列的消费进度也按照ConsumerGroup进行管理，不同ConsumerGroup之间的消费进度不受影响。

消费offset

用来保存消费进度，集群模式消费进度保存在broker，广播模式保存在消费者本地。

消息消费方式

分为push和pull两种方式。pull为消费者向broker主动拉取消息，可以根据负载控制消费速度。push为broker将消息推送到消费者，可以做到消息实时推送。在RocketMQ中，push模式是通过pull模式实现的。这里以push模式的实现作为分析。

相关类

• DefaultMQPushConsumer 消费客户端
• DefaultMQPushConsumerImpl 默认push消费者实现
• RebalanceImpl 负载均衡
• MQClientInstance 客户端实例
• PullAPIWrapper 拉取消息api
• OffsetStore 消费offset存储

消费者启动

DefaultMQPushConsumer.start()为启动入口，调用DefaultMQPushConsumerImpl.start():

    public synchronized void start() throws MQClientException {
   
        switch (this.serviceState) {
   
            case CREATE_JUST: // 默认值为CREATE_JUST
                log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
                    this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
                this.serviceState = ServiceState.START_FAILED;
                // 检查相关配置
                this.checkConfig();
                // 复制订阅信息给RebalanceImpl负载均衡使用
                this.copySubscription();

                if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
   
                    this.defaultMQPushConsumer.changeInstanceNameToPID();
                }
                // 创建 MQClientInstance
                this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
                // 负载均衡设置
                this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
                this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
                this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
                this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

                this.pullAPIWrapper = new PullAPIWrapper(
                    mQClientFactory,
                    this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
                this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

                // OffsetStore c初始化，分为集群和广播模式
                if (this.defaultMQPushConsumer.getOffsetStore() != null) {
   
                    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
                } else {
   
                    switch (this.defaultMQPushConsumer.getMessageModel()) {
   
                        case BROADCASTING:
                            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        case CLUSTERING:
                            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        default:
                            break;
                    }
                    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
                }
                this.offsetStore.load();

                // consumeMessageService 消费消息服务初始化
                if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
   
                    this.consumeOrderly = true;
                    this.consumeMessageService =
                        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
                } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
   
                    this.consumeOrderly = false;
                    this.consumeMessageService =
                        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
                }
                this.consumeMessageService.start();
                // 注册消费者
                boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
                if (!registerOK) {
   
                    this.serviceState = ServiceState.CREATE_JUST;
                    this.consumeMessageService.shutdown();
                    throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
                        + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
                        null);
                }

                mQClientFactory.start();
                log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
                    + this.serviceState
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }
		
		// 更新订阅信息
        this.updateTopicSubscribeInfoWhenSubscriptionChanged();
        this.mQClientFactory.checkClientInBroker();
        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
        // 立即执行负载均衡
        this.mQClientFactory.rebalanceImmediately();
    }

到这里启动完成。

拉取消息

拉取消息的入口在哪里呢？上面启动的时候，在MQClientInstanced.start()中启动了PullMessageService。PullMessageService继承自熟悉的ServiceThread，来看run方法：

    public void run() {
   
        while (!this.isStopped()) {
   
            try {
   
                // 从LinkedBlockingQueue取出请求，没有请求则阻塞
                PullRequest pullRequest = this.pullRequestQueue.take();
                // 执行拉取操作
                this.pullMessage(pullRequest);
            } catch (InterruptedException ignored) {
   
            } catch (Exception e) {
   
                log.error("Pull Message Service Run Method exception", e);
            }
        }

        log.info(this.getServiceName() + " service end");
    }

PullRequest放入：

• executePullRequestImmediately 立即放入，在消费者启动和拉取完成后放入
• executePullRequestLater 拉取异常情况下延迟放入

PullRequest类介绍

public class PullRequest {
   
    // 消费组
    private String consumerGroup;
    // 消费队列信息
    private MessageQueue messageQueue;
    // 拉取到的消息存储
    private ProcessQueue processQueue;
    // 下次拉取offset
    private long nextOffset;
}

PullMessageService拉取消息，交由DefaultMQPushConsumerImpl.pullMessage

    private void pullMessage(final PullRequest pullRequest) {
   
        final MQConsumerInner consumer = this.mQClientFactory.selectConsumer(pullRequest.getConsumerGroup());
        if (consumer != null) {
   
        // 转成 DefaultMQPushConsumerImpl，这个方法只为DefaultMQPushConsumerImpl服务，pull模式后面分析
            DefaultMQPushConsumerImpl impl = (DefaultMQPushConsumerImpl) consumer;
            impl.pullMessage(pullRequest);
        } else {
   
            log.warn("No matched consumer for the PullRequest {}, drop it", pullRequest);
        }
    }

DefaultMQPushConsumerImpl.pullMessage 拉取逻辑，交由pullAPIWrapper.pullKernelImpl

    public void pullMessage(final PullRequest pullRequest) {
   
        // 获取 消息存储队列
        final ProcessQueue processQueue = pullRequest.getProcessQueue();
        if (processQueue.isDropped()) {
   
            log.info("the pull request[{}] is dropped.", pullRequest.toString());
            return;
        }
        // 更新拉取时间戳
        pullRequest.getProcessQueue().setLastPullTimestamp(System.currentTimeMillis