RocketMQ消息拉取consumer端实现

1.consume实现

        在Apache RocketMQ中，消费者（Consumer）通过拉取（Pull）或推送（Push）模式来获取消息。但是在传统Push模式下可能会造成consumer端压力过大或处理不过来的场景，所以在RocketMQ统一采取手动Pull模式。

2.PullRequest初始化

        RocketMQ对每个Topic下的每个Queue维护了一个PullRequest，每个PullRequest只有在本次处理完毕后才能进行下一次使用，主要使用到了CallBack机制。初始化是在Rebalance中，由于在重平衡过程中Queue可能出现新增或删除的场景，所以需要在Rebalance中维护。每个Consumer在创建过程中会创建一个MQClientInstance(之前版本中同一个JVM里只会创建一个)，每个PullRequest中有processQueue表示结果处理的Queue，consumer消费的消息。MessageQueue表示需要处理的Queue

org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#start
删除无效代码
public synchronized void start() throws MQClientException {
        switch (this.serviceState) {
            case CREATE_JUST:
                
                this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
// 消息拉取的client
                if (this.pullAPIWrapper == null) {
                    this.pullAPIWrapper = new PullAPIWrapper(
                            mQClientFactory,
                            this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
   
// 消费消息的offset，默认Cluster模式，从Broker拉取
                if (this.defaultMQPushConsumer.getOffsetStore() != null) {
                    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
                } else {
                    switch (this.defaultMQPushConsumer.getMessageModel()) {
                        case BROADCASTING:
                            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        case CLUSTERING:
                            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        default:
                            break;
                    }
                   
        }
        this.updateTopicSubscribeInfoWhenSubscriptionChanged();
        this.mQClientFactory.checkClientInBroker();
// 启动后立马进行一次重平衡，运行过程中如果consumer新增减少都会触发重平衡，broker扩容topic队列也会触发
        if (this.mQClientFactory.sendHeartbeatToAllBrokerWithLock()) {
            this.mQClientFactory.rebalanceImmediately();
        }
    }

看一下重平衡做的事，重平衡是由org.apache.rocketmq.client.impl.consumer.RebalanceService线程执行，每20s或收到事件执行都会，事件位置org.apache.rocketmq.client.impl.ClientRemotingProcessor#notifyConsumerIdsChanged

org.apache.rocketmq.client.impl.consumer.RebalanceImpl 相关代码

    public boolean doRebalance(final boolean isOrder) {
        boolean balanced = true;
        // 获取订阅的topic列表
        Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
        if (subTable != null) {
            for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
                final String topic = entry.getKey();
                try {
                    // 判断是不是有需要client重平衡，如果不是说明需要broker帮助重平衡-- 默认不走这里
                    if (!clientRebalance(topic) && tryQueryAssignment(topic)) {
                        boolean result = this.getRebalanceResultFromBroker(topic, isOrder);
                        if (!result) {
                            balanced = false;
                        }
                    } else {

                        // 对每个topic做重平衡，计算出当前consumer分配的Queue
                        boolean result = this.rebalanceByTopic(topic, isOrder);
                        if (!result) {
                            balanced = false;
                        }
                    }
                } catch (Throwable e) {
                    if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        log.warn("rebalance Exception", e);
                        balanced = false;
                    }
                }
            }
        }
        // 清理不再属于自己的queue，可能有些Queue被分配到了其他consumer
        this.truncateMessageQueueNotMyTopic();

        return balanced;
    }
 

private boolean rebalanceByTopic(final String topic, final boolean isOrder) {
        boolean balanced = true;
        switch (messageModel) {
            case BROADCASTING:// 删除这个代码，默认不会使用广播模式
            case CLUSTERING: {
                // 获取topic对应的queue
                Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
                // 获取topic对应的consumer
                List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
                if (null == mqSet) {
                    if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        this.messageQueueChanged(topic, Collections.<MessageQueue>emptySet(), Collections.<MessageQueue>emptySet());
                        log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
                    }
                }

                if (null == cidAll) {
                    log.warn("doRebalance, {} {}, get consumer id list failed", consumerGroup, topic);
                }

                if (mqSet != null && cidAll != null) {
                    List<MessageQueue> mqAll = new ArrayList<>();
                    mqAll.addAll(mqSet);
                    // 为什么可以每个consumer自己计算，由于所有consumer使用相同的计算策略，保证所有consumer计算结果相同
                    Collections.sort(mqAll);
                    Collections.sort(cidAll);

                    AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;

                    List<MessageQueue> allocateResult = null;
                    try {
                        allocateResult = strategy.allocate(
                                this.consumerGroup,
                                this.mQClientFactory.getClientId(),
                                mqAll,
                                cidAll);
                    } catch (Throwable e) {
                        log.error("allocate message queue exception. strategy name: {}, ex: {}", strategy.getName(), e);
                        return false;
                    }

                    Set<MessageQueue> allocateResultSet = new HashSet<>();
                    if (allocateResult != null) {
                        allocateResultSet.addAll(allocateResult);
                    }
                    // 依据结果构建ProcessQueue，每个Queue都会持有一个PullRequest
                    boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
                    if (changed) {
                        log.info(
                                "client rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}",
                                strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
                                allocateResultSet.size(), allocateResultSet);
                        this.messageQueueChanged(topic, mqSet, allocateResultSet);
                    }

                    balanced = allocateResultSet.equals(getWorkingMessageQueue(topic));
                }
                break;
            }
            default:
                break;
        }

        return balanced;
    }

3.PullRequest重复使用

        RocketMQ是在Netty基础上实现，所以在这里使用了事件驱动机制，在从队列中获取消息后，调用Netty发送消息后，org.apache.rocketmq.remoting.netty.NettyRemotingAbstract本地维护一个responseTable。在获取结果后，从map中获取处理的Reponse，根据回调函数函数来执行逻辑。

class NettyClientHandler extends SimpleChannelInboundHandler<RemotingCommand> {

        @Override
        protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
    //处理逻辑
            processMessageReceived(ctx, msg);
        }
    }
}

   public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
        final int opaque = cmd.getOpaque();
        final ResponseFuture responseFuture = responseTable.get(opaque);
        if (responseFuture != null) {
            responseFuture.setResponseCommand(cmd);

            responseTable.remove(opaque);

            if (responseFuture.getInvokeCallback() != null) {
// 执行response的Callback。更具代码找到每个callback的位置很重要，在拉取消息过程中。callback位于PullMessageService中
                executeInvokeCallback(responseFuture);
            } else {
                responseFuture.putResponse(cmd);
                responseFuture.release();
            }
        } else {
            log.warn("receive response, cmd={}, but not matched any request, address={}", cmd, RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
        }
    }

下面我们看看真正执行拉取消息的逻辑

    @Override
    public void run() {
        while (!this.isStopped()) {
            try {
                // 从阻塞队列中获取请求，初始请求是在Rebalance中生成并放入队列中
                MessageRequest messageRequest = this.messageRequestQueue.take();
                if (messageRequest.getMessageRequestMode() == MessageRequestMode.POP) {
                    this.popMessage((PopRequest) messageRequest);
                } else {
                    this.pullMessage((PullRequest) messageRequest);
                }
            } catch (Exception e) {
                logger.error("Pull Message Service Run Method exception", e);
            }
        }
    }


// 删除了无关代码

   public void pullMessage(final PullRequest pullRequest) {
        // 获取pullRequest对应的processor
        final ProcessQueue processQueue = pullRequest.getProcessQueue();
        if (processQueue.isDropped()) {
            log.info("the pull request[{}] is dropped.", pullRequest.toString());
            return;
        }

        pullRequest.getProcessQueue().setLastPullTimestamp(System.currentTimeMillis());

        final MessageQueue messageQueue = pullRequest.getMessageQueue();
        final SubscriptionData subscriptionData = this.rebalanceImpl.getSubscriptionInner().get(messageQueue.getTopic());
        if (null == subscriptionData) {
            this.executePullRequestLater(pullRequest, pullTimeDelayMillsWhenException);
            log.warn("find the consumer's subscription failed, {}", pullRequest);
            return;
        }

        final long beginTimestamp = System.currentTimeMillis();

        PullCallback pullCallback = new PullCallback() {

            @Override
            public void onSuccess(PullResult pullResult) {
                if (pullResult != null) {
                    pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
                            subscriptionData);

                    switch (pullResult.getPullStatus()) {
                        case FOUND: //找到消息，需要进行处理
                            long prevRequestOffset = pullRequest.getNextOffset();
                            pullRequest.setNextOffset(pullResult.getNextBeginOffset());
                            long pullRT = System.currentTimeMillis() - beginTimestamp;
                            DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullRT(pullRequest.getConsumerGroup(),
                                    pullRequest.getMessageQueue().getTopic(), pullRT);

                            long firstMsgOffset = Long.MAX_VALUE;
                            if (pullResult.getMsgFoundList() == null || pullResult.getMsgFoundList().isEmpty()) {
                                // 如果消息列表为空，则直接拉取下一次消息
                                DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                            } else {
                                // 获取第一条消息的offset
                                firstMsgOffset = pullResult.getMsgFoundList().get(0).getQueueOffset();

                                DefaultMQPushConsumerImpl.this.getConsumerStatsManager().incPullTPS(pullRequest.getConsumerGroup(),
                                        pullRequest.getMessageQueue().getTopic(), pullResult.getMsgFoundList().size());
                                // 将消息列表放入processQueue
                                boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
                                // 提交给对应的listener处理，用ConsumeMessageConcurrentlyService举例
                                DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                        pullResult.getMsgFoundList(),
                                        processQueue,
                                        pullRequest.getMessageQueue(),
                                        dispatchToConsume);

                                if (DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval() > 0) {
                                    DefaultMQPushConsumerImpl.this.executePullRequestLater(pullRequest,
                                            DefaultMQPushConsumerImpl.this.defaultMQPushConsumer.getPullInterval());
                                } else {
                                    DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                                }
                            }

                            if (pullResult.getNextBeginOffset() < prevRequestOffset
                                    || firstMsgOffset < prevRequestOffset) {
                                log.warn(
                                        "[BUG] pull message result maybe data wrong, nextBeginOffset: {} firstMsgOffset: {} prevRequestOffset: {}",
                                        pullResult.getNextBeginOffset(),
                                        firstMsgOffset,
                                        prevRequestOffset);
                            }

                            break;
                        case NO_NEW_MSG:
                        case NO_MATCHED_MSG: // 没有新消息，或没有匹配的消息 可能被tag过滤掉了 ，这里需要将offset进行移动
                            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                            DefaultMQPushConsumerImpl.this.correctTagsOffset(pullRequest);
                            DefaultMQPushConsumerImpl.this.executePullRequestImmediately(pullRequest);
                            break;
                        case OFFSET_ILLEGAL: // offset非法，说明系统有问题，将PullRequest废弃，执行重平衡
                            log.warn("the pull request offset illegal, {} {}",
                                    pullRequest.toString(), pullResult.toString());
                            pullRequest.setNextOffset(pullResult.getNextBeginOffset());

                            pullRequest.getProcessQueue().setDropped(true);
                            DefaultMQPushConsumerImpl.this.executeTask(new Runnable() {

                                @Override
                                public void run() {
                                    try {
                                        DefaultMQPushConsumerImpl.this.offsetStore.updateAndFreezeOffset(pullRequest.getMessageQueue(),
                                                pullRequest.getNextOffset());

                                        DefaultMQPushConsumerImpl.this.offsetStore.persist(pullRequest.getMessageQueue());

                                        // removeProcessQueue will also remove offset to cancel the frozen status.
                                        DefaultMQPushConsumerImpl.this.rebalanceImpl.removeProcessQueue(pullRequest.getMessageQueue());
                                        DefaultMQPushConsumerImpl.this.rebalanceImpl.getmQClientFactory().rebalanceImmediately();

                                        log.warn("fix the pull request offset, {}", pullRequest);
                                    } catch (Throwable e) {
                                        log.error("executeTaskLater Exception", e);
                                    }
                                }
                            });
                            break;
                        default:
                            break;
                    }
                }
            }

上面代码需要结合源码看，需要了解底层的设计以及如何执行的callback，以及netty底层基础技术，这块还是可以学习到很多东西