rocketmq client-启动及消息获取流程分析

文章基于rocket-mq4.0 代码分析

主要分析消息拉取流程

Client端启动入口

以Push模式为例

org.apache.rocketmq.client.consumer.DefaultMQPushConsumer#start
-->org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#start

  ------------------------------------------------------ 

public void start() throws MQClientException {
        System.out.println(this.getSubscriptionInner());
        switch (this.serviceState) {
            case CREATE_JUST:
                
              //....省略代码....

                mQClientFactory.start();
                log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The PushConsumer service state not OK, maybe started once, "//
                    + this.serviceState//
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }

        this.updateTopicSubscribeInfoWhenSubscriptionChanged();

        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();

        this.mQClientFactory.rebalanceImmediately();
    }

主要是下面标注代码

mQClientFactory 是 MQClientInstance的实例，主要通过

this.pullMessageService = new PullMessageService(this);
this.rebalanceService = new RebalanceService(this);

两个service配合完成消息的拉取

PullMessageService 和 RebalanceService 都继承了 ServiceThread

PullMessageService 启动后会阻塞在

PullRequest pullRequest = this.pullRequestQueue.take();

获取PullRequest对象，再根据这个对象请求broker获得消息体

而PullRequest对象是RebalanceService启动后根据topic定时到broker查询该topic是否变更了（定时调 org.apache.rocketmq.client.impl.consumer.RebalanceImpl#doRebalance方法

），如果变更了才会生成该对象并放入到

org.apache.rocketmq.client.impl.consumer.PullMessageService#pullRequestQueue 中去

 

RebalanceService

在run方法启动后，通过一些列调用（类相互关系复杂）；以push模式为例

MQClientInstance.doRebalance()
-->DefaultMQPushConsumerImpl.doRebalance()
-->RebalanceImpl.doRebalance(final boolean isOrder)
-->RebalanceImpl.rebalanceByTopic(final String topic, final boolean isOrder)
最后更消息模式[BROADCASTING,CLUSTERING]去获取是否改拉取消息了

以 CLUSTERING 为例

case CLUSTERING: {

//mqSet 为该topic在broker端逻辑队列comsumequeue集合的映射，该值会有单独线程定时更新；
//例如：org.apache.rocketmq.client.impl.factory.MQClientInstance#startScheduledTask，还有其他地方也会更新该值

                Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
                List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);

                //省略代码.....

                if (mqSet != null && cidAll != null) {
                    List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
                    mqAll.addAll(mqSet);

                    Collections.sort(mqAll);
                    Collections.sort(cidAll);

                    //通过消费端负载均衡策略，client获得自己的MessageQueue

                    AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
                    List<MessageQueue> allocateResult = null;
                    try {
                        allocateResult = strategy.allocate(
                            this.consumerGroup,
                            this.mQClientFactory.getClientId(),
                            mqAll,
                            cidAll);
                    } catch (Throwable e) {
                        log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
                            e);
                        return;
                    }

                    Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
                    if (allocateResult != null) {
                        allocateResultSet.addAll(allocateResult);
                    }

                    //判断改topic是否有新消息可以拉取了                    

                    boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
                    if (changed) {
                       
                        //省略代码.....

                        //组装PullMessageService拉取消息所需的PullRequest对象并放入队列中
                           
                        this.messageQueueChanged(topic, mqSet, allocateResultSet);
                    }
                }
                break;
            }

 

 

PullMessageService

PullMessageService 获取到 PullRequest后最终执行方法到：

org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl#pullMessage

然后是一个很长的 PullCallback，会在拉取消息成功后调用 ConsumeMessageService执行（说实话，这代码不是一般的绕）

只有结果为  pullResult.getPullStatus()为FOUND才会执行 submitConsumeRequest 请求（用户自定义注册监听执行逻辑），其他的貌似都会放入重试

最终是我们注册的监听执行了自定义逻辑（DefaultMQPushConsumer 即push模式下我们需要实现  MessageListenerOrderly 或者 MessageListenerConcurrently  ）

 

消费进度更新

ConsumeMessageService   目前框架提供了两种实现，

ConsumeMessageOrderlyService 和 ConsumeMessageConcurrentlyService，框架会根据我们实现的自定义监听类型注入对应的实现类；

我理解的ConsumeMessageConcurrentlyService 和 ConsumeMessageOrderlyService关键区别在于

其内部类ConsumeRequest在执行run方法时后者通过加锁的方式，旨在保证获得锁的时候框架才处理对应的消息；

 

如果从comsumequeue里获取到了消息，

会执行用户自定义监听：

 

这里会根据自定义程序返回的status做一个处理，如果监听程序返回的是null，则会被赋值成 ：ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT

 

continueConsume = ConsumeMessageOrderlyService.this.processConsumeResult(msgs, status, context, this);

 这个方法会根据status的值做不同的逻辑处理，如果是  SUSPEND_CURRENT_QUEUE_A_MOMENT，则会在校验是否可以重复消费后，另外提交一个线程池任务处理该请求

是否可重复消费校验：

private boolean checkReconsumeTimes(List<MessageExt> msgs) {
        boolean suspend = false;
        if (msgs != null && !msgs.isEmpty()) {
            for (MessageExt msg : msgs) {

//********这里的 getMaxReconsumeTimes() 默认是 Integer.MAX_VALUE

                if (msg.getReconsumeTimes() >= getMaxReconsumeTimes()) {
                    MessageAccessor.setReconsumeTime(msg, String.valueOf(msg.getReconsumeTimes()));

//*******如果客户端重试次数大于了设置的次数，则构建 RETRY 消息重新发送至broker端

                    if (!sendMessageBack(msg)) {
                        suspend = true;
                        msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                    }
                } else {
                    suspend = true;
                    msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
                }
            }
        }
        return suspend;
    }

提交线程池新任务：

private void submitConsumeRequestLater(
        final ProcessQueue processQueue,
        final MessageQueue messageQueue,
        final long suspendTimeMillis
    ) {
        long timeMillis = suspendTimeMillis;
        if (timeMillis == -1) {
            timeMillis = this.defaultMQPushConsumer.getSuspendCurrentQueueTimeMillis();
        }

        if (timeMillis < 10) {
            timeMillis = 10;
        } else if (timeMillis > 30000) {
            timeMillis = 30000;
        }


//*****重新提交线程池处理*****

        this.scheduledExecutorService.schedule(new Runnable() {

            @Override
            public void run() {
                ConsumeMessageOrderlyService.this.submitConsumeRequest(null, processQueue, messageQueue, true);
            }
        }, timeMillis, TimeUnit.MILLISECONDS);
    }

 

 在执行完自定义逻辑后就会向broker发起更新消费offset的请求

 

****to continue

 

附客户端代码：

public class Consumer {

    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("ConsumerGroupNamecc4");
        consumer.setNamesrvAddr("127.0.0.1:9876");

        consumer.subscribe("topicaaa","TagA");

        consumer.registerMessageListener(new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                ConsumeConcurrentlyContext context) {
                System.out.printf(Thread.currentThread().getName() + " Receive1 New Messages: " + msgs + "%n");
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();

        System.out.printf("Consumer Started.%n");
    }
}