rocketMQ-consumer源码_rocketmqconsumer源码-CSDN博客

本文链接：https://blog.csdn.net/ToBeABetterOne/article/details/89322798

RocketMQ消费模式包括集群和广播模式，本文详细剖析了消费者启动、消息拉取流程、负载均衡机制以及消息消费过程。启动时，消费者订阅信息初始化并启动消费服务线程。消息拉取涉及PullRequest和ProcessQueue，通过PullMessageService不断从队列中获取任务。拉取成功后回调处理消息，更新消费进度，并依据负载均衡策略重新分配消息队列。在消费过程中，消息通过ConsumeMessageService处理，支持并发和顺序消费，同时管理消费确认和定时消息功能。

摘要由CSDN通过智能技术生成

RocketMQ 服务器与消费之之前的消息传送方式分为拉模式和推模式，其中推模式是基于拉模式实现的，一个拉取任务完成之后立刻开始下一个拉取任务。

消费模式分为集群消费（clustering）和广播消费（broadcasting）。

集群模式，同一个主题下的同一条消息只能被一个集群中的一个消费者消费；
广播模式，同一个主题下的同一条消息要被集群中所有消费者消费。

启动

以DefaultMQPushConsumerImpl的start方法为例

获取订阅信息SubscriptionData，并放入RebalanceImp的订阅信息中
订阅信息来源于：

调用DefaultMQPushConsumerImpl。subscribe方法订阅
订阅重试主题：%RETRY%+消费组名，消费者在启动时会自动订阅这个主题

private void copySubscription() throws MQClientException {
   
    try {
   
        Map<String, String> sub = this.defaultMQPushConsumer.getSubscription();
        if (sub != null) {
   
            for (final Map.Entry<String, String> entry : sub.entrySet()) {
   
                final String topic = entry.getKey();
                final String subString = entry.getValue();
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    topic, subString);
                this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
            }
        }

        if (null == this.messageListenerInner) {
   
            this.messageListenerInner = this.defaultMQPushConsumer.getMessageListener();
        }

        switch (this.defaultMQPushConsumer.getMessageModel()) {
   
            case /BROADCASTING/:
                break;
            case /CLUSTERING/:
                final String retryTopic = MixAll./getRetryTopic/(this.defaultMQPushConsumer.getConsumerGroup());
                SubscriptionData subscriptionData = FilterAPI./buildSubscriptionData/(this.defaultMQPushConsumer.getConsumerGroup(),
                    retryTopic, SubscriptionData./SUB_ALL/);
                this.rebalanceImpl.getSubscriptionInner().put(retryTopic, subscriptionData);
                break;
            default:
                break;
        }
    } catch (Exception e) {
   
        throw new MQClientException(“subscription exception”, e);
    }
}

初始化MqClientInstance和rebalanceImpl

if (this.defaultMQPushConsumer.getMessageModel() == MessageModel./CLUSTERING/) {
   
    this.defaultMQPushConsumer.changeInstanceNameToPID();
}

this.mQClientFactory = MQClientManager./getInstance/().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);

this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);

this.pullAPIWrapper = new PullAPIWrapper(
    mQClientFactory,
    this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

初始化消息进度，如是集群消费，则消息进度存储在broker，如是广播消费则在consumer

if (this.defaultMQPushConsumer.getOffsetStore() != null) {
   
    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
   
    switch (this.defaultMQPushConsumer.getMessageModel()) {
   
        case /BROADCASTING/:
            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        case /CLUSTERING/:
            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
            break;
        default:
            break;
    }
    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();

根据是否是顺序消费启动ConsumeMessageOrderlyService或ConsumeMessageConcurrentlyService服务线程负责消息消费，内部维护一个线程池

if (this.getMessageListenerInner() instance MessageListenerOrderly) {
   
    this.consumeOrderly = true;
    this.consumeMessageService =
        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instance MessageListenerConcurrently) {
   
    this.consumeOrderly = false;
    this.consumeMessageService =
        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}

this.consumeMessageService.start();

向MQClientInstance注册消费者，并启动MQClientInstance。一个JVM中所有生产者和消费者公用一个MQClientInstance，MQClientInstance只会启动一次。

boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
   
    this.serviceState = ServiceState./CREATE_JUST/;
    this.consumeMessageService.shutdown();
    throw new MQClientException(“The consumer group[“ + this.defaultMQPushConsumer.getConsumerGroup()
        + “] has been created before, specify another name please.” + FAQUrl./suggestTodo/(FAQUrl./GROUP_NAME_DUPLICATE_URL/),
        null);
}

mQClientFactory.start();

消息拉取

在MQClientINstance启动的时候，启动了pullMessageService服务。pull消费模式不同，pull方法是客户端自己手动调用，不会在队列中放pullRequset对象
在它的run方法中可以看到

public void run() {
   
    log.info(this.getServiceName() + “ service started”);

    while (!this.isStopped()) {
   
        try {
   
            PullRequest pullRequest = this.pullRequestQueue.take();
            this.pullMessage(pullRequest);
        } catch (InterruptedException ignored) {
   
        } catch (Exception e) {
   
            log.error(“Pull Message Service Run Method exception”, e);
        }
    }

    log.info(this.getServiceName() + “ service end”);
}

只要服务没有停止，就不停的从pullRequestQueue中take出pullRequest任务。pullRequestQueue是一个阻塞队列

LinkedBlockingQueue<PullRequest> pullRequestQueue

pullRequest是每一个消息队列一个pullRequest，放进去的时机：

执行完一次拉取任务之后又将pullRequest放入到pullRequestQueue
真正创建是在RebalanceImpl中

PullRequest

private String consumerGroup; //消费组
private MessageQueue messageQueue; //待拉取消息队列
private ProcessQueue processQueue; //消息处理队列，从broker拉取的消息先存放在这里再提交到消费者消费线程池消费，是messageQueue在消费者的快照
private long nextOffset; //待拉取的messageQueue偏移量
private boolean lockedFirst = false; //是否被锁定

ProcessQueue

processQueueTable是map结构，key是MessageQueue，value是processQueue，Rebalance做负载均衡时会把consumer分配到的messageQueue和processTable做对比，将匹配上的processQueue构造一个PullRequest，获取下一个消息的offset设置到pullRequset中，并把pullrequest放入PullrequestQueue。

ProcessQueue是messageQueue的快照，保存了从broker拉取了但是还没有消费的消息。有一个读写锁控制并发，一个TreeMap保存拉取的消息

PullMessageService每次从消息服务器默认拉取32条消息，按照消息队列偏移量顺序存放在processQueue，然后PullMessageService将消息提交到消费者消费线程池，消费成功之后从processQueue中移除。

public final static long /REBALANCE_LOCK_MAX_LIVE_TIME/=
    Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockMaxLiveTime", "30000"));
public final static long /REBALANCE_LOCK_INTERVAL/= Long./parseLong/(System./getProperty/("rocketmq.client.rebalance.lockInterval", "20000"));
private final static long /PULL_MAX_IDLE_TIME/= Long./parseLong/(System./getProperty/("rocketmq.client.pull.pullMaxIdleTime", "120000"));
private final InternalLogger log = ClientLogger./getLog/();
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();//消息存储容器，键为消息在consumerQueue中的偏移量，value为消息实体
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
//**/
/ * A subset of msgTreeMap, will only be used when orderly consume/
/ *//
private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>();//消息临时存储容器，用于处理顺序消息，从processQueue中的msgTressMap中取出消息前，先临时存储在这里
private final AtomicLong tryUnlockTimes = new AtomicLong(0);
private volatile long queueOffsetMax = 0L;
private volatile boolean dropped = false;
private volatile long lastPullTimestamp = System./currentTimeMillis/();
private volatile long lastConsumeTimestamp = System./currentTimeMillis/();
private volatile boolean locked = false;
private volatile long lastLockTimestamp = System./currentTimeMillis/();
private volatile boolean consuming = false;
private volatile long msgAccCnt = 0;

消息拉取流程

consumer端

主要入口在DefaultMQPushConsumerImpl的pullMessage
方法。

检查processQueue的状态
判断是否触发流控，如果出发流量控制，则延迟50ms执行，流控的触发点有：
- processQueue当前处理的消息条数超过了pullThresholdForQueue = 1000
- processQueue当前处理的消息大小超过了pullThresholdSizeForQueue = 100M
- 顺序消费时，processQueue中队列最大偏移量与最小偏移量间距超过consumeConcurrentlyMaxSpan = 2000，主要担心一条消息堵塞，消息进度无法向前推进，从而可能造成大量消息重复消费
根据主题获取订阅信息,如果为空，则延迟3s执行
构建消息拉取系统标记PullSysFlag

private final static int /FLAG_COMMIT_OFFSET/= 0x1 << 0;//从内存中读取的消费进度大于0
private final static int /FLAG_SUSPEND/= 0x1 << 1;//消息拉取时支持挂起
private final static int /FLAG_SUBSCRIPTION/= 0x1 << 2;//消息过滤机制为表达式，这位置1
private final static int /FLAG_CLASS_FILTER/= 0x1 << 3;//消息过滤机制为类过滤模式，这位置1<