RocketMQ源码分析之pull模式consumer
DefaultMQPullConsumer
DefaultMQPullConsumer的实现及使用比较简单(已被废弃),从下面示例中可以看到消息的消费主要包含以下几点:
- 构造DefaultMQPullConsumer对象,为其设置nameserver的地址并启动consumer
- 通过fetchSubscribeMessageQueues方法获取该topic所有的消息队列
- 遍历消息队列,通过pull相关的API从broker指定的消息队列中拉取消息到客户端
- 调用updateConsumeOffset方法更新消息消费进展(注意:这里并不是实时的向broker端更新消费进度,updateConsumeOffset方法只是更新了DefaultMQPullConsumer缓存中的offsetstore,在consumer启动过程中会有一个定时任务persistAllConsumerOffset:默认5秒向broker发送请求更新一次数据)
public class PullConsumer {
private static final Map<MessageQueue, Long> OFFSE_TABLE = new HashMap<MessageQueue, Long>();
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("please_rename_unique_group_name_5");
consumer.setNamesrvAddr("127.0.0.1:9876");
consumer.start();
Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("broker-a");
for (MessageQueue mq : mqs) {
System.out.printf("Consume from the queue: %s%n", mq);
SINGLE_MQ:
while (true) {
try {
PullResult pullResult =
consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
System.out.printf("%s%n", pullResult);
putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
switch (pullResult.getPullStatus()) {
case FOUND:
//这里可以添加消费端对消息的处理逻辑
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break SINGLE_MQ;
case OFFSET_ILLEGAL:
break;
default:
break;
}
consumer.updateConsumeOffset(mq, pullResult.getNextBeginOffset());
} catch (Exception e) {
e.printStackTrace();
}
}
}
consumer.shutdown();
}
private static long getMessageQueueOffset(MessageQueue mq) {
Long offset = OFFSE_TABLE.get(mq);
if (offset != null)
return offset;
return 0;
}
private static void putMessageQueueOffset(MessageQueue mq, long offset) {
OFFSE_TABLE.put(mq, offset);
}
}
上面示例演示的是一个consumerGroup中只包含一个consumer,如果有多个消费者的话就会涉及到消息队列分配的问题,而消息队列分配的问题并没有被包含在DefaultMQPullConsumer的实现中。与PUSH模式DefaultMQPushConsumer的实现相比,后者的实现中包含了消息队列负载均衡、消费进度更新等,用户只需要在消费端注册MessageListener关注消息消费逻辑即可。
DefaultLitePullConsumer
1.使用demo
鉴于DefaultMQPullConsumer的问题,官方在4.6.0中提供了pull模式consumer的另一种实现DefaultLitePullConsumer,并推荐使用DefaultLitePullConsumer。同样先来看下其使用demo,其使用可以分为两种方式分别是订阅方式和分配方式。
- 分配方式
public class LitePullConsumerAssign {
public static volatile boolean running = true;
public static void main(String[] args) throws Exception {
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("please_rename_unique_group_name");
litePullConsumer.setNamesrvAddr("127.0.0.1:9876");
//设置是否自动提交offset,默认值是true
litePullConsumer.setAutoCommit(false);
litePullConsumer.start();
//获取topic的MessageQueue集合
Collection<MessageQueue> mqSet = litePullConsumer.fetchMessageQueues("TopicTest");
List<MessageQueue> list = new ArrayList<>(mqSet);
List<MessageQueue> assignList = new ArrayList<>();
for (int i = 0; i < list.size() / 2; i++) {
assignList.add(list.get(i));
}
//指定consumer消费的消息队列,使用该模式后消息队列不会自动重平衡,此时SubscriptionType为SubscriptionType.ASSIGN
litePullConsumer.assign(assignList);
//修改consumer拉取消息的偏移量
litePullConsumer.seek(assignList.get(0), 10);
try {
while (running) {
//消息拉取
List<MessageExt> messageExts = litePullConsumer.poll();
System.out.printf("%s %n", messageExts);
//由于前面调用setAutoCommit方法将自动提交位点属性设置为false,所以这里调用commitSync将消费位点提交到内存中的offsetstore,最终会通过定时任务将消费位点提交给broker
litePullConsumer.commitSync();
}
} finally {
litePullConsumer.shutdown();
}
}
}
- 订阅方式
public class LitePullConsumerSubscribe {
public static volatile boolean running = true;
public static void main(String[] args) throws Exception {
DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("lite_pull_consumer_test");
litePullConsumer.setNamesrvAddr("127.0.0.1:9876");
litePullConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
//此时SubscriptionType为SubscriptionType.SUBSCRIBE,使用该方式后用户不需要考虑消息队列分配的问题
litePullConsumer.subscribe("TopicTest", "*");
litePullConsumer.start();
try {
while (running) {
List<MessageExt> messageExts = litePullConsumer.poll();
System.out.printf("%s%n", messageExts);
}
} finally {
litePullConsumer.shutdown();
}
}
}
2.消费流程
接着来看DefaultLitePullConsumer的消费流程:
订阅方式的消费流程
广播模式与集群模式的区别在于第一步更新assignedMessageQueueState时是将topic所有的MessageQueue添加到assignedMessageQueueState中。使用订阅方式的DefaultLitePullConsumer,消息队列分配不需要应用来考虑。
订阅方式消费流程的步骤如下:
(1)consumer端调用subscribe方法订阅topic,在这个过程中会完成以下操作:
- 将订阅类型设置为SubscriptionType.SUBSCRIBE(在RocketMQ中订阅类型分为三种,分别是NONE, SUBSCRIBE, ASSIGN,默认是NONE)
- 构建订阅信息并将其添加到rebalanceImpl的注册表subscriptionInner
- 为assignedMessageQueue设置rebalanceImpl(assignedMessageQueue是consumer端用来存储当前为当前consumer分配的MessageQueue及其MessageQueueState)
(2)启动consumer,在这个过程中会启动RebalanceService,它会每20秒执行一次doRebalance方法完成消息队列分配,当consumer分配的消息队列发生变化时会对assignMessageQueueState以及拉取任务(在DefaultLitePullConsumer的实现中消息拉取任务被封装为PullTaskImpl,一个消息队列对应一个PullTaskImpl),在更新拉取任务时会完成两个任务:一个是更新taskTable(taskTable用来存放PullTaskImpl);一个是根据新分配的消息队列构建拉取任务PullTaskImpl,将该任务添加到taskTable中并使用ScheduledThreadPoolExecutor来调度该拉取任务(ScheduledThreadPoolExecutor中消息拉取的线程数量默认是20个,这个是与PUSH模式所不同的)
(3)执行拉取任务,即PullTaskImpl的run方法,在其执行过程中会完成以下操作:
- 在该方法中会向broker发送RequestCode.PULL_MESSAGE请求拉取消息,broker在收到请求后会返回给consumer端PullResult
- PullResult中pullStatus如果为FOUND,则会完成两个任务:一个是更新assignedMessageQueueState中消息队列对应的processQueue中存储的消息;一个是构建一个ConsumeRequest并将其提交到consumeRequestCache(ConsumeRequest中包含messageQueue、processQueue和messageExts)
- 更新assignedMessageQueueState中该消息队列的pullOffset
(4)consumer端调用poll方法从consumeRequestCache中获取消息并更新assignedMessageQueueState中该消息队列的consumeOffset
最后因为Rebalanceservice会每20秒进行一次消息队列分配,如果当前的consumer的消息队列发生变化则会调用messageQueueChanged(topic, mqAll, mqDivided)方法来更新assignedMessageQueueState和拉取任务(包含taskTable)
分配方式的消费流程
下图的消费流程是consumer先启动然后调用assign方法分配给该consumer的消息队列
使用DefaultLitePullConsumer的分配方式时,如果先调用assign来分配消息队列再启动consumer与上图流程有些区别:
- 在assign方法执行完成后仅仅更新了assignedMessageQueueState
- 在consumer启动过程中会调用operateAfterRunning方法,在该方法中会调用updateAssignPullTask(assignedMessageQueue.messageQueues())来更新拉取任务
分配方式消费流程的步骤如下:
(1)调用assign方法完成两个任务:一个是更新assignedMessageQueueState;一个是更新拉取任务(包含taskTable和构建拉取消息PullTaskImpl)
(2)执行拉取任务,即PullTaskImplement的run方法,在其执行过程中会完成以下操作:
- 在该方法中会想broker发送RequestCode.PULL_MESSAGE请求拉取消息,broker在收到请求后会返回给consumer端PullResult
- PullResult中pullStatus如果为FOUND,则会完成两个任务:一个是更新assignedMessageQueueState中消息队列对应的processQueue中存储的消息;一个是构建一个ConsumeRequest并将其提交到consumeRequestCache(ConsumeRequest中包含messageQueue、processQueue和messageExts)
- 更新assignedMessageQueueState中该消息队列的pullOffset
(3)consumer端调用poll方法从consumeRequestCache中获取消息并更新assignedMessageQueueState中该消息队列的consumeOffset
重点方法分析
(1)messageQueueChanged
通过subscribe方法订阅topic,消息队列的分配及平衡是由Rebalanceservice管理的,如果consumer的数量、topic的MessageQueue数量发生变化时,各个consumer分配的MessageQueue会动态发生变化。当consumer所分配的MessageQueue发生变化时,其最终会调用messageQueueChanged方法来更新assignedMessageQueueState以及拉取任务(taskTable以及构建PullTaskImpl)
public void messageQueueChanged(String topic, Set<MessageQueue> mqAll, Set<MessageQueue> mqDivided) {
MessageModel messageModel = defaultLitePullConsumer.getMessageModel();
switch (messageModel) {
case BROADCASTING:
updateAssignedMessageQueue(topic, mqAll);
updatePullTask(topic, mqAll);
break;
case CLUSTERING:
updateAssignedMessageQueue(topic, mqDivided);
updatePullTask(topic, mqDivided);
break;
default:
break;
}
}
messageQueueChanged方法中最终会调用以下三个函数:
public void updateAssignedMessageQueue(String topic, Collection<MessageQueue> assigned) {
synchronized (this.assignedMessageQueueState) {
Iterator<Map.Entry<MessageQueue, MessageQueueState>> it = this.assignedMessageQueueState.entrySet().iterator();
while (it.hasNext()) {
Map.Entry<MessageQueue, MessageQueueState> next = it.next();
if (next.getKey().getTopic().equals(topic)) {
//如果当前分配的消息队列中不包含之前分配的消息队列,则将该消息队列对应的MessageQueueState中的ProcessQueue的dropped属性设置为true并将该消息队列从assignedMessageQueueState中删除
if (!assigned.contains(next.getKey())) {
next.getValue().getProcessQueue().setDropped(true);
it.remove();
}
}
}
addAssignedMessageQueue(assigned);
}
}
private void updatePullTask(String topic, Set<MessageQueue> mqNewSet) {
Iterator<Map.Entry<MessageQueue, PullTaskImpl>> it = this.taskTable.entrySet().iterator();
while (it.hasNext()) {
Map.Entry<MessageQueue, PullTaskImpl> next = it.next();
if (next.getKey().getTopic().equals(topic)) {
//如果当前分配的消息队列不包含之前分配的消息队列,则将后者对应的拉取任务PullTaskImpl中的cancelled设置为true表示该任务被取消了
if (!mqNewSet.contains(next.getKey())) {
next.getValue().setCancelled(true);
it.remove();
}
}
}
startPullTask(mqNewSet);
}
private void startPullTask(Collection<MessageQueue> mqSet) {
for (MessageQueue messageQueue : mqSet) {
//如果taskTable不包含新分配的消息队列,则对该消息队列构建新的拉取任务然后将其放入到taskTable中,最后scheduledThreadPoolExecutor调度该任务
if (!this.taskTable.containsKey(messageQueue)) {
PullTaskImpl pullTask = new PullTaskImpl(messageQueue);
this.taskTable.put(messageQueue, pullTask);
this.scheduledThreadPoolExecutor.schedule(pullTask, 0, TimeUnit.MILLISECONDS);
}
}
}
(2)PullTaskImpl的run方法
在这个方法里面需要注意以下几点:
- 条件判断,需要注意各种场景:当前消息队列是否还是被分给当前的consumer、当前消息队列是否被暂停消费等
- 计算下次拉取消息的位点
public void run() {
//判断该任务是否被取消
if (!this.isCancelled()) {
//判断该消息队列是否被客户端设置为暂停消费,如果被暂停消费则让该拉取任务延迟1秒后继续执行
if (assignedMessageQueue.isPaused(messageQueue)) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_PAUSE, TimeUnit.MILLISECONDS);
log.debug("Message Queue: {} has been paused!", messageQueue);
return;
}
//获取消息队列对应的ProcessQueue
ProcessQueue processQueue = assignedMessageQueue.getProcessQueue(messageQueue);
//if条件满足的场景是当前消息队列已经不再被consumer消费
if (null == processQueue || processQueue.isDropped()) {
log.info("The message queue not be able to poll, because it's dropped. group={}, messageQueue={}", defaultLitePullConsumer.getConsumerGroup(), this.messageQueue);
return;
}
//流控
if (consumeRequestCache.size() * defaultLitePullConsumer.getPullBatchSize() > defaultLitePullConsumer.getPullThresholdForAll()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((consumeRequestFlowControlTimes++ % 1000) == 0)
log.warn("The consume request count exceeds threshold {}, so do flow control, consume request count={}, flowControlTimes={}", consumeRequestCache.size(), consumeRequestFlowControlTimes);
return;
}
long cachedMessageCount = processQueue.getMsgCount().get();
long cachedMessageSizeInMiB = processQueue.getMsgSize().get() / (1024 * 1024);
//流控
if (cachedMessageCount > defaultLitePullConsumer.getPullThresholdForQueue()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((queueFlowControlTimes++ % 1000) == 0) {
log.warn(
"The cached message count exceeds the threshold {}, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, flowControlTimes={}",
defaultLitePullConsumer.getPullThresholdForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, queueFlowControlTimes);
}
return;
}
//流控
if (cachedMessageSizeInMiB > defaultLitePullConsumer.getPullThresholdSizeForQueue()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((queueFlowControlTimes++ % 1000) == 0) {
log.warn(
"The cached message size exceeds the threshold {} MiB, so do flow control, minOffset={}, maxOffset={}, count={}, size={} MiB, flowControlTimes={}",
defaultLitePullConsumer.getPullThresholdSizeForQueue(), processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), cachedMessageCount, cachedMessageSizeInMiB, queueFlowControlTimes);
}
return;
}
//流控
if (processQueue.getMaxSpan() > defaultLitePullConsumer.getConsumeMaxSpan()) {
scheduledThreadPoolExecutor.schedule(this, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL, TimeUnit.MILLISECONDS);
if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
log.warn(
"The queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, flowControlTimes={}",
processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(), queueMaxSpanFlowControlTimes);
}
return;
}
//获取下次拉取消息的位点
long offset = nextPullOffset(messageQueue);
long pullDelayTimeMills = 0;
try {
SubscriptionData subscriptionData;
if (subscriptionType == SubscriptionType.SUBSCRIBE) {
String topic = this.messageQueue.getTopic();
subscriptionData = rebalanceImpl.getSubscriptionInner().get(topic);
} else {
String topic = this.messageQueue.getTopic();
subscriptionData = FilterAPI.buildSubscriptionData(topic, SubscriptionData.SUB_ALL);
}
//拉取消息
PullResult pullResult = pull(messageQueue, subscriptionData, offset, defaultLitePullConsumer.getPullBatchSize());
switch (pullResult.getPullStatus()) {
//消息拉取成功后需要更新processQueue以及提交ConsumeRequest供客户端拉取
case FOUND:
final Object objLock = messageQueueLock.fetchLockObject(messageQueue);
synchronized (objLock) {
if (pullResult.getMsgFoundList() != null && !pullResult.getMsgFoundList().isEmpty() && assignedMessageQueue.getSeekOffset(messageQueue) == -1) {
processQueue.putMessage(pullResult.getMsgFoundList());
submitConsumeRequest(new ConsumeRequest(pullResult.getMsgFoundList(), messageQueue, processQueue));
}
}
break;
case OFFSET_ILLEGAL:
log.warn("The pull request offset illegal, {}", pullResult.toString());
break;
default:
break;
}
//更新下次拉取消息的位点
updatePullOffset(messageQueue, pullResult.getNextBeginOffset());
} catch (Throwable e) {
pullDelayTimeMills = pullTimeDelayMillsWhenException;
log.error("An error occurred in pull message process.", e);
}
//这一步很关键,在完成一次消息拉取后通过这一步会进行下一次的消息拉取(保证消息持续拉取),延迟时间取决于上次一消息拉取的情况,如果上次拉取正常则会立刻启动本次消息拉取;如果上次拉取异常则延迟1秒后再进行消息拉取
if (!this.isCancelled()) {
scheduledThreadPoolExecutor.schedule(this, pullDelayTimeMills, TimeUnit.MILLISECONDS);
} else {
log.warn("The Pull Task is cancelled after doPullTask, {}", messageQueue);
}
}
}
在拉取任务运行过程中会计算本次拉取消息的位点,即nextPullOffset方法。该方法中首先会获取消息队列对应的seekOffset,它的默认值为-1,在DefaultLitePullConsumer的实现中封装了一个seek方法,调用该方法可以修改消息队列下次拉取消息的偏移量,seek方法最终修改的就是seekOffset。这里首先判断seekOffset是否被修改(即其值是否为-1),如果不为-1则使用seekOffset的值就是消息拉取的偏移量,然后使用该值更新其consumeOffset,最后再将seekOffset修改为-1。如果seekOffset的值为-1,则从assignedMessageQueue获取其pullOffset,如果pullOffset为-1,则调用fetchConsumeOffset方法获取拉取位点,否则返回pullOffset。
private long nextPullOffset(MessageQueue messageQueue) {
long offset = -1;
long seekOffset = assignedMessageQueue.getSeekOffset(messageQueue);
if (seekOffset != -1) {
offset = seekOffset;
assignedMessageQueue.updateConsumeOffset(messageQueue, offset);
assignedMessageQueue.setSeekOffset(messageQueue, -1);
} else {
offset = assignedMessageQueue.getPullOffset(messageQueue);
if (offset == -1) {
offset = fetchConsumeOffset(messageQueue);
}
}
return offset;
}
(3)assign
该方法实现的功能是为consumer分配消息队列,该方法涉及的操作如下:
- 设置consumer的订阅类型为SubscriptionType.ASSIGN
- 更新assignedMessageQueueState
- 如果consumer的状态是running,则调用updateAssignPullTask更新拉取任务
public synchronized void assign(Collection<MessageQueue> messageQueues) {
if (messageQueues == null || messageQueues.isEmpty()) {
throw new IllegalArgumentException("Message queues can not be null or empty.");
}
setSubscriptionType(SubscriptionType.ASSIGN);
assignedMessageQueue.updateAssignedMessageQueue(messageQueues);
if (serviceState == ServiceState.RUNNING) {
updateAssignPullTask(messageQueues);
}
}
(4)seek
seek方法实现的功能是修改下次消息拉取的偏移量,该方法涉及的操作如下:
- 判断当前消息队列是否被分配给当前的consumer,如果没有则抛出异常
- 从broker端获取该消息队列的最小及最大的逻辑位点
- 判断下次消息拉取的位点是否在上一步获取的最小最大逻辑位点的范围内,如果在则更新该消息队列在assignedMessageQueueState中的seekOffset并清理该消息队列对应的processQueue以及ConsumeRequestCache中对应的ConsumeRequest
public synchronized void seek(MessageQueue messageQueue, long offset) throws MQClientException {
if (!assignedMessageQueue.messageQueues().contains(messageQueue)) {
if (subscriptionType == SubscriptionType.SUBSCRIBE) {
throw new MQClientException("The message queue is not in assigned list, may be rebalancing, message queue: " + messageQueue, null);
} else {
throw new MQClientException("The message queue is not in assigned list, message queue: " + messageQueue, null);
}
}
long minOffset = minOffset(messageQueue);
long maxOffset = maxOffset(messageQueue);
if (offset < minOffset || offset > maxOffset) {
throw new MQClientException("Seek offset illegal, seek offset = " + offset + ", min offset = " + minOffset + ", max offset = " + maxOffset, null);
}
final Object objLock = messageQueueLock.fetchLockObject(messageQueue);
synchronized (objLock) {
assignedMessageQueue.setSeekOffset(messageQueue, offset);
clearMessageQueueInCache(messageQueue);
}
}
(5)pause
pause方法实现的功能是暂停指定消息队列的消费,其具体实现是在assignedMessageQueueState中获取该消息队里对应的MessageQueueState,将MessageQueueState中的pause属性设置为true
public void pause(Collection<MessageQueue> messageQueues) {
for (MessageQueue messageQueue : messageQueues) {
MessageQueueState messageQueueState = assignedMessageQueueState.get(messageQueue);
if (assignedMessageQueueState.get(messageQueue) != null) {
messageQueueState.setPaused(true);
}
}
}
(6)resume
resume方法实现的功能是恢复指定消息队列的消费,其具体实现是在assignedMessageQueueState中获取该消息队里对应的MessageQueueState,将MessageQueueState中的pause属性设置为false
public void resume(Collection<MessageQueue> messageQueueCollection) {
for (MessageQueue messageQueue : messageQueueCollection) {
MessageQueueState messageQueueState = assignedMessageQueueState.get(messageQueue);
if (assignedMessageQueueState.get(messageQueue) != null) {
messageQueueState.setPaused(false);
}
}
}
DefaultMQPullConsumer与DefaultLitePullConsumer对比
(1)DefaultMQPullConsumer
应用使用DefaultMQPullConsumer则需要应用考虑以下问题:
- 消息队列负载均衡
- 消费位点提交及存储
(2)DefaultLitePullConsumer
DefaultLitePullConsumer的实现提供了以下特性:
- 订阅方式消费消息支持消息队列负载均衡
- 分配方式消费消息支持收到分配消息队列,此时不支持负载均衡
- 提供了seek方法方便用户重置消费位点
- 提供commitSync方法方便用户手动提交消费位点