版本：4.2.0
RocketMQ消费可以分为并发消费和顺序消费。
顺序消费顾名思义就是按照发送的顺序消费消息，一般的业务需求用MessageListenerConcurrently就可以了。
有些特殊的业务需求可能需要保证消息的消费是有序的，比如binlog同步等等，那么就用到MessageListenerOrderly，仅仅用MessageListenerOrderly就能保证顺序消费吗？
可能也许应该能吧？其实只需要写个单元测试就能测试出来，多试几次。
既然不能保证那为啥要搞个MessageListenerOrderly呢。
先来了解一下MessageListenerOrderly起到的作用
定义消费者的时候，我们注册了一个listener
consumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> list, ConsumeOrderlyContext consumeOrderlyContext) {
for(MessageExt message : list){
System.out.println(new String(message.getBody()));
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
在DefaultMQPushConsumerImpl的start方法里面判断是顺序消费还是并发消费
public synchronized void start() throws MQClientException {
。。。
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
this.consumeOrderly = true;
this.consumeMessageService =
new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
this.consumeOrderly = false;
this.consumeMessageService =
new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
this.consumeMessageService.start();
。。。
}
根据listener的类型判断是顺序消费还是并发消费。
consumer内部维护了一个pullRequestQueue，consumer刚启动的时候会为每个queue生成一个pullRequest并放到这个队列中，由另外一个线程不断的take，然后发送请求拉取消息。那么是如何控制顺序消费的呢？
这里的consumeMessageService已经被初始化为ConsumeMessageOrderlyService，
ConsumeMessageOrderlyService启动的时候启动了一个定时任务
public void start() {
if (MessageModel.CLUSTERING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())) {
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
ConsumeMessageOrderlyService.this.lockMQPeriodically();
}
}, 1000 * 1, ProcessQueue.REBALANCE_LOCK_INTERVAL, TimeUnit.MILLISECONDS);
}
}
public synchronized void lockMQPeriodically() {
if (!this.stopped) {
this.defaultMQPushConsumerImpl.getRebalanceImpl().lockAll();
}
}
locakAll看起来像是锁住所有的。具体是锁啥？
public void lockAll() {
// 将订阅的queue按照broker分组
HashMap<String, Set<MessageQueue>> brokerMqs = this.buildProcessQueueTableByBrokerName();
Iterator<Entry<String, Set<MessageQueue>>> it = brokerMqs.entrySet().iterator();
while (it.hasNext()) {
Entry<String, Set<MessageQueue>> entry = it.next();
final String brokerName = entry.getKey();
// broker的queue
final Set<MessageQueue> mqs = entry.getValue();
if (mqs.isEmpty())
continue;
// 获取到broker的addr
FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true);
if (findBrokerResult != null) {
// 封装请求参数
LockBatchRequestBody requestBody = new LockBatchRequestBody();
requestBody.setConsumerGroup(this.consumerGroup);
requestBody.setClientId(this.mQClientFactory.getClientId());
requestBody.setMqSet(mqs);
try {
// 请求broker，broker会将请求中的MessageQueue加入到map中然后返回
Set<MessageQueue> lockOKMQSet =
this.mQClientFactory.getMQClientAPIImpl().lockBatchMQ(findBrokerResult.getBrokerAddr(), requestBody, 1000);
for (MessageQueue mq : lockOKMQSet) {
ProcessQueue processQueue = this.processQueueTable.get(mq);
if (processQueue != null) {
if (!processQueue.isLocked()) {
log.info("the message queue locked OK, Group: {} {}", this.consumerGroup, mq);
}
// 设置为locaked，拉取消息的时候会用到这个参数
processQueue.setLocked(true);
processQueue.setLastLockTimestamp(System.currentTimeMillis());
}
}
for (MessageQueue mq : mqs) {
if (!lockOKMQSet.contains(mq)) {
ProcessQueue processQueue = this.processQueueTable.get(mq);
if (processQueue != null) {
// 设置为false表示当前queue不被当前consumer订阅
processQueue.setLocked(false);
log.warn("the message queue locked Failed, Group: {} {}", this.consumerGroup, mq);
}
}
}
} catch (Exception e) {
log.error("lockBatchMQ exception, " + mqs, e);
}
}
}
}
再来看看DefaultMQPushConsumerImpl拉取消息的时候如何控制顺序
public void pullMessage(final PullRequest pullRequest) {
。。。
if (!this.consumeOrderly) {
if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL);
if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) {
log.warn(
"the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
pullRequest, queueMaxSpanFlowControlTimes);
}
return;
}
} else {
// ConsumeMessageOrderlyService启动时已经将locked置为true
if (processQueue.isLocked()) {
// lockedFirst默认为false
if (!pullRequest.isLockedFirst()) {
// 从broker拉取lastOffset
final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue());
boolean brokerBusy = offset < pullRequest.getNextOffset();
log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}",
pullRequest, offset, brokerBusy);
if (brokerBusy) {
log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}",
pullRequest, offset);
}
// consumer貌似只有这个地方更新lockedFirst
pullRequest.setLockedFirst(true);
pullRequest.setNextOffset(offset);
}
} else {
// 重新加入到pullRequestQueue等待下次
this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
log.info("pull message later because not locked in broker, {}", pullRequest);
return;
}
}
。。。
}
接下来就是顺序消费
public void run() {
if (this.processQueue.isDropped()) {
log.warn("run, the message queue not be able to consume, because it's dropped. {}", this.messageQueue);
return;
}
final Object objLock = messageQueueLock.fetchLockObject(this.messageQueue);
synchronized (objLock) {
if (MessageModel.BROADCASTING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())
|| (this.processQueue.isLocked() && !this.processQueue.isLockExpired())) {
final long beginTime = System.currentTimeMillis();
for (boolean continueConsume = true; continueConsume; ) {
// 一堆校验，不重要
。。。
// 批量消费的大小
final int consumeBatchSize =
ConsumeMessageOrderlyService.this.defaultMQPushConsumer.getConsumeMessageBatchMaxSize();
// 取出消息，并且加入到另外一个treemap
List<MessageExt> msgs = this.processQueue.takeMessags(consumeBatchSize);
if (!msgs.isEmpty()) {
final ConsumeOrderlyContext context = new ConsumeOrderlyContext(this.messageQueue);
ConsumeOrderlyStatus status = null;
。。。
long beginTimestamp = System.currentTimeMillis();
ConsumeReturnType returnType = ConsumeReturnType.SUCCESS;
boolean hasException = false;
try {
this.processQueue.getLockConsume().lock();
if (this.processQueue.isDropped()) {
log.warn("consumeMessage, the message queue not be able to consume, because it's dropped. {}",
this.messageQueue);
break;
}
// 消息消息并返回状态
status = messageListener.consumeMessage(Collections.unmodifiableList(msgs), context);
} catch (Throwable e) {
log.warn("consumeMessage exception: {} Group: {} Msgs: {} MQ: {}",
RemotingHelper.exceptionSimpleDesc(e),
ConsumeMessageOrderlyService.this.consumerGroup,
msgs,
messageQueue);
hasException = true;
} finally {
this.processQueue.getLockConsume().unlock();
}
if (null == status
|| ConsumeOrderlyStatus.ROLLBACK == status
|| ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT == status) {
log.warn("consumeMessage Orderly return not OK, Group: {} Msgs: {} MQ: {}",
ConsumeMessageOrderlyService.this.consumerGroup,
msgs,
messageQueue);
}
long consumeRT = System.currentTimeMillis() - beginTimestamp;
if (null == status) {
if (hasException) {
returnType = ConsumeReturnType.EXCEPTION;
} else {
returnType = ConsumeReturnType.RETURNNULL;
}
} else if (consumeRT >= defaultMQPushConsumer.getConsumeTimeout() * 60 * 1000) {
returnType = ConsumeReturnType.TIME_OUT;
} else if (ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT == status) {
returnType = ConsumeReturnType.FAILED;
} else if (ConsumeOrderlyStatus.SUCCESS == status) {
returnType = ConsumeReturnType.SUCCESS;
}
if (ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.hasHook()) {
consumeMessageContext.getProps().put(MixAll.CONSUME_CONTEXT_TYPE, returnType.name());
}
if (null == status) {
status = ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT;
}
。。。
// 记录消费时间
ConsumeMessageOrderlyService.this.getConsumerStatsManager()
.incConsumeRT(ConsumeMessageOrderlyService.this.consumerGroup, messageQueue.getTopic(), consumeRT);
// 处理消费结果
continueConsume = ConsumeMessageOrderlyService.this.processConsumeResult(msgs, status, context, this);
} else {
continueConsume = false;
}
}
} else {
if (this.processQueue.isDropped()) {
log.warn("the message queue not be able to consume, because it's dropped. {}", this.messageQueue);
return;
}
ConsumeMessageOrderlyService.this.tryLockLaterAndReconsume(this.messageQueue, this.processQueue, 100);
}
}
}
上面逻辑就是消费，重要的在processConsumeResult方法里面
public boolean processConsumeResult(
final List<MessageExt> msgs,
final ConsumeOrderlyStatus status,
final ConsumeOrderlyContext context,
final ConsumeRequest consumeRequest
) {
boolean continueConsume = true;
long commitOffset = -1L;
if (context.isAutoCommit()) {
switch (status) {
case COMMIT:
case ROLLBACK:
log.warn("the message queue consume result is illegal, we think you want to ack these message {}",
consumeRequest.getMessageQueue());
case SUCCESS:
// 如果消费成功，会更新offset
commitOffset = consumeRequest.getProcessQueue().commit();
this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
break;
case SUSPEND_CURRENT_QUEUE_A_MOMENT:
this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size());
// 检查消费的次数，默认设置的int最大值
if (checkReconsumeTimes(msgs)) {
// 又放回到原先的treeMap
consumeRequest.getProcessQueue().makeMessageToCosumeAgain(msgs);
// 一秒之后再消费
this.submitConsumeRequestLater(
consumeRequest.getProcessQueue(),
consumeRequest.getMessageQueue(),
context.getSuspendCurrentQueueTimeMillis());
continueConsume = false;
} else {
// 超过限定的消费次数也会提交offset
commitOffset = consumeRequest.getProcessQueue().commit();
}
break;
default:
break;
}
} else {
。。。
}
if (commitOffset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
// 提交offset
this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), commitOffset, false);
}
return continueConsume;
}
看到这里我豁然开朗，其实就是控制当前queue的offset的消费进度来控制消费顺序，但是每个topic一般情况包含多个queue，当前消费只能控制当前queue的消费进度并没有同时控制所有queue的消费进度，可想而知多个queue的情况下并不能保证顺序消费，如果只用一个queue是不是就会保证顺序消费呢？从逻辑来看是可以的。
单个queue的情况下如果用多线程会不会影响消费顺序？
其实并不会。实时消费主要是PullMessageService消费pullRequestQueue，pullMessageService是单线程，顺序消费的情况下，消费成功之后才会提交offset并将pullRequest重新放到pullRequestQueue，并不会存在同一个queue同时被两个线程消费，消费失败会存在重试，并不会提交offset也不会将pullRequest重新放到pullRequestQueue，这样别的线程也消费不到当前pullRequest里的消息。
说了那么多，怎么实现顺序消费？
producer发送消息选择queue：
DefaultMQProducer producer = new DefaultMQProducer("producer");
// 设置nameserver地址
producer.setNamesrvAddr("localhost:9876");
// 启动producer
producer.start();
for (int i = 0; i < 10; i++) {
Message message = new Message("order", ("" + i).getBytes());
// send方法里的最后一个参数会被传到select方法的最后一个参数，选择queue可以根据数据的特征，也可以像这样直接指定一个队列
producer.send(message, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
return list.get(0);
}
}, 0);
}
// 关闭producer
//producer.shutdown();
consumer顺序消费：
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer");
consumer.setNamesrvAddr("localhost:9876");
consumer.subscribe("order", "*");
// 这里使用单线程消费。
consumer.setConsumeThreadMin(1);
consumer.setConsumeThreadMax(1);
// 使用顺序消费的listener
consumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> list, ConsumeOrderlyContext consumeOrderlyContext) {
for(MessageExt message : list){
String str = new String(message.getBody());
System.out.println(str);
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
总结：
要保证业务的顺序
1.消费者使用单线程发送消息
2.消息被发送到同一个queue
3.消费者使用MessageListenerOrderly
4.消费者用单线程消费
2555
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
