版本:4.2.0
RocketMQ消费可以分为并发消费和顺序消费。
顺序消费顾名思义就是按照发送的顺序消费消息,一般的业务需求用MessageListenerConcurrently就可以了。
有些特殊的业务需求可能需要保证消息的消费是有序的,比如binlog同步等等,那么就用到MessageListenerOrderly,仅仅用MessageListenerOrderly就能保证顺序消费吗?
可能也许应该能吧?其实只需要写个单元测试就能测试出来,多试几次。
既然不能保证那为啥要搞个MessageListenerOrderly呢。
先来了解一下MessageListenerOrderly起到的作用
定义消费者的时候,我们注册了一个listener
consumer.registerMessageListener(new MessageListenerOrderly() { @Override public ConsumeOrderlyStatus consumeMessage(List<MessageExt> list, ConsumeOrderlyContext consumeOrderlyContext) { for(MessageExt message : list){ System.out.println(new String(message.getBody())); } return ConsumeOrderlyStatus.SUCCESS; } });
在DefaultMQPushConsumerImpl的start方法里面判断是顺序消费还是并发消费
public synchronized void start() throws MQClientException { 。。。 if (this.getMessageListenerInner() instanceof MessageListenerOrderly) { this.consumeOrderly = true; this.consumeMessageService = new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner()); } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) { this.consumeOrderly = false; this.consumeMessageService = new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner()); } this.consumeMessageService.start(); 。。。 }
根据listener的类型判断是顺序消费还是并发消费。
consumer内部维护了一个pullRequestQueue,consumer刚启动的时候会为每个queue生成一个pullRequest并放到这个队列中,由另外一个线程不断的take,然后发送请求拉取消息。那么是如何控制顺序消费的呢?
这里的consumeMessageService已经被初始化为ConsumeMessageOrderlyService,
ConsumeMessageOrderlyService启动的时候启动了一个定时任务
public void start() { if (MessageModel.CLUSTERING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())) { this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { ConsumeMessageOrderlyService.this.lockMQPeriodically(); } }, 1000 * 1, ProcessQueue.REBALANCE_LOCK_INTERVAL, TimeUnit.MILLISECONDS); } }
public synchronized void lockMQPeriodically() { if (!this.stopped) { this.defaultMQPushConsumerImpl.getRebalanceImpl().lockAll(); } }
locakAll看起来像是锁住所有的。具体是锁啥?
public void lockAll() { // 将订阅的queue按照broker分组 HashMap<String, Set<MessageQueue>> brokerMqs = this.buildProcessQueueTableByBrokerName(); Iterator<Entry<String, Set<MessageQueue>>> it = brokerMqs.entrySet().iterator(); while (it.hasNext()) { Entry<String, Set<MessageQueue>> entry = it.next(); final String brokerName = entry.getKey(); // broker的queue final Set<MessageQueue> mqs = entry.getValue(); if (mqs.isEmpty()) continue; // 获取到broker的addr FindBrokerResult findBrokerResult = this.mQClientFactory.findBrokerAddressInSubscribe(brokerName, MixAll.MASTER_ID, true); if (findBrokerResult != null) { // 封装请求参数 LockBatchRequestBody requestBody = new LockBatchRequestBody(); requestBody.setConsumerGroup(this.consumerGroup); requestBody.setClientId(this.mQClientFactory.getClientId()); requestBody.setMqSet(mqs); try { // 请求broker,broker会将请求中的MessageQueue加入到map中然后返回 Set<MessageQueue> lockOKMQSet = this.mQClientFactory.getMQClientAPIImpl().lockBatchMQ(findBrokerResult.getBrokerAddr(), requestBody, 1000); for (MessageQueue mq : lockOKMQSet) { ProcessQueue processQueue = this.processQueueTable.get(mq); if (processQueue != null) { if (!processQueue.isLocked()) { log.info("the message queue locked OK, Group: {} {}", this.consumerGroup, mq); } // 设置为locaked,拉取消息的时候会用到这个参数 processQueue.setLocked(true); processQueue.setLastLockTimestamp(System.currentTimeMillis()); } } for (MessageQueue mq : mqs) { if (!lockOKMQSet.contains(mq)) { ProcessQueue processQueue = this.processQueueTable.get(mq); if (processQueue != null) { // 设置为false表示当前queue不被当前consumer订阅 processQueue.setLocked(false); log.warn("the message queue locked Failed, Group: {} {}", this.consumerGroup, mq); } } } } catch (Exception e) { log.error("lockBatchMQ exception, " + mqs, e); } } } }
再来看看DefaultMQPushConsumerImpl拉取消息的时候如何控制顺序
public void pullMessage(final PullRequest pullRequest) { 。。。 if (!this.consumeOrderly) { if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) { this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_FLOW_CONTROL); if ((queueMaxSpanFlowControlTimes++ % 1000) == 0) { log.warn( "the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}", processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(), pullRequest, queueMaxSpanFlowControlTimes); } return; } } else { // ConsumeMessageOrderlyService启动时已经将locked置为true if (processQueue.isLocked()) { // lockedFirst默认为false if (!pullRequest.isLockedFirst()) { // 从broker拉取lastOffset final long offset = this.rebalanceImpl.computePullFromWhere(pullRequest.getMessageQueue()); boolean brokerBusy = offset < pullRequest.getNextOffset(); log.info("the first time to pull message, so fix offset from broker. pullRequest: {} NewOffset: {} brokerBusy: {}", pullRequest, offset, brokerBusy); if (brokerBusy) { log.info("[NOTIFYME]the first time to pull message, but pull request offset larger than broker consume offset. pullRequest: {} NewOffset: {}", pullRequest, offset); } // consumer貌似只有这个地方更新lockedFirst pullRequest.setLockedFirst(true); pullRequest.setNextOffset(offset); } } else { // 重新加入到pullRequestQueue等待下次 this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION); log.info("pull message later because not locked in broker, {}", pullRequest); return; } } 。。。 }
接下来就是顺序消费
public void run() { if (this.processQueue.isDropped()) { log.warn("run, the message queue not be able to consume, because it's dropped. {}", this.messageQueue); return; } final Object objLock = messageQueueLock.fetchLockObject(this.messageQueue); synchronized (objLock) { if (MessageModel.BROADCASTING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel()) || (this.processQueue.isLocked() && !this.processQueue.isLockExpired())) { final long beginTime = System.currentTimeMillis(); for (boolean continueConsume = true; continueConsume; ) { // 一堆校验,不重要 。。。 // 批量消费的大小 final int consumeBatchSize = ConsumeMessageOrderlyService.this.defaultMQPushConsumer.getConsumeMessageBatchMaxSize(); // 取出消息,并且加入到另外一个treemap List<MessageExt> msgs = this.processQueue.takeMessags(consumeBatchSize); if (!msgs.isEmpty()) { final ConsumeOrderlyContext context = new ConsumeOrderlyContext(this.messageQueue); ConsumeOrderlyStatus status = null; 。。。 long beginTimestamp = System.currentTimeMillis(); ConsumeReturnType returnType = ConsumeReturnType.SUCCESS; boolean hasException = false; try { this.processQueue.getLockConsume().lock(); if (this.processQueue.isDropped()) { log.warn("consumeMessage, the message queue not be able to consume, because it's dropped. {}", this.messageQueue); break; } // 消息消息并返回状态 status = messageListener.consumeMessage(Collections.unmodifiableList(msgs), context); } catch (Throwable e) { log.warn("consumeMessage exception: {} Group: {} Msgs: {} MQ: {}", RemotingHelper.exceptionSimpleDesc(e), ConsumeMessageOrderlyService.this.consumerGroup, msgs, messageQueue); hasException = true; } finally { this.processQueue.getLockConsume().unlock(); } if (null == status || ConsumeOrderlyStatus.ROLLBACK == status || ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT == status) { log.warn("consumeMessage Orderly return not OK, Group: {} Msgs: {} MQ: {}", ConsumeMessageOrderlyService.this.consumerGroup, msgs, messageQueue); } long consumeRT = System.currentTimeMillis() - beginTimestamp; if (null == status) { if (hasException) { returnType = ConsumeReturnType.EXCEPTION; } else { returnType = ConsumeReturnType.RETURNNULL; } } else if (consumeRT >= defaultMQPushConsumer.getConsumeTimeout() * 60 * 1000) { returnType = ConsumeReturnType.TIME_OUT; } else if (ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT == status) { returnType = ConsumeReturnType.FAILED; } else if (ConsumeOrderlyStatus.SUCCESS == status) { returnType = ConsumeReturnType.SUCCESS; } if (ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.hasHook()) { consumeMessageContext.getProps().put(MixAll.CONSUME_CONTEXT_TYPE, returnType.name()); } if (null == status) { status = ConsumeOrderlyStatus.SUSPEND_CURRENT_QUEUE_A_MOMENT; } 。。。 // 记录消费时间 ConsumeMessageOrderlyService.this.getConsumerStatsManager() .incConsumeRT(ConsumeMessageOrderlyService.this.consumerGroup, messageQueue.getTopic(), consumeRT); // 处理消费结果 continueConsume = ConsumeMessageOrderlyService.this.processConsumeResult(msgs, status, context, this); } else { continueConsume = false; } } } else { if (this.processQueue.isDropped()) { log.warn("the message queue not be able to consume, because it's dropped. {}", this.messageQueue); return; } ConsumeMessageOrderlyService.this.tryLockLaterAndReconsume(this.messageQueue, this.processQueue, 100); } } }
上面逻辑就是消费,重要的在processConsumeResult方法里面
public boolean processConsumeResult( final List<MessageExt> msgs, final ConsumeOrderlyStatus status, final ConsumeOrderlyContext context, final ConsumeRequest consumeRequest ) { boolean continueConsume = true; long commitOffset = -1L; if (context.isAutoCommit()) { switch (status) { case COMMIT: case ROLLBACK: log.warn("the message queue consume result is illegal, we think you want to ack these message {}", consumeRequest.getMessageQueue()); case SUCCESS: // 如果消费成功,会更新offset commitOffset = consumeRequest.getProcessQueue().commit(); this.getConsumerStatsManager().incConsumeOKTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size()); break; case SUSPEND_CURRENT_QUEUE_A_MOMENT: this.getConsumerStatsManager().incConsumeFailedTPS(consumerGroup, consumeRequest.getMessageQueue().getTopic(), msgs.size()); // 检查消费的次数,默认设置的int最大值 if (checkReconsumeTimes(msgs)) { // 又放回到原先的treeMap consumeRequest.getProcessQueue().makeMessageToCosumeAgain(msgs); // 一秒之后再消费 this.submitConsumeRequestLater( consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue(), context.getSuspendCurrentQueueTimeMillis()); continueConsume = false; } else { // 超过限定的消费次数也会提交offset commitOffset = consumeRequest.getProcessQueue().commit(); } break; default: break; } } else { 。。。 } if (commitOffset >= 0 && !consumeRequest.getProcessQueue().isDropped()) { // 提交offset this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), commitOffset, false); } return continueConsume; }
看到这里我豁然开朗,其实就是控制当前queue的offset的消费进度来控制消费顺序,但是每个topic一般情况包含多个queue,当前消费只能控制当前queue的消费进度并没有同时控制所有queue的消费进度,可想而知多个queue的情况下并不能保证顺序消费,如果只用一个queue是不是就会保证顺序消费呢?从逻辑来看是可以的。
单个queue的情况下如果用多线程会不会影响消费顺序?
其实并不会。实时消费主要是PullMessageService消费pullRequestQueue,pullMessageService是单线程,顺序消费的情况下,消费成功之后才会提交offset并将pullRequest重新放到pullRequestQueue,并不会存在同一个queue同时被两个线程消费,消费失败会存在重试,并不会提交offset也不会将pullRequest重新放到pullRequestQueue,这样别的线程也消费不到当前pullRequest里的消息。
说了那么多,怎么实现顺序消费?
producer发送消息选择queue:
DefaultMQProducer producer = new DefaultMQProducer("producer"); // 设置nameserver地址 producer.setNamesrvAddr("localhost:9876"); // 启动producer producer.start(); for (int i = 0; i < 10; i++) { Message message = new Message("order", ("" + i).getBytes()); // send方法里的最后一个参数会被传到select方法的最后一个参数,选择queue可以根据数据的特征,也可以像这样直接指定一个队列 producer.send(message, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { return list.get(0); } }, 0); } // 关闭producer //producer.shutdown();
consumer顺序消费:
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer"); consumer.setNamesrvAddr("localhost:9876"); consumer.subscribe("order", "*"); // 这里使用单线程消费。 consumer.setConsumeThreadMin(1); consumer.setConsumeThreadMax(1); // 使用顺序消费的listener consumer.registerMessageListener(new MessageListenerOrderly() { @Override public ConsumeOrderlyStatus consumeMessage(List<MessageExt> list, ConsumeOrderlyContext consumeOrderlyContext) { for(MessageExt message : list){ String str = new String(message.getBody()); System.out.println(str); } return ConsumeOrderlyStatus.SUCCESS; } }); consumer.start();
总结:
要保证业务的顺序
1.消费者使用单线程发送消息
2.消息被发送到同一个queue
3.消费者使用MessageListenerOrderly
4.消费者用单线程消费