RocketMq的消息消费-延时消息消费(三)

概述

延迟消息也叫定时消息,是指需要在一段时间后被消费的消息。考虑这种场景:用户下单后10分钟后不付款将订单取消,可以在用户下单后发送延迟消息,延迟10分钟检查订单如果未支付则取消订单。RocketMQ本身的commitLog采用顺序存储消息的方式,如果在commitLog实现延迟消息,需要对commitLog排序等处理,这与RocketMQ的设计理念不合。故RocketMQ针对延迟消息,只支持特定时间的延迟处理,不支持任意延迟时间,具体的时间可以为1s、 5s、 10s、 30s、 1m、 2m、 3m、 4m、 5m、 6m、 7m、 8m、 9m、 10m、 20m、 30m、 1h、 2h。

延时消息的主要设计思路

Broker端内置延迟消息处理能力,核心实现思路都是一样:将延迟消息通过一个临时存储进行暂存,到期后才投递到目标Topic中。如下图所示:
在这里插入图片描述

步骤说明如下:

  1. producer要将一个延迟消息发送到某个Topic中
  2. Broker判断这是一个延迟消息后,将其通过临时存储进行暂存。
  3. Broker内部通过一个延迟服务(delay
    service)检查消息是否到期,将到期的消息投递到目标Topic中。这个的延迟服务名字为delay
    service,不同消息中间件的延迟服务模块名称可能不同。
  4. 消费者消费目标topic中的延迟投递的消息

在这里插入图片描述

整体设计

在这里插入图片描述
延迟消息在RocketMQ Broker端的流转如下图所示:
在这里插入图片描述

可以看到,总共有6个步骤,下面会对这6个步骤进行详细的讲解:

  1. 修改消息Topic名称和队列信息
  2. 转发消息到延迟主题的CosumeQueue中
  3. 延迟服务消费SCHEDULE_TOPIC_XXXX消息
  4. 将信息重新存储到CommitLog中
  5. 将消息投递到目标Topic中
  6. 消费者消费目标topic中的数据

源码剖析

topic替换
在org.apache.rocketmq.store.CommitLog#putMessage;

    public PutMessageResult putMessage(final MessageExtBrokerInner msg) {
        msg.setStoreTimestamp(System.currentTimeMillis());
        msg.setBodyCRC(UtilAll.crc32(msg.getBody()));
        AppendMessageResult result = null;
        StoreStatsService storeStatsService = this.defaultMessageStore.getStoreStatsService();

        String topic = msg.getTopic();
        int queueId = msg.getQueueId();

        final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
        if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE
            || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
            // 延迟消息
            if (msg.getDelayTimeLevel() > 0) {
                if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
                    msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
                }

                topic = ScheduleMessageService.SCHEDULE_TOPIC;
                queueId = ScheduleMessageService.delayLevel2QueueId(msg.getDelayTimeLevel());

                // 存储原始的topic, queueId
                MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_TOPIC, msg.getTopic());
                MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_QUEUE_ID, String.valueOf(msg.getQueueId()));
                msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));
                // 设置topic为延迟队列topic
                msg.setTopic(topic);
                msg.setQueueId(queueId);
            }
        }
    }

延迟队列的topic为 SCHEDULE_TOPIC_XXXX:

生成consumequeue
说一下延迟级别和queueid的对应关系,queueid=延迟级别-1。再一个是对consumequeue消息的tagcode特殊处理,将tagcode值设置为消息触发时间,逻辑见.CommitLog.checkMessageAndReturnSize:

// Timing message processing
                {
                    String t = propertiesMap.get(MessageConst.PROPERTY_DELAY_TIME_LEVEL);
                    if (ScheduleMessageService.SCHEDULE_TOPIC.equals(topic) && t != null) {
                        int delayLevel = Integer.parseInt(t);

                        if (delayLevel > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
                            delayLevel = this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel();
                        }

                        if (delayLevel > 0) {
                            tagsCode = this.defaultMessageStore.getScheduleMessageService().computeDeliverTimestamp(delayLevel,
                                storeTimestamp);
                        }
                    }
                }


延迟消息定时任务 ScheduleMessageService

继承自ConfigManager,初始化延迟级别相关的配置和定时任务。

定时任务执行;

直接看具体逻辑;

        public void executeOnTimeup() {
            // 获取consumequeue
            ConsumeQueue cq =
                ScheduleMessageService.this.defaultMessageStore.findConsumeQueue(SCHEDULE_TOPIC,
                    delayLevel2QueueId(delayLevel));

            long failScheduleOffset = offset;

            if (cq != null) {
                // 消息获取
                SelectMappedBufferResult bufferCQ = cq.getIndexBuffer(this.offset);
                if (bufferCQ != null) {
                    try {
                        long nextOffset = offset;
                        int i = 0;
                        ConsumeQueueExt.CqExtUnit cqExtUnit = new ConsumeQueueExt.CqExtUnit();
                        // 消息遍历,因为是consumequeue ,消息长度固定
                        for (; i < bufferCQ.getSize(); i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
                            long offsetPy = bufferCQ.getByteBuffer().getLong();
                            int sizePy = bufferCQ.getByteBuffer().getInt();
                            long tagsCode = bufferCQ.getByteBuffer().getLong();

                            // 延迟消息的tagcode值为消息触发时间
                            if (cq.isExtAddr(tagsCode)) {
                                if (cq.getExt(tagsCode, cqExtUnit)) {
                                    tagsCode = cqExtUnit.getTagsCode();
                                } else {
                                    //can't find ext content.So re compute tags code.
                                    log.error("[BUG] can't find consume queue extend file content!addr={}, offsetPy={}, sizePy={}",
                                        tagsCode, offsetPy, sizePy);
                                    long msgStoreTime = defaultMessageStore.getCommitLog().pickupStoreTimestamp(offsetPy, sizePy);
                                    tagsCode = computeDeliverTimestamp(delayLevel, msgStoreTime);
                                }
                            }

                            long now = System.currentTimeMillis();
                            long deliverTimestamp = this.correctDeliverTimestamp(now, tagsCode);

                            nextOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);

                            long countdown = deliverTimestamp - now;
                            // 达到触发条件
                            if (countdown <= 0) {
                                MessageExt msgExt =
                                    ScheduleMessageService.this.defaultMessageStore.lookMessageByOffset(
                                        offsetPy, sizePy);

                                if (msgExt != null) {
                                    try {
                                        // 还原原始消息
                                        MessageExtBrokerInner msgInner = this.messageTimeup(msgExt);
                                        // put 到commitlog
                                        PutMessageResult putMessageResult =
                                            ScheduleMessageService.this.writeMessageStore
                                                .putMessage(msgInner);

                                        if (putMessageResult != null
                                            && putMessageResult.getPutMessageStatus() == PutMessageStatus.PUT_OK) {
                                            continue;
                                        } else {
                                            // XXX: warn and notify me
                                            log.error(
                                                "ScheduleMessageService, a message time up, but reput it failed, topic: {} msgId {}",
                                                msgExt.getTopic(), msgExt.getMsgId());
                                            ScheduleMessageService.this.timer.schedule(
                                                new DeliverDelayedMessageTimerTask(this.delayLevel,
                                                    nextOffset), DELAY_FOR_A_PERIOD);
                                            ScheduleMessageService.this.updateOffset(this.delayLevel,
                                                nextOffset);
                                            return;
                                        }
                                    } catch (Exception e) {
                                        /*
                                         * XXX: warn and notify me



                                         */
                                        log.error(
                                            "ScheduleMessageService, messageTimeup execute error, drop it. msgExt="
                                                + msgExt + ", nextOffset=" + nextOffset + ",offsetPy="
                                                + offsetPy + ",sizePy=" + sizePy, e);
                                    }
                                }
                            } else {
                                ScheduleMessageService.this.timer.schedule(
                                    new DeliverDelayedMessageTimerTask(this.delayLevel, nextOffset),
                                    countdown);
                                ScheduleMessageService.this.updateOffset(this.delayLevel, nextOffset);
                                return;
                            }
                        } // end of for

                        nextOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);
                        ScheduleMessageService.this.timer.schedule(new DeliverDelayedMessageTimerTask(
                            this.delayLevel, nextOffset), DELAY_FOR_A_WHILE);
                        ScheduleMessageService.this.updateOffset(this.delayLevel, nextOffset);
                        return;
                    } finally {

                        bufferCQ.release();
                    }
                } // end of if (bufferCQ != null)
                else {

                    long cqMinOffset = cq.getMinOffsetInQueue();
                    if (offset < cqMinOffset) {
                        failScheduleOffset = cqMinOffset;
                        log.error("schedule CQ offset invalid. offset=" + offset + ", cqMinOffset="
                            + cqMinOffset + ", queueId=" + cq.getQueueId());
                    }
                }
            } // end of if (cq != null)

            ScheduleMessageService.this.timer.schedule(new DeliverDelayedMessageTimerTask(this.delayLevel,
                failScheduleOffset), DELAY_FOR_A_WHILE);
        }

推介文章:https://cloud.tencent.com/developer/article/1581368

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
RocketMQ延时消息实现是基于timerWheel的算法,主要包括以下几个步骤: 1. 发送延时消息时,将消息的过期时间与当前时间进行比较,如果过期时间比当前时间早,则直接丢弃消息。 2. 如果消息没有过期,则计算该消息应该被放置在timerWheel的哪个槽中,这个槽的位置是通过将消息的过期时间与timerWheel的时间轮进行计算得出的。 3. 将消息放置在相应的槽中,如果该槽中已经存在消息,则将新的消息插入到该槽中的链表的尾部。 4. 启动一个后台线程,定时遍历timerWheel,查找到期的消息,并将这些消息发送到对应的消费者。 以下是RocketMQ延时消息源码实现: 1. 发送延时消息时,将消息的过期时间与当前时间进行比较,如果过期时间比当前时间早,则直接丢弃消息。 ``` public SendResult send(Message msg, long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException { long maxTimeout = System.currentTimeMillis() + timeout; if (msg.getDelayTimeLevel() > 0) { long delayTime = calculateDelayTime(msg.getDelayTimeLevel()); if ((msg.getBornTimestamp() + delayTime) < maxTimeout) { msg.setDelayTimeLevel(0); } } //...省略其他代码 } ``` 2. 如果消息没有过期,则计算该消息应该被放置在timerWheel的哪个槽中,这个槽的位置是通过将消息的过期时间与timerWheel的时间轮进行计算得出的。 ``` private static long calculateDelayTimeLevel(long delayTime) { for (int i = 0; i < DELAY_LEVEL.length; i++) { if (delayTime < DELAY_LEVEL[i]) { return i; } } return DELAY_LEVEL.length - 1; } private static long calculateDelayTime(long level) { return DELAY_LEVEL[(int) Math.min(level, DELAY_LEVEL.length - 1)]; } private long computeDeliverTimestamp(final long startDeliverTime, final long delayTimeLevel) { long offset = DELAY_LEVEL[(int) delayTimeLevel]; return startDeliverTime + offset; } ``` 3. 将消息放置在相应的槽中,如果该槽中已经存在消息,则将新的消息插入到该槽中的链表的尾部。 ``` public void scheduleMessage(MessageExt msg) { long now = System.currentTimeMillis(); long deliverTimestamp = msg.getStoreTimestamp() + msg.getDelayTimeLevel(); long delay = deliverTimestamp - now; if (delay < 0) { delay = 0; } if (delay > config.getMaxDelayTime()) { delay = config.getMaxDelayTime(); } int index = (int) (delay / config.getTickTime()); long expireTime = deliverTimestamp - (deliverTimestamp % config.getTickTime()); try { lock.writeLock().lockInterruptibly(); TimerTaskList taskList = timerWheel[index]; if (taskList == null) { taskList = new TimerTaskList(expireTime); timerWheel[index] = taskList; } taskList.add(msg); } catch (InterruptedException e) { log.error("scheduleMessage error, ", e); } finally { lock.writeLock().unlock(); } } ``` 4. 启动一个后台线程,定时遍历timerWheel,查找到期的消息,并将这些消息发送到对应的消费者。 ``` private class TimerTask implements Runnable { private int index; private long currentTime; public TimerTask(int index, long currentTime) { this.index = index; this.currentTime = currentTime; } @Override public void run() { try { lock.writeLock().lockInterruptibly(); TimerTaskList taskList = timerWheel[index]; if (taskList != null) { List<MessageExt> messages = taskList.remove(currentTime); if (messages != null && !messages.isEmpty()) { putMessagesInQueue(messages); } } } catch (InterruptedException e) { log.error("TimerTask error, ", e); } finally { lock.writeLock().unlock(); } } } private class TimerThread implements Runnable { private volatile boolean stop = false; @Override public void run() { long lastExpireTime = 0; while (!stop) { long currentTime = System.currentTimeMillis(); if (currentTime > lastExpireTime) { for (int i = 0; i < config.getWheelSize(); i++) { long expireTime = lastExpireTime + (i + 1) * config.getTickTime(); if (expireTime <= currentTime) { executor.execute(new TimerTask(i, expireTime)); } } lastExpireTime = currentTime; } else { try { Thread.sleep(1); } catch (InterruptedException e) { log.error("TimerThread error, ", e); } } } } } ```

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值