rocketmq核心源码分析第二十四篇一延时消息

rocketmq采用了固定级别延时消息实现
不同于时间轮算法 其延时是固定18个级别的离散性实现,无法做到任意级别连续延迟

原理图

延迟级别

messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h

• msg.setDelayTimeLevel()设置消息发送的延迟级别
• 每一个延迟级别对应一个延迟时间
• broker会将延迟消息的consumequeue转存储到主题名为SCHEDULE_TOPIC_XXXX的消息队列
• SCHEDULE_TOPIC_XXXX为rocketMq内部定时主题
• 该主题含有18个消费队列,对应18个消息延迟级别,延迟时间为1s至2h
• 每一个messagequeue由一个ScheduleMessageService.DeliverDelayedMessageTimerTask管理
• DeliverDelayedMessageTimerTask负责当延迟时间到达后将消息转储到的topic
  

源码分析一ScheduleMessageService

• 对18个延迟队列构建18个DeliverDelayedMessageTimerTask
• 启动调度执行DeliverDelayedMessageTimerTask
• 启动调度持久化消费进度

public class ScheduleMessageService extends ConfigManager {
 	public void start() {
	    if (started.compareAndSet(false, true)) {
	        this.timer = new Timer("ScheduleMessageTimerThread", true);
	        对18个延迟队列构建18个DeliverDelayedMessageTimerTask
	        for (Map.Entry<Integer, Long> entry : this.delayLevelTable.entrySet()) {
	            Integer level = entry.getKey();
	            Long timeDelay = entry.getValue();
	            消费进度
	            Long offset = this.offsetTable.get(level);
	            if (null == offset) {
	                offset = 0L;
	            }

	            if (timeDelay != null) {
	                1秒后开始调度
	                this.timer.schedule(new DeliverDelayedMessageTimerTask(level, offset), FIRST_DELAY_TIME);
	            }
	        }
	        持久化消费进度
	        this.timer.scheduleAtFixedRate(new TimerTask() {

	            @Override
	            public void run() {
	                try {
	                    if (started.get()) ScheduleMessageService.this.persist();
	                } catch (Throwable e) {
	                    log.error("scheduleAtFixedRate flush exception", e);
	                }
	            }
	        }, 10000, this.defaultMessageStore.getMessageStoreConfig().getFlushDelayOffsetInterval());
	    }
	}
}

源码分析一DeliverDelayedMessageTimerTask

• 获取延时主题的指定level对应的consumequeue
• 获取consumequeue信息
• 根据tag算出消费时间
• 消费时间未到达则构建新的task在指定时间后消费
• 否则获取消息体并写入真正的topic

/**
 * 延时消息 被scheduleMessageService处理
 * 默认18个延时task都在处理
 */
class DeliverDelayedMessageTimerTask extends TimerTask {
    private final int delayLevel;
    初始消费进度为0
    private final long offset;

    @Override
    public void run() {
     	...... 删除其他代码 
        if (isStarted()) {
            执行延时任务
            this.executeOnTimeup();
        }
        
    }


    public void executeOnTimeup() {
        获取延时主题的指定level对应的consumequeue
        ConsumeQueue cq =
            ScheduleMessageService.this.defaultMessageStore.findConsumeQueue(SCHEDULE_TOPIC,
                delayLevel2QueueId(delayLevel));

        long failScheduleOffset = offset;

        if (cq != null) {
            获取consumequeue信息
            SelectMappedBufferResult bufferCQ = cq.getIndexBuffer(this.offset);
            if (bufferCQ != null) {
                long nextOffset = offset;
                int i = 0;
                ConsumeQueueExt.CqExtUnit cqExtUnit = new ConsumeQueueExt.CqExtUnit();
                for (; i < bufferCQ.getSize(); i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
                    获取20字节索引信息
                    long offsetPy = bufferCQ.getByteBuffer().getLong();
                    int sizePy = bufferCQ.getByteBuffer().getInt();
                    long tagsCode = bufferCQ.getByteBuffer().getLong();
                    ...... 删除其他代码 
                    long now = System.currentTimeMillis();
                    tagsCode已经被改写成重试消息应当被消费的时间戳  此处进行一次校准
                    long deliverTimestamp = this.correctDeliverTimestamp(now, tagsCode);
                    nextOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);

                    算出消费应该被消费还需要多久
                    long countdown = deliverTimestamp - now;
                    if (countdown <= 0) { 说明应该被投递
                        通过commitlog获取定时消息对应的消息体
                        MessageExt msgExt =
                            ScheduleMessageService.this.defaultMessageStore.lookMessageByOffset(
                                offsetPy, sizePy);
                        if (msgExt != null) {
                            根据定时消息处理成新的消息 发回真正的主题 【retry topic或者业务topic】
                              消息体转换
                            MessageExtBrokerInner msgInner = this.messageTimeup(msgExt);
                            if (MixAll.RMQ_SYS_TRANS_HALF_TOPIC.equals(msgInner.getTopic())) {
                                log.error("[BUG] the real topic of schedule msg is {}, discard the msg. msg={}",
                                        msgInner.getTopic(), msgInner);
                                continue;
                            }
                              写回真正的topic
                            PutMessageResult putMessageResult =
                                ScheduleMessageService.this.writeMessageStore
                                    .putMessage(msgInner);
                           ...... 删除其他代码 
                        }
                    } else {
                    	消息未到达消费时间 则设置一定countdown时间后消费
                        ScheduleMessageService.this.timer.schedule(
                            new DeliverDelayedMessageTimerTask(this.delayLevel, nextOffset),
                            countdown);
                        ScheduleMessageService.this.updateOffset(this.delayLevel, nextOffset);
                        return;
                    }
                } 
                正常消费DELAY_FOR_A_WHILE100毫秒 再次消费
                nextOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);
                ScheduleMessageService.this.timer.schedule(new DeliverDelayedMessageTimerTask(
                    this.delayLevel, nextOffset), DELAY_FOR_A_WHILE);
                ScheduleMessageService.this.updateOffset(this.delayLevel, nextOffset);
                return;
            } 
            ...... 删除其他代码 
        } 
        ScheduleMessageService.this.timer.schedule(new DeliverDelayedMessageTimerTask(this.delayLevel,
            每100毫秒一次
            failScheduleOffset), DELAY_FOR_A_WHILE);
    }
    定时topic主题消息转原topic主题消息
    private MessageExtBrokerInner messageTimeup(MessageExt msgExt) {
    	...... 删除赋值代码
       	通过property属性完成原topic的存储提取与转换
        MessageAccessor.clearProperty(msgInner, MessageConst.PROPERTY_DELAY_TIME_LEVEL);
        msgInner.setTopic(msgInner.getProperty(MessageConst.PROPERTY_REAL_TOPIC));
        return msgInner;
    }
}

总结

• 延时消息主要通过延时级别和内置主题及任务处理
• 延迟主题和原主题的转换主要通过消息property属性完成转换

扩展点一topic转换

• 延时消息或者事务消息等等,其消息并非直接发送至业务定义topic,而是发往中间topic
• 中间topic的消息在处理时需要转换后发送至原topic
• topic的转换借助msg的附件属性property完成