RocketMQ源码解析——消息存储

目录

• RocketMQ源码解析——搭建源码环境
• RocketMQ源码解析——NameServer
• RocketMQ源码解析——Broker
• RocketMQ源码解析——Producer
• RocketMQ源码解析——消息存储
• RocketMQ源码解析——Consumer

1. 了解RocketMQ中的消息存储：

我们可以将RocketMQ整个发送消息的过程看做以下几个步骤：

1. 消息生产者发送消息，并等待队列返回ACK
2. 消息队列收到消息，并将消息进行持久化存储，在存储中新增一条消息记录
3. 返回ACK给生产者（相当于告知生产者消息已经发送成功）
4. 消息队列push消息给对应的消费者，然后等待消费者返回ACK
5. 如果消息消费者在指定时间内成功返回ACK，那么消息队列就会认为消息消费成功，在存储中删除消息；如果消息队列在指定时间内没有收到ACK，则会认为消息消费失败，就会尝试重新push消息，重复执行4、5步骤

通过以上步骤不难发现，之所以会有这个消息的持久化存储，其实是为了消息消费的高可靠性，在消息消费失败的时候可以将已持久化的消息重新发送

那么在RocketMQ中的消息持久化这个动作是有一个专业的名词：刷盘
其实就是将消息保存到磁盘中，称之为“刷盘”：

• 当消息发送到达到消息队列后，会将该消息首先保存到磁盘缓存中，然后启动刷盘线程，将缓存中的消息保存到磁盘中

而刷盘的方式有两种：

1. 异步刷盘：当刷盘线程启动后，需要等待刷盘线程返回结果后，再唤醒下一个刷盘线程执行
2. 同步刷盘：当刷盘线程启动后，不需要等待刷盘线程返回结果，直接可以唤醒下一个刷盘线程执行

2. 了解RocketMQ的消息存储文件系统：

通过Broker的配置文件查看消息存储文件路径，具体可以看到以下文件夹以及文件：

1. commitLog：消息存储目录
2. config：运行期间的一些配置信息存储目录
3. consumerqueue：消息消费队列存储目录
4. index：消息索引文件存储目录
5. abort：记录消息发送失败信息
6. checkpoint：文件检查点（存储commitLog文件、consumerqueue、index文件最后一次刷盘时间戳）

3. 源码分析：

RocketMQ中的消息存储依赖于核心类——DefaultMessageStore
因此，首先来了解下DefaultMessageStore这个核心类有哪些重要属性：

private final MessageStoreConfig messageStoreConfig; // 消息存储配置

private final CommitLog commitLog; // CommitLog文件存储实现类

private final ConcurrentMap<String/* topic */, ConcurrentMap<Integer/* queueId */, ConsumeQueue>> consumeQueueTable; // 消息队列存储映射表，按照消息主题分组

private final FlushConsumeQueueService flushConsumeQueueService; // 消息队列文件刷盘服务

private final CleanCommitLogService cleanCommitLogService; // 清除CommitLog文件服务

private final CleanConsumeQueueService cleanConsumeQueueService; // 清除消费者队列服务

private final IndexService indexService; // 索引实现类

private final AllocateMappedFileService allocateMappedFileService; // MappedFile分配服务

private final ReputMessageService reputMessageService; // CommitLog消息分发，根据CommitLog文件构建ConsumerQueue、IndexFile文件

private final HAService haService; // 存储HA机制

private final ScheduleMessageService scheduleMessageService; // 消息计划任务服务

private final StoreStatsService storeStatsService; // 消息存储服务

private final TransientStorePool transientStorePool; // 消息堆外内存缓存

private final BrokerStatsManager brokerStatsManager; // Broker状态管理器

private final MessageArrivingListener messageArrivingListener; // 消息拉取长轮询模式消息达到监听器

private final BrokerConfig brokerConfig; // Broker配置类

private StoreCheckpoint storeCheckpoint; // 文件刷盘监测点

private final LinkedList<CommitLogDispatcher> dispatcherList; // CommitLog文件转发请求

对消息存储核心类有了初步的了解，下面开始分析消息存储的代码实现过程。为了能够更好的了解代码实现的细节，先来看下消息存储整体的时序图：

• 消息存储入口：

public PutMessageResult putMessage(MessageExtBrokerInner msg) {
    PutMessageStatus checkStoreStatus = this.checkStoreStatus(); // 检查消息存储状态
    if (checkStoreStatus != PutMessageStatus.PUT_OK) {
        return new PutMessageResult(checkStoreStatus, null);
    }

    PutMessageStatus msgCheckStatus = this.checkMessage(msg); // 检查消息
    if (msgCheckStatus == PutMessageStatus.MESSAGE_ILLEGAL) {
        return new PutMessageResult(msgCheckStatus, null);
    }

    long beginTime = this.getSystemClock().now();
    PutMessageResult result = this.commitLog.putMessage(msg); // 存储消息
    long elapsedTime = this.getSystemClock().now() - beginTime;
    if (elapsedTime > 500) {
        log.warn("not in lock elapsed time(ms)={}, bodyLength={}", elapsedTime, msg.getBody().length);
    }

    this.storeStatsService.setPutMessageEntireTimeMax(elapsedTime);

    if (null == result || !result.isOk()) {
        this.storeStatsService.getPutMessageFailedTimes().incrementAndGet();
    }

    return result;
}

• 检查消息存储状态：

private PutMessageStatus checkStoreStatus() {
    if (this.shutdown) { // 判断是否处于关闭状态
        log.warn("message store has shutdown, so putMessage is forbidden");
        return PutMessageStatus.SERVICE_NOT_AVAILABLE;
    }

    if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) { // 判断接收消息的Broker角色（若为从结点，则不需要进行消息存储）
        long value = this.printTimes.getAndIncrement();
        if ((value % 50000) == 0) {
            log.warn("broke role is slave, so putMessage is forbidden");
        }
        return PutMessageStatus.SERVICE_NOT_AVAILABLE;
    }

    if (!this.runningFlags.isWriteable()) { // 判断当前Broker是否能够允许写入
        long value = this.printTimes.getAndIncrement();
        if ((value % 50000) == 0) {
            log.warn("the message store is not writable. It may be caused by one of the following reasons: the broker's disk is full, write to logic queue error, write to index file error, etc");
        }
        return PutMessageStatus.SERVICE_NOT_AVAILABLE;
    } else {
        this.printTimes.set(0);
    }

    if (this.isOSPageCacheBusy()) { // 判断磁盘缓存空间是否被占用
        return PutMessageStatus.OS_PAGECACHE_BUSY;
    }
    return PutMessageStatus.PUT_OK;
}

• 检查消息：

private PutMessageStatus checkMessage(MessageExtBrokerInner msg) {
    if (msg.getTopic().length() > Byte.MAX_VALUE) { // 判断消息主题长度是否超过最大长度
        log.warn("putMessage message topic length too long " + msg.getTopic().length());
        return PutMessageStatus.MESSAGE_ILLEGAL;
    }

    if (msg.getPropertiesString() != null && msg.getPropertiesString().length() > Short.MAX_VALUE) { // 判断消息属性长度是否超过限制
        log.warn("putMessage message properties length too long " + msg.getPropertiesString().length());
        return PutMessageStatus.MESSAGE_ILLEGAL;
    }
    return PutMessageStatus.PUT_OK;
}

• 存储消息：

public PutMessageResult putMessage(final MessageExtBrokerInner msg) {
    msg.setStoreTimestamp(System.currentTimeMillis()); // 设置消息存储时间戳
    msg.setBodyCRC(UtilAll.crc32(msg.getBody()));
    AppendMessageResult result = null;

    StoreStatsService storeStatsService = this.defaultMessageStore.getStoreStatsService(); // 获取消息存储服务

    String topic = msg.getTopic(); // 获取消息主题
    int queueId = msg.getQueueId(); // 获取消息队列id

    final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
    if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) { 
        // 设置延时级别（应用于延时队列）
        if (msg.getDelayTimeLevel() > 0) {
            if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
                msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
            }

            // 使用延时队列专属的主题和队列id，并备份消息真正的主题和队列id
            topic = TopicValidator.RMQ_SYS_SCHEDULE_TOPIC;
            queueId = ScheduleMessageService.delayLevel2QueueId(msg.getDelayTimeLevel());

            MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_TOPIC, msg.getTopic());
            MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_QUEUE_ID, String.valueOf(msg.getQueueId()));
            msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));

            msg.setTopic(topic);
            msg.setQueueId(queueId);
        }
    }

    InetSocketAddress bornSocketAddress = (InetSocketAddress) msg.getBornHost();
    if (bornSocketAddress.getAddress() instanceof Inet6Address) {
        msg.setBornHostV6Flag();
    }

    InetSocketAddress storeSocketAddress = (InetSocketAddress) msg.getStoreHost();
    if (storeSocketAddress.getAddress() instanceof Inet6Address) {
        msg.setStoreHostAddressV6Flag();
    }

    long elapsedTimeInLock = 0;

    MappedFile unlockMappedFile = null;
    MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile(); // 获取上一次操作的mappedFile

    putMessageLock.lock(); // 加锁，保证消息存储写入的线程安全
    try {
        long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
        this.beginTimeInLock = beginLockTimestamp;

        // Here settings are stored timestamp, in order to ensure an orderly global
        msg.setStoreTimestamp(beginLockTimestamp);

        if (null == mappedFile || mappedFile.isFull()) { // 判断mappedFile为空，或者空间已满
            mappedFile = this.mappedFileQueue.getLastMappedFile(0); // 则创建新的mappedFile对象，并返回
        }
        
        if (null == mappedFile) { // 创建mappedFile失败
            log.error("create mapped file1 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
            beginTimeInLock = 0;
            return new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null);
        }

        result = mappedFile.appendMessage(msg, this.appendMessageCallback); // 将消息写入mappedFile中，并返回结果
        switch (result.getStatus()) {
            case PUT_OK: // 写入消息成功
                break;
            case END_OF_FILE: // 空间不足，无法存储此次消息
                unlockMappedFile = mappedFile;
                // 创建新的mappedFile对象，并重新存储消息
                mappedFile = this.mappedFileQueue.getLastMappedFile(0); 
                if (null == mappedFile) {
                    log.error("create mapped file2 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
                    beginTimeInLock = 0;
                    return new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result);
                }
                result = mappedFile.appendMessage(msg, this.appendMessageCallback);
                break;
            case MESSAGE_SIZE_EXCEEDED: // 消息体积过大
            case PROPERTIES_SIZE_EXCEEDED: // 消息中的属性体积过大
                beginTimeInLock = 0;
                return new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, result);
            case UNKNOWN_ERROR: // 未知错误
                beginTimeInLock = 0;
                return new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result);
            default:
                beginTimeInLock = 0;
                return new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result);
        }

        elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginLockTimestamp;
        beginTimeInLock = 0;
    } finally {
        putMessageLock.unlock(); // 释放锁
    }

    if (elapsedTimeInLock > 500) {
        log.warn("[NOTIFYME]putMessage in lock cost time(ms)={}, bodyLength={} AppendMessageResult={}", elapsedTimeInLock, msg.getBody().length, result);
    }

    if (null != unlockMappedFile && this.defaultMessageStore.getMessageStoreConfig().isWarmMapedFileEnable()) {
        this.defaultMessageStore.unlockMappedFile(unlockMappedFile);
    }

    PutMessageResult putMessageResult = new PutMessageResult(PutMessageStatus.PUT_OK, result);

    // Statistics
    storeStatsService.getSinglePutMessageTopicTimesTotal(msg.getTopic()).incrementAndGet();
    storeStatsService.getSinglePutMessageTopicSizeTotal(topic).addAndGet(result.getWroteBytes());

    handleDiskFlush(result, putMessageResult, msg); // 刷盘
    handleHA(result, putMessageResult, msg);

    return putMessageResult;
}

以上看到的putMessage方法就是消息存储的核心逻辑，其中还是有很多的细节需要展开分析，例如：消息是如何写入mappedFile？消息又是如何刷盘的？等等

为了搞清楚以上问题，需要先来了解什么是mappedFile和mappedFileQueue：

• mappedFile的概念和重点属性：

针对于commitlog、consumequeue、index三类大文件进行磁盘读写操作，均是通过MapedFile类来完成。因此也可以称mappedFile为RocketMQ中的文件操作对象

public static final int OS_PAGE_SIZE = 1024 * 4; // 操作系统每页大小，默认4k

private static final AtomicLong TOTAL_MAPPED_VIRTUAL_MEMORY = new AtomicLong(0); // 当前JVM实例中mappedFile对象所有的虚拟内存大小

private static final AtomicInteger TOTAL_MAPPED_FILES = new AtomicInteger(0); // 当前JVM实例中mappedFile对象的个数

protected final AtomicInteger wrotePosition = new AtomicInteger(0); // 当前文件的写指针

protected final AtomicInteger committedPosition = new AtomicInteger(0); // 当前文件的提交指针
 
private final AtomicInteger flushedPosition = new AtomicInteger(0); // 当前文件刷盘指针

protected int fileSize; // 文件大小

protected FileChannel fileChannel; // 文件通道

protected ByteBuffer writeBuffer = null; // 文件写缓存区

protected TransientStorePool transientStorePool = null; // 堆外内存池

private String fileName; // 文件名

private long fileFromOffset; // 文件的处理偏移量

private File file; // 物理文件对象

private MappedByteBuffer mappedByteBuffer; // 物理文件对应的内容映射缓冲区

private volatile long storeTimestamp = 0; // 文件最后一次写入时间戳

private boolean firstCreateInQueue = false; // 是否是mappedFileQueue队列中第一个文件

• mappedFileQueue的概念和重点属性：

mappedFileQueue其实就是存储mappedFIle的一个队列，存放所有创建的mappedFile对象

private static final int DELETE_FILES_BATCH_MAX = 10;

private final String storePath; // 存储目录

private final int mappedFileSize; // 单个文件大小

private final CopyOnWriteArrayList<MappedFile> mappedFiles = new CopyOnWriteArrayList<MappedFile>(); // 文件集合

private final AllocateMappedFileService allocateMappedFileService; // 创建mappedFile的服务类

private long flushedWhere = 0; // 当前刷盘指正

private long committedWhere = 0; // 当前数据提交指针

private volatile long storeTimestamp = 0; // 存储消息时间戳

• 获取上一次操作的mappedFile：

public MappedFile getLastMappedFile() {
    MappedFile mappedFileLast = null;
    while (!this.mappedFiles.isEmpty()) {
        try {
            mappedFileLast = this.mappedFiles.get(this.mappedFiles.size() - 1); // 获取最新创建的mappedFile对象（也是上一次操作的mappedFile对象）
            break;
        } catch (IndexOutOfBoundsException e) {
            //continue;
        } catch (Exception e) {
            log.error("getLastMappedFile has exception.", e);
            break;
        }
    }
    return mappedFileLast;
}

• 将消息写入mappedFile中，并返回结果：

public AppendMessageResult appendMessage(final MessageExtBrokerInner msg, final AppendMessageCallback cb) {
    return appendMessagesInner(msg, cb);
}

public AppendMessageResult appendMessagesInner(final MessageExt messageExt, final AppendMessageCallback cb) {
    assert messageExt != null;
    assert cb != null;

    int currentPos = this.wrotePosition.get(); // 获取当前mappedFile写入内容位置

    if (currentPos < this.fileSize) { // 写入内容指针位置小于文件大小，说明有剩余空间，允许写入
        ByteBuffer byteBuffer = writeBuffer != null ? writeBuffer.slice() : this.mappedByteBuffer.slice();
        byteBuffer.position(currentPos);
        AppendMessageResult result;
        // 通过回调方法写入消息
        if (messageExt instanceof MessageExtBrokerInner) {
            result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBrokerInner) messageExt);
        } else if (messageExt instanceof MessageExtBatch) {
            result = cb.doAppend(this.getFileFromOffset(), byteBuffer, this.fileSize - currentPos, (MessageExtBatch) messageExt);
        } else {
            return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
        }
        this.wrotePosition.addAndGet(result.getWroteBytes()); // 更新写入内容位置
        this.storeTimestamp = result.getStoreTimestamp(); // 更新消息存储时间戳
        return result; // 返回写入结果
    }
    log.error("MappedFile.appendMessage return null, wrotePosition: {} fileSize: {}", currentPos, this.fileSize);
    return new AppendMessageResult(AppendMessageStatus.UNKNOWN_ERROR);
}

• 写入消息：

public AppendMessageResult doAppend(final long fileFromOffset, final ByteBuffer byteBuffer, final int maxBlank, final MessageExtBrokerInner msgInner) {
    long wroteOffset = fileFromOffset + byteBuffer.position(); // 获取文件写入位置

    int sysflag = msgInner.getSysFlag();
    
    int bornHostLength = (sysflag & MessageSysFlag.BORNHOST_V6_FLAG) == 0 ? 4 + 4 : 16 + 4;
    int storeHostLength = (sysflag & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 : 16 + 4;
    ByteBuffer bornHostHolder = ByteBuffer.allocate(bornHostLength);
    ByteBuffer storeHostHolder = ByteBuffer.allocate(storeHostLength);

    this.resetByteBuffer(storeHostHolder, storeHostLength);
    
    // 生成消息id
    String msgId;
    if ((sysflag & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0) {
        msgId = MessageDecoder.createMessageId(this.msgIdMemory, msgInner.getStoreHostBytes(storeHostHolder), wroteOffset);
    } else {
        msgId = MessageDecoder.createMessageId(this.msgIdV6Memory, msgInner.getStoreHostBytes(storeHostHolder), wroteOffset);
    }

    // 获取消息在队列中的偏移量
    keyBuilder.setLength(0);
    keyBuilder.append(msgInner.getTopic());
    keyBuilder.append('-');
    keyBuilder.append(msgInner.getQueueId());
    String key = keyBuilder.toString();
    Long queueOffset = CommitLog.this.topicQueueTable.get(key);
    if (null == queueOffset) {
        queueOffset = 0L;
        CommitLog.this.topicQueueTable.put(key, queueOffset);
    }

    // 事务类型的消息需要进行特殊处理
    final int tranType = MessageSysFlag.getTransactionValue(msgInner.getSysFlag());
    switch (tranType) {
        // Prepared and Rollback message is not consumed, will not enter the consumer queuec
        case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
        case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
            queueOffset = 0L;
            break;
        case MessageSysFlag.TRANSACTION_NOT_TYPE:
        case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
        default:
            break;
    }
	
	// 序列化消息
    final byte[] propertiesData = msgInner.getPropertiesString() == null ? null : msgInner.getPropertiesString().getBytes(MessageDecoder.CHARSET_UTF8);
    final int propertiesLength = propertiesData == null ? 0 : propertiesData.length;
    if (propertiesLength > Short.MAX_VALUE) {
        log.warn("putMessage message properties length too long. length={}", propertiesData.length);
        return new AppendMessageResult(AppendMessageStatus.PROPERTIES_SIZE_EXCEEDED);
    }
    final byte[] topicData = msgInner.getTopic().getBytes(MessageDecoder.CHARSET_UTF8);
    final int topicLength = topicData.length;
    final int bodyLength = msgInner.getBody() == null ? 0 : msgInner.getBody().length;
    final int msgLen = calMsgLength(msgInner.getSysFlag(), bodyLength, topicLength, propertiesLength);

    if (msgLen > this.maxMessageSize) { // 消息过大
        CommitLog.log.warn("message size exceeded, msg total size: " + msgLen + ", msg body size: " + bodyLength + ", maxMessageSize: " + this.maxMessageSize);
        return new AppendMessageResult(AppendMessageStatus.MESSAGE_SIZE_EXCEEDED);
    }

    if ((msgLen + END_FILE_MIN_BLANK_LENGTH) > maxBlank) { // 文件剩余的空间不足以存储此次消息
        this.resetByteBuffer(this.msgStoreItemMemory, maxBlank);
        // 1 TOTALSIZE
        this.msgStoreItemMemory.putInt(maxBlank);
        // 2 MAGICCODE
        this.msgStoreItemMemory.putInt(CommitLog.BLANK_MAGIC_CODE);
        final long beginTimeMills = CommitLog.this.defaultMessageStore.now();
        byteBuffer.put(this.msgStoreItemMemory.array(), 0, maxBlank);
        return new AppendMessageResult(AppendMessageStatus.END_OF_FILE, wroteOffset, maxBlank, msgId, msgInner.getStoreTimestamp(),
            queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills); // 返回END_OF_FILE结果状态，让上层方法创建新的mappedFile文件，并重新写入消息
    }

	// 文件剩余空间足够存储消息
    // 初始化存储空间，将序列化后的消息相关信息存入msgStoreItemMemory对象
    this.resetByteBuffer(msgStoreItemMemory, msgLen);
    // 1 TOTALSIZE
    this.msgStoreItemMemory.putInt(msgLen);
    // 2 MAGICCODE
    this.msgStoreItemMemory.putInt(CommitLog.MESSAGE_MAGIC_CODE);
    // 3 BODYCRC
    this.msgStoreItemMemory.putInt(msgInner.getBodyCRC());
    // 4 QUEUEID
    this.msgStoreItemMemory.putInt(msgInner.getQueueId());
    // 5 FLAG
    this.msgStoreItemMemory.putInt(msgInner.getFlag());
    // 6 QUEUEOFFSET
    this.msgStoreItemMemory.putLong(queueOffset);
    // 7 PHYSICALOFFSET
    this.msgStoreItemMemory.putLong(fileFromOffset + byteBuffer.position());
    // 8 SYSFLAG
    this.msgStoreItemMemory.putInt(msgInner.getSysFlag());
    // 9 BORNTIMESTAMP
    this.msgStoreItemMemory.putLong(msgInner.getBornTimestamp());
    // 10 BORNHOST
    this.resetByteBuffer(bornHostHolder, bornHostLength);
    this.msgStoreItemMemory.put(msgInner.getBornHostBytes(bornHostHolder));
    // 11 STORETIMESTAMP
    this.msgStoreItemMemory.putLong(msgInner.getStoreTimestamp());
    // 12 STOREHOSTADDRESS
    this.resetByteBuffer(storeHostHolder, storeHostLength);
    this.msgStoreItemMemory.put(msgInner.getStoreHostBytes(storeHostHolder));
    // 13 RECONSUMETIMES
    this.msgStoreItemMemory.putInt(msgInner.getReconsumeTimes());
    // 14 Prepared Transaction Offset
    this.msgStoreItemMemory.putLong(msgInner.getPreparedTransactionOffset());
    // 15 BODY
    this.msgStoreItemMemory.putInt(bodyLength);
    if (bodyLength > 0)
        this.msgStoreItemMemory.put(msgInner.getBody());
    // 16 TOPIC
    this.msgStoreItemMemory.put((byte) topicLength);
    this.msgStoreItemMemory.put(topicData);
    // 17 PROPERTIES
    this.msgStoreItemMemory.putShort((short) propertiesLength);
    if (propertiesLength > 0)
        this.msgStoreItemMemory.put(propertiesData);
        
    final long beginTimeMills = CommitLog.this.defaultMessageStore.now();
    byteBuffer.put(this.msgStoreItemMemory.array(), 0, msgLen); // 将消息写入到缓冲区中
    AppendMessageResult result = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, msgLen, msgId,
        msgInner.getStoreTimestamp(), queueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills); // 创建PUT_OK结果状态

    switch (tranType) {
        case MessageSysFlag.TRANSACTION_PREPARED_TYPE:
        case MessageSysFlag.TRANSACTION_ROLLBACK_TYPE:
            break;
        case MessageSysFlag.TRANSACTION_NOT_TYPE:
        case MessageSysFlag.TRANSACTION_COMMIT_TYPE:
            // The next update ConsumeQueue information
            CommitLog.this.topicQueueTable.put(key, ++queueOffset);
            break;
        default:
            break;
    }
    return result; // 返回PUT_OK结果状态
}

通过以上方法可以看到，当返回PUT_OK的结果状态时候，已经将消息成功写入字节缓冲区。那么下一步便是将缓冲区中的消息刷入磁盘当中，进行持久化存储

• 同步 / 异步刷盘：

public void handleDiskFlush(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {
    // 同步刷盘
    if (FlushDiskType.SYNC_FLUSH == this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {
        final GroupCommitService service = (GroupCommitService) this.flushCommitLogService; // 获取刷盘服务
        if (messageExt.isWaitStoreMsgOK()) {
            GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes()); // 封装刷盘请求
            service.putRequest(request); // 添加刷盘请求
            CompletableFuture<PutMessageStatus> flushOkFuture = request.future();
            PutMessageStatus flushStatus = null;
            try {
                flushStatus = flushOkFuture.get(this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout(), TimeUnit.MILLISECONDS); // 线程阻塞5秒，等待刷盘结束，并返回刷盘结果
            } catch (InterruptedException | ExecutionException | TimeoutException e) {
                //flushOK=false;
            }
            if (flushStatus != PutMessageStatus.PUT_OK) { // 刷盘超时
                log.error("do groupcommit, wait for flush failed, topic: " + messageExt.getTopic() + " tags: " + messageExt.getTags() + " client address: " + messageExt.getBornHostString());
                putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_DISK_TIMEOUT);
            }
        } else {
            service.wakeup();
        }
    } else { // 异步刷盘
        if (!this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {
            flushCommitLogService.wakeup();
        } else {
            commitLogService.wakeup();
        }
    }
}

当程序执行到此，即消息写入缓存，后再刷写进磁盘，消息存储的整个执行过程已经完成。但是还缺少最后一步：删除过期文件

试想，消息确实是持久化到磁盘中了，但是如果没有删除文件的机制，磁盘的空间只会越来越大，而无用数据也只会越存越多。因此，RocketMQ中还拥有一种删除过期文件的机制

而这个机制是通过计划任务的方式进行执行调用，这个计划任务是随着Broker控制器启动而启动

• 启动删除过期文件计划任务：

找到DefaultMessageStore类中的addScheduleTask方法，即添加并启动删除过期文件的计划任务入口

private void addScheduleTask() {
    this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
        @Override
        public void run() {
            DefaultMessageStore.this.cleanFilesPeriodically(); // 删除过期文件
        }
    }, 1000 * 60, this.messageStoreConfig.getCleanResourceInterval(), TimeUnit.MILLISECONDS); // 启动任务，首次执行延时60秒，此后按照规定时间间隔执行
	...
}

• 删除过期文件：

private void cleanFilesPeriodically() {
    this.cleanCommitLogService.run(); // 清除commitLog文件
    this.cleanConsumeQueueService.run(); // 清除消费队列文件
}

• 清除过期commitLog文件：

public void run() {
    try {
        this.deleteExpiredFiles(); // 删除过期文件
        this.redeleteHangedFile();
    } catch (Throwable e) {
        DefaultMessageStore.log.warn(this.getServiceName() + " service has exception. ", e);
    }
}

private void deleteExpiredFiles() {
    int deleteCount = 0; // 删除文件的数量
    long fileReservedTime = DefaultMessageStore.this.getMessageStoreConfig().getFileReservedTime(); // 文件的保留时间
    int deletePhysicFilesInterval = DefaultMessageStore.this.getMessageStoreConfig().getDeleteCommitLogFilesInterval(); // 删除物理文件的间隔
    int destroyMapedFileIntervalForcibly = DefaultMessageStore.this.getMessageStoreConfig().getDestroyMapedFileIntervalForcibly(); // 文件第一次拒绝删除后能保留的最大时间，超过该时间，文件将被强制删除

    boolean timeup = this.isTimeToDelete(); // 是否到达删除时间
    boolean spacefull = this.isSpaceToDelete(); // 磁盘空间是否足够
    boolean manualDelete = this.manualDeleteFileSeveralTimes > 0; // 手动删除

    if (timeup || spacefull || manualDelete) { // 满足以上三个条件之一，则执行删除文件逻辑
        if (manualDelete)
            this.manualDeleteFileSeveralTimes--;

        boolean cleanAtOnce = DefaultMessageStore.this.getMessageStoreConfig().isCleanFileForciblyEnable() && this.cleanImmediately;

        log.info("begin to delete before {} hours file. timeup: {} spacefull: {} manualDeleteFileSeveralTimes: {} cleanAtOnce: {}", fileReservedTime, timeup, spacefull,  manualDeleteFileSeveralTimes, cleanAtOnce);

        fileReservedTime *= 60 * 60 * 1000;

        deleteCount = DefaultMessageStore.this.commitLog.deleteExpiredFile(fileReservedTime, deletePhysicFilesInterval,
            destroyMapedFileIntervalForcibly, cleanAtOnce); // 删除磁盘中的物理文件，并返回删除数量

		// 删除文件数量为0，同时磁盘空间已满，则打印日志（说实话，这if-else写得很拉胯）
        if (deleteCount > 0) {
        
        } else if (spacefull) {
            log.warn("disk space will be full soon, but delete file failed.");
        }
    }
}

public int deleteExpiredFile(final long expiredTime, final int deleteFilesInterval, final long intervalForcibly, final boolean cleanImmediately) {
    return this.mappedFileQueue.deleteExpiredFileByTime(expiredTime, deleteFilesInterval, intervalForcibly, cleanImmediately); // 通过mappedFileQueue删除其中满足条件的mappedFile文件
}

public int deleteExpiredFileByTime(final long expiredTime, final int deleteFilesInterval, final long intervalForcibly, final boolean cleanImmediately) {
    Object[] mfs = this.copyMappedFiles(0); // 以mappedFileQueue为源，复制一份包含当前所有mappedFile对象新的数组
    if (null == mfs)
        return 0;
    int mfsLength = mfs.length - 1;
    int deleteCount = 0; // 删除文件数量
    List<MappedFile> files = new ArrayList<MappedFile>(); // 待删除文件集合
    if (null != mfs) {
        for (int i = 0; i < mfsLength; i++) { // 遍历mappedFile对象
            MappedFile mappedFile = (MappedFile) mfs[i];
            long liveMaxTimestamp = mappedFile.getLastModifiedTimestamp() + expiredTime; // 最新一次修改（写入消息）的时间 + 最大过期时间 = 当前文件最大存活时间
            if (System.currentTimeMillis() >= liveMaxTimestamp || cleanImmediately) { //  当前mappedFile对象已超过文件最大存活时间，或者标识为立即删除
                if (mappedFile.destroy(intervalForcibly)) { // 物理删除文件
                    files.add(mappedFile); // 将其存入待删除文件集合
                    deleteCount++; // 删除文件数量+1
                    if (files.size() >= DELETE_FILES_BATCH_MAX) { // 待删除文件集合长度 >= 一次批量删除文件最大数量，则退出循环（一次最多只能批量删除指定的最大删除数量文件）
                        break;
                    }
                    if (deleteFilesInterval > 0 && (i + 1) < mfsLength) { 
                        try {
                            Thread.sleep(deleteFilesInterval);
                        } catch (InterruptedException e) {
                        }
                    }
                } else {
                    break;
                }
            } else {
                //avoid deleting files in the middle
                break;
            }
        }
    }
    deleteExpiredFile(files); // 逻辑删除文件
    return deleteCount; // 返回删除文件数量
}

• 物理删除文件：

public boolean destroy(final long intervalForcibly) {
    this.shutdown(intervalForcibly);
    if (this.isCleanupOver()) {
        try {
            this.fileChannel.close(); // 关闭文件操作通道
            log.info("close file channel " + this.fileName + " OK");
            long beginTime = System.currentTimeMillis();
            boolean result = this.file.delete(); // 删除磁盘中的文件
        } catch (Exception e) {
            log.warn("close file channel " + this.fileName + " Failed. ", e);
        }
        return true;
    } else {
        log.warn("destroy mapped file[REF:" + this.getRefCount() + "] " + this.fileName + " Failed. cleanupOver: " + this.cleanupOver);
    }
    return false;
}

• 逻辑删除文件：

void deleteExpiredFile(List<MappedFile> files) {
    if (!files.isEmpty()) {
        Iterator<MappedFile> iterator = files.iterator();
        while (iterator.hasNext()) {
            MappedFile cur = iterator.next();
            if (!this.mappedFiles.contains(cur)) {
                iterator.remove();
                log.info("This mappedFile {} is not contained by mappedFiles, so skip it.", cur.getFileName());
            }
        }

        try {
            if (!this.mappedFiles.removeAll(files)) { // 删除mappedFile对象
                log.error("deleteExpiredFile remove failed.");
            }
        } catch (Exception e) {
            log.error("deleteExpiredFile has exception.", e);
        }
    }
}

到此，消息存储的核心源码解析结束