RocketMq源码分析（四）--消息存储服务MessageStore

一、消息存储实现

1、消息存储实现类

  消息存储实现类为：org.apache.rocketmq.store.DefaultMessageStore，在borker启动创建BrokerController实例时，会创建DefaultMessageStore用于消息存储。
  DefaultMessageStore构造器如下：

public DefaultMessageStore(final MessageStoreConfig messageStoreConfig, final BrokerStatsManager brokerStatsManager,
        final MessageArrivingListener messageArrivingListener, final BrokerConfig brokerConfig) throws IOException {
        //消息拉取长轮询模式消息到达监听器
        this.messageArrivingListener = messageArrivingListener;
        //broker配置
        this.brokerConfig = brokerConfig;
        //消息存储配置
        this.messageStoreConfig = messageStoreConfig;
        //broker状态管理器
        this.brokerStatsManager = brokerStatsManager;
        //MappedFile分配服务
        this.allocateMappedFileService = new AllocateMappedFileService(this);
        //初始化commitLog存储实现
        if (messageStoreConfig.isEnableDLegerCommitLog()) {
            this.commitLog = new DLedgerCommitLog(this);
        } else {
            this.commitLog = new CommitLog(this);
        }
        //cqTable
        this.consumeQueueTable = new ConcurrentHashMap<>(32);
        //cq队列文件刷盘服务
        this.flushConsumeQueueService = new FlushConsumeQueueService();
        //清除commitLog服务
        this.cleanCommitLogService = new CleanCommitLogService();
        //清除cq队列文件服务
        this.cleanConsumeQueueService = new CleanConsumeQueueService();

        this.storeStatsService = new StoreStatsService();
        //索引文件服务
        this.indexService = new IndexService(this);
        //HA机制服务
        if (!messageStoreConfig.isEnableDLegerCommitLog()) {
            this.haService = new HAService(this);
        } else {
            this.haService = null;
        }
        //commitLog消息分发，cq、indexFile文件构建
        this.reputMessageService = new ReputMessageService();

        this.scheduleMessageService = new ScheduleMessageService(this);
        //消息堆内存缓存
        this.transientStorePool = new TransientStorePool(messageStoreConfig);

        if (messageStoreConfig.isTransientStorePoolEnable()) {
            this.transientStorePool.init();
        }

        this.allocateMappedFileService.start();

        this.indexService.start();

        this.dispatcherList = new LinkedList<>();
        this.dispatcherList.addLast(new CommitLogDispatcherBuildConsumeQueue());
        this.dispatcherList.addLast(new CommitLogDispatcherBuildIndex());

        File file = new File(StorePathConfigHelper.getLockFile(messageStoreConfig.getStorePathRootDir()));
        MappedFile.ensureDirOK(file.getParent());
        MappedFile.ensureDirOK(getStorePathPhysic());
        MappedFile.ensureDirOK(getStorePathLogic());
        lockFile = new RandomAccessFile(file, "rw");
    }

  其主要属性为：

• MessageStoreConfig messageStoreConfig：消息存储服务配置类
• CommitLog commitLog：commitLog文件对应实现类
• ConcurrentMap<String/* topic /, ConcurrentMap<Integer/ queueId */, ConsumeQueue>> consumeQueueTable：消息队列缓存map，根据topic进行分组
• FlushConsumeQueueService flushConsumeQueueService：消息队列刷盘服务实现类
• CleanCommitLogService cleanCommitLogService：清除commitLog文件服务
• CleanConsumeQueueService cleanConsumeQueueService：清除消息队列服务
• IndexService indexService：索引服务
• AllocateMappedFileService allocateMappedFileService：MappedFile分配服务
• ReputMessageService reputMessageService：commitLog消息分发服务
• HAService haService：HA机制服务
• TransientStorePool transientStorePool：消息内存缓存
• MessageArrivingListener messageArrivingListener：消息拉取长轮询模式消息到达监听器
• BrokerConfig brokerConfig：broker配置类
• StoreCheckpoint storeCheckpoint：文件刷盘检测点

二、消息存储

1、消息存储入口

  消息存储入口位于org.apache.rocketmq.store.DefaultMessageStore#putMessage

    @Override
    public PutMessageResult putMessage(MessageExtBrokerInner msg) {
        return waitForPutResult(asyncPutMessage(msg));
    }

  其核心是调用异步asyncPutMessage方法实现消息保存，然后waitForPutResult阻塞等待结果

2、asyncPutMessage保存

 @Override
    public CompletableFuture<PutMessageResult> asyncPutMessage(MessageExtBrokerInner msg) {
        //检查是否可写消息
        PutMessageStatus checkStoreStatus = this.checkStoreStatus();
        if (checkStoreStatus != PutMessageStatus.PUT_OK) {
            return CompletableFuture.completedFuture(new PutMessageResult(checkStoreStatus, null));
        }
        //检查topic长度、消息长度
        PutMessageStatus msgCheckStatus = this.checkMessage(msg);
        if (msgCheckStatus == PutMessageStatus.MESSAGE_ILLEGAL) {
            return CompletableFuture.completedFuture(new PutMessageResult(msgCheckStatus, null));
        }
        //LMQ消息检查
        PutMessageStatus lmqMsgCheckStatus = this.checkLmqMessage(msg);
        if (msgCheckStatus == PutMessageStatus.LMQ_CONSUME_QUEUE_NUM_EXCEEDED) {
            return CompletableFuture.completedFuture(new PutMessageResult(lmqMsgCheckStatus, null));
        }


        long beginTime = this.getSystemClock().now();
        //消息保存
        CompletableFuture<PutMessageResult> putResultFuture = this.commitLog.asyncPutMessage(msg);

        putResultFuture.thenAccept(result -> {
            long elapsedTime = this.getSystemClock().now() - beginTime;
            if (elapsedTime > 500) {
                log.warn("putMessage not in lock elapsed time(ms)={}, bodyLength={}", elapsedTime, msg.getBody().length);
            }
            this.storeStatsService.setPutMessageEntireTimeMax(elapsedTime);

            if (null == result || !result.isOk()) {
                //消息保存失败后，失败数加1
                this.storeStatsService.getPutMessageFailedTimes().add(1);
            }
        });

        return putResultFuture;
    }

  asyncPutMessage方法主要分三步：

• 前置检查，检查当前broker可写状态，检查消息合法性
• 调用commitLog.asyncPutMessage(msg)保存消息
• 后置处理，记录保存消息最大时间消耗、消息保存失败次数

1）前置检查

检查broker可写状态

  检查broker状态逻辑在org.apache.rocketmq.store.DefaultMessageStore#checkStoreStatus方法，代码如下：

 private PutMessageStatus checkStoreStatus() {
        if (this.shutdown) {
            //broker实例停止，拒绝
            log.warn("message store has shutdown, so putMessage is forbidden");
            return PutMessageStatus.SERVICE_NOT_AVAILABLE;
        }

        if (BrokerRole.SLAVE == this.messageStoreConfig.getBrokerRole()) {
            //slave拒绝写入
            long value = this.printTimes.getAndIncrement();
            if ((value % 50000) == 0) {
                log.warn("broke role is slave, so putMessage is forbidden");
            }
            return PutMessageStatus.SERVICE_NOT_AVAILABLE;
        }

        if (!this.runningFlags.isWriteable()) {
            //broker不支持写入，可能磁盘满了/写consumeQueue错误/写IndexFile等原因，拒绝写入
            long value = this.printTimes.getAndIncrement();
            if ((value % 50000) == 0) {
                log.warn("the message store is not writable. It may be caused by one of the following reasons: " +
                    "the broker's disk is full, write to logic queue error, write to index file error, etc");
            }
            return PutMessageStatus.SERVICE_NOT_AVAILABLE;
        } else {
            this.printTimes.set(0);
        }

        if (this.isOSPageCacheBusy()) {
            //操作系统PageCache繁忙
            return PutMessageStatus.OS_PAGECACHE_BUSY;
        }
        return PutMessageStatus.PUT_OK;
    }

  分析上述检查broker状态代码，broker拒绝写入消息有以下几种情况：

• broker处于停止状态。broker在停止时，会通知MessageStore将shutdown标识设置为true
• broker是slave角色
• broker不支持写入，可能是磁盘满了、写consumeQueue消息队列错误或是写IndexFile错误等原因
• 操作系统PageCache繁忙

检查消息合法性

  检查消息合法性，如果消息topic长度超过127或是消息属性长度超过32767则拒绝写入

    private PutMessageStatus checkMessage(MessageExtBrokerInner msg) {
        if (msg.getTopic().length() > Byte.MAX_VALUE) {
            //topic长度不能超过127
            log.warn("putMessage message topic length too long " + msg.getTopic().length());
            return PutMessageStatus.MESSAGE_ILLEGAL;
        }

        if (msg.getPropertiesString() != null && msg.getPropertiesString().length() > Short.MAX_VALUE) {
            //消息属性长度不能超过32767，2^15 -1
            log.warn("putMessage message properties length too long " + msg.getPropertiesString().length());
            return PutMessageStatus.MESSAGE_ILLEGAL;
        }
        return PutMessageStatus.PUT_OK;
    }

LMQ检查

  如果开启light mq功能，超过Lmq队列数量则拒绝写入

    private PutMessageStatus checkLmqMessage(MessageExtBrokerInner msg) {
        if (msg.getProperties() != null
            && StringUtils.isNotBlank(msg.getProperty(MessageConst.PROPERTY_INNER_MULTI_DISPATCH))
            && this.isLmqConsumeQueueNumExceeded()) {
            return PutMessageStatus.LMQ_CONSUME_QUEUE_NUM_EXCEEDED;
        }
        return PutMessageStatus.PUT_OK;
    }

2）调用commitLog.asyncPutMessage保存

消息CRC校验码

  在保存消息内容之前，先将消息内容计算出一个CRC校验码，用于后续消息内容校验

    public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
        // Set the storage time
        msg.setStoreTimestamp(System.currentTimeMillis());
        // Set the message body BODY CRC (consider the most appropriate setting
        // on the client)
        msg.setBodyCRC(UtilAll.crc32(msg.getBody()));
        // ...
     }   

延时消息处理

  对于延时消息，把消息原始topic及queueId存入到消息的属性中，然后将当前topic、queueId替换为延时消息固定的topic（SCHEDULE_TOPIC_XXXX）以及根据延时级别计算出来的queueId

    public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
        // ...
        
        String topic = msg.getTopic();
//        int queueId msg.getQueueId();
        final int tranType = MessageSysFlag.getTransactionValue(msg.getSysFlag());
        if (tranType == MessageSysFlag.TRANSACTION_NOT_TYPE
                || tranType == MessageSysFlag.TRANSACTION_COMMIT_TYPE) {
            // Delay Delivery
            if (msg.getDelayTimeLevel() > 0) {
                //延时消息

                if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
                    msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
                }
                //先用用延时消息的topic（SCHEDULE_TOPIC_XXXX）、对应延时级别的queueId代替
                topic = TopicValidator.RMQ_SYS_SCHEDULE_TOPIC;
                int queueId = ScheduleMessageService.delayLevel2QueueId(msg.getDelayTimeLevel());

                // Backup real topic, queueId
                MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_TOPIC, msg.getTopic());
                MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_QUEUE_ID, String.valueOf(msg.getQueueId()));
                msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));

                msg.setTopic(topic);
                msg.setQueueId(queueId);
            }
        }
        // ...
     } 

创建PutMessageContext

  创建PutMessageContext对象，该对象中topicQueueTableKey属性值为generateKey(putMessageThreadLocal.getKeyBuilder(), msg)方法生成的结果，这个值规则为 topic+"-"+queueId组成

    public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
        // ...
        PutMessageThreadLocal putMessageThreadLocal = this.putMessageThreadLocal.get();
        if (!multiDispatch.isMultiDispatchMsg(msg)) {
            PutMessageResult encodeResult = putMessageThreadLocal.getEncoder().encode(msg);
            if (encodeResult != null) {
                //encodeResult不为null标识消息不合法
                return CompletableFuture.completedFuture(encodeResult);
            }
            msg.setEncodedBuff(putMessageThreadLocal.getEncoder().getEncoderBuffer());
        }
        //创建topicQueueTableKey = topic+"-"+queueId
        PutMessageContext putMessageContext = new PutMessageContext(generateKey(putMessageThreadLocal.getKeyBuilder(), msg));
        // ...
     } 

消息添加到内存commitLog对象中

  在加锁完成后，进入消息内容保存阶段
  在保存阶段过程中，先从mappedFileQueue获取最后一个commitLog对象，即mappedFile对象；然后调用mappedFile.appendMessage方法把消息添加到mappedFile对象中。如果当前mappedFile剩余可添加消息空间大小不足以添加当前消息，这个时候会返回END_OF_FILE结果，此时会重新创建一个mappedFile并再次执行消息添加操作

    public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
        // ...
        //加锁
        putMessageLock.lock(); //spin or ReentrantLock ,depending on store config
        try {
            //从mappedFileQueue中获取最后一个commitLog对象
            MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();
            long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now();
            this.beginTimeInLock = beginLockTimestamp;

            // Here settings are stored timestamp, in order to ensure an orderly
            // global
            msg.setStoreTimestamp(beginLockTimestamp);

            if (null == mappedFile || mappedFile.isFull()) {
                //当前没有commitLog或commitLog已满，重新创建一个
                mappedFile = this.mappedFileQueue.getLastMappedFile(0); // Mark: NewFile may be cause noise
            }
            if (null == mappedFile) {
                log.error("create mapped file1 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
                return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, null));
            }
            //当前commitLog对象添加消息，添加消息还在内存对象中
            result = mappedFile.appendMessage(msg, this.appendMessageCallback, putMessageContext);
            switch (result.getStatus()) {
                case PUT_OK:
                    break;
                case END_OF_FILE:
                    unlockMappedFile = mappedFile;
                    // Create a new file, re-write the message
                    mappedFile = this.mappedFileQueue.getLastMappedFile(0);
                    if (null == mappedFile) {
                        // XXX: warn and notify me
                        log.error("create mapped file2 error, topic: " + msg.getTopic() + " clientAddr: " + msg.getBornHostString());
                        return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.CREATE_MAPEDFILE_FAILED, result));
                    }
                    result = mappedFile.appendMessage(msg, this.appendMessageCallback, putMessageContext);
                    break;
                case MESSAGE_SIZE_EXCEEDED:
                case PROPERTIES_SIZE_EXCEEDED:
                    return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, result));
                case UNKNOWN_ERROR:
                    return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
                default:
                    return CompletableFuture.completedFuture(new PutMessageResult(PutMessageStatus.UNKNOWN_ERROR, result));
            }

            elapsedTimeInLock = this.defaultMessageStore.getSystemClock().now() - beginLockTimestamp;
        } finally {
            beginTimeInLock = 0;
            putMessageLock.unlock();
        }
        // ...
     } 

执行刷盘

  在调用mappedFile.appendMessage方法保存消息之后，消息此时还只是保存在内存中，并未真正保存到commitLog文件中
  所以在保存消息的最后，需要提交一次刷盘，然后根据消息的持久化策略，选择进行同步刷盘还是异步刷盘，将内存中的消息数据持久化到磁盘文件中
  在提交刷盘请求后，执行主从同步复制

    public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBrokerInner msg) {
        // ...
        //提交刷盘
        CompletableFuture<PutMessageStatus> flushResultFuture = submitFlushRequest(result, msg);
        //HA机制同步
        CompletableFuture<PutMessageStatus> replicaResultFuture = submitReplicaRequest(result, msg);
        return flushResultFuture.thenCombine(replicaResultFuture, (flushStatus, replicaStatus) -> {
            if (flushStatus != PutMessageStatus.PUT_OK) {
                putMessageResult.setPutMessageStatus(flushStatus);
            }
            if (replicaStatus != PutMessageStatus.PUT_OK) {
                putMessageResult.setPutMessageStatus(replicaStatus);
            }
            return putMessageResult;
        });
     } 

3）后置处理

  commitLog.asyncPutMessage保存消息方法，会返回一个CompletableFuture<PutMessageResult>异步结果对象。在消息保存的基础上，增加MessageStore增加了thenAccept后置处理，该部分代码在执行完commitLog.asyncPutMessage后才会执行。后置处理主要干了2个事情：一个是记录本次保存消息的时间消耗；一个是将记录消息保存失败的次数

        putResultFuture.thenAccept(result -> {
            long elapsedTime = this.getSystemClock().now() - beginTime;
            if (elapsedTime > 500) {
                log.warn("putMessage not in lock elapsed time(ms)={}, bodyLength={}", elapsedTime, msg.getBody().length);
            }
            this.storeStatsService.setPutMessageEntireTimeMax(elapsedTime);

            if (null == result || !result.isOk()) {
                //消息保存失败后，失败数加1
                this.storeStatsService.getPutMessageFailedTimes().add(1);
            }
        });