Rocketmq发送消息原理（含事务消息）

在消息队列中，生产者负责发送消息到broker,本文讲解Rocketmq发送消息的实现原理以及一些注意的事项。

一、生产者端的发送流程

大致步骤

1、根据topic从nameserver或本地获取路由信息，队列信息，broker信息等

2、根据重试次数，循环发送消息

3、消息内容组装成RemotingCommand对象，包括请求头和请求体

4、分oneway,sync,async的方式进行发送

5、如果是async,oneway会获取令牌再发送

6、组装请求头，调用netty组件发送消息

7、结果处理 

二、broker端接收发送消息请求与处理流程 

源码入口

class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {​        //netty监听客户端消息        @Override        protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {            processMessageReceived(ctx, msg);        }    }

发送消息的请求处理器

public class SendMessageProcessor extends AbstractSendMessageProcessor implements NettyRequestProcessor {​    private List<ConsumeMessageHook> consumeMessageHookList;​    public SendMessageProcessor(final BrokerController brokerController) {        super(brokerController);    }​    @Override    public RemotingCommand processRequest(ChannelHandlerContext ctx,                                          RemotingCommand request) throws RemotingCommandException {        SendMessageContext mqtraceContext;        switch (request.getCode()) {        //消费者消费消息 回执            case RequestCode.CONSUMER_SEND_MSG_BACK:                return this.consumerSendMsgBack(ctx, request);                //发送消息请求            default:                //解析请求头                SendMessageRequestHeader requestHeader = parseRequestHeader(request);                if (requestHeader == null) {                    return null;                }                mqtraceContext = buildMsgContext(ctx, requestHeader);                this.executeSendMessageHookBefore(ctx, request, mqtraceContext);                RemotingCommand response;                //发送消息逻辑                if (requestHeader.isBatch()) {                    response = this.sendBatchMessage(ctx, request, mqtraceContext, requestHeader);                } else {                    response = this.sendMessage(ctx, request, mqtraceContext, requestHeader);                }                this.executeSendMessageHookAfter(response, mqtraceContext);                return response;        }    }}

获取消息存储路径

// 用户目录+store+commitlog​private String storePathCommitLog = System.getProperty("user.home") + File.separator + "store"        + File.separator + "commitlog";            // CommitLog file size,default is 1Gprivate int mapedFileSizeCommitLog = 1024 * 1024 * 1024;    ​//路径String nextFilePath = this.storePath + File.separator + UtilAll.offset2FileName(createOffset);            String nextNextFilePath = this.storePath + File.separator                + //commitlog文件名    开始偏移量+文件大小 1g=1073741824         UtilAll.offset2FileName(createOffset + this.mappedFileSize);            MappedFile mappedFile = null;

文件存储

//存储文件中，新建文件,使用nio读写文件private void init(final String fileName, final int fileSize) throws IOException {        this.fileName = fileName;        this.fileSize = fileSize;        this.file = new File(fileName);        this.fileFromOffset = Long.parseLong(this.file.getName());        boolean ok = false;        ensureDirOK(this.file.getParent());        try {            this.fileChannel = new RandomAccessFile(this.file, "rw").getChannel();            this.mappedByteBuffer = this.fileChannel.map(MapMode.READ_WRITE, 0, fileSize);            TOTAL_MAPPED_VIRTUAL_MEMORY.addAndGet(fileSize);            TOTAL_MAPPED_FILES.incrementAndGet();            ok = true;        } catch (FileNotFoundException e) {            log.error("create file channel " + this.fileName + " Failed. ", e);            throw e;        } catch (IOException e) {            log.error("map file " + this.fileName + " Failed. ", e);            throw e;        } finally {            if (!ok && this.fileChannel != null) {                this.fileChannel.close();            }        }    }​//最终调用bytebuffer将消息写入缓冲区,并没有调用刷缓冲到磁盘的方法            messagesByteBuff.position(0);            messagesByteBuff.limit(totalMsgLen);            byteBuffer.put(messagesByteBuff);            messageExtBatch.setEncodedBuff(null);            AppendMessageResult result = new AppendMessageResult(AppendMessageStatus.PUT_OK, wroteOffset, totalMsgLen, msgIdBuilder.toString(),                messageExtBatch.getStoreTimestamp(), beginQueueOffset, CommitLog.this.defaultMessageStore.now() - beginTimeMills);            result.setMsgNum(msgNum);            CommitLog.this.topicQueueTable.put(key, queueOffset);            return result;​

判断刷盘方式，如果是同步刷盘，立即刷新缓冲数据到磁盘

public void handleDiskFlush(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {        // 同步刷盘 Synchronization flush        if (FlushDiskType.SYNC_FLUSH == this.defaultMessageStore.getMessageStoreConfig().getFlushDiskType()) {            final GroupCommitService service = (GroupCommitService) this.flushCommitLogService;            if (messageExt.isWaitStoreMsgOK()) {                GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes());                service.putRequest(request);                //立即刷盘，等待时间是5s                boolean flushOK = request.waitForFlush(this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());                if (!flushOK) {                    log.error("do groupcommit, wait for flush failed, topic: " + messageExt.getTopic() + " tags: " + messageExt.getTags()                        + " client address: " + messageExt.getBornHostString());                    putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_DISK_TIMEOUT);                }            } else {                service.wakeup();            }        }        //异步刷盘 Asynchronous flush        else {            //异步刷盘 默认等待200ms            if (!this.defaultMessageStore.getMessageStoreConfig().isTransientStorePoolEnable()) {                flushCommitLogService.wakeup();            } else {                commitLogService.wakeup();            }        }    }

最终调用

java.nio.MappedByteBuffer#force0 或sun.nio.ch.FileDispatcherImpl#force0 从缓冲写入磁盘

public int flush(final int flushLeastPages) {        if (this.isAbleToFlush(flushLeastPages)) {            if (this.hold()) {                int value = getReadPosition();                try {                    //We only append data to fileChannel or mappedByteBuffer, never both.                    if (writeBuffer != null || this.fileChannel.position() != 0) {                    //刷新到磁盘                                            this.fileChannel.force(false);                    } else {                     //刷新到磁盘                      this.mappedByteBuffer.force();                    }                } catch (Throwable e) {                    log.error("Error occurred when force data to disk.", e);                }                this.flushedPosition.set(value);                this.release();            } else {                log.warn("in flush, hold failed, flush offset = " + this.flushedPosition.get());                this.flushedPosition.set(getReadPosition());            }        }        return this.getFlushedPosition();    }

通过同步树刷盘异步刷盘可用在一定程度上保证消息不丢失，rocketmq还支持集群模式，主从同步模式支持同步或异步，实现数据在多个节点上备份。

//等5s，如果slave未返回，则超时private int syncFlushTimeout = 1000 * 5;public void handleHA(AppendMessageResult result, PutMessageResult putMessageResult, MessageExt messageExt) {            //同步复制到slave            if (BrokerRole.SYNC_MASTER == this.defaultMessageStore.getMessageStoreConfig().getBrokerRole()) {            HAService service = this.defaultMessageStore.getHaService();            if (messageExt.isWaitStoreMsgOK()) {                // Determine whether to wait                if (service.isSlaveOK(result.getWroteOffset() + result.getWroteBytes())) {                    GroupCommitRequest request = new GroupCommitRequest(result.getWroteOffset() + result.getWroteBytes());                    service.putRequest(request);                    service.getWaitNotifyObject().wakeupAll();                    boolean flushOK =                        request.waitForFlush(this.defaultMessageStore.getMessageStoreConfig().getSyncFlushTimeout());                    if (!flushOK) {                        log.error("do sync transfer other node, wait return, but failed, topic: " + messageExt.getTopic() + " tags: "                            + messageExt.getTags() + " client address: " + messageExt.getBornHostNameString());                        putMessageResult.setPutMessageStatus(PutMessageStatus.FLUSH_SLAVE_TIMEOUT);                    }                }                // Slave problem                else {                    // Tell the producer, slave not available                    putMessageResult.setPutMessageStatus(PutMessageStatus.SLAVE_NOT_AVAILABLE);                }            }        }    }

每个broker有三种broker角色，

broker端接收发送消息请求后的总体处理流程图如下 

总的来说包括如下步骤：

1、netty接收到请求，转发到SendMessageProcessor处理器

2、消息头解码

3、判断是是重试消息，并且判断是否达到最大重试次数，如果到了则转换topic和队列，加入死信队列

4、是否是事务消息，转换内置的事务消息topic和queueId

5、不是事务消息，判断是否是延迟消息，是延迟消息，转换成延迟消息topic（SCHEDULE_TOPIC_XXXX）和队列（延迟等级-1，延迟等级从1开始到18）

6、创建或获取消息文件，bytebuffer

7、通过bytebuffer写入缓冲

8、如果是SYNC_FLUSH刷盘方式，立即刷盘 ,刷盘类型有同步和异步两种

public enum FlushDiskType {    SYNC_FLUSH,    ASYNC_FLUSH}

如果是事务消息，流程是怎样的？

在Rocketmq中，事务消息是用来保证本地事务和发送消息逻辑同时成功的一种机制。

事务消息标记存在消息的properties,第一步是将properties解码 

public static final String PROPERTY_TRANSACTION_PREPARED = "TRAN_MSG";

然后将消息发送到一个内置的topic里

private MessageExtBrokerInner parseHalfMessageInner(MessageExtBrokerInner msgInner) {//存储真实topic和队列id到properties        MessageAccessor.putProperty(msgInner, MessageConst.PROPERTY_REAL_TOPIC, msgInner.getTopic());        MessageAccessor.putProperty(msgInner, MessageConst.PROPERTY_REAL_QUEUE_ID,            String.valueOf(msgInner.getQueueId()));        msgInner.setSysFlag(            MessageSysFlag.resetTransactionValue(msgInner.getSysFlag(), MessageSysFlag.TRANSACTION_NOT_TYPE));            //topic 转换成内置事务消息topic 默认RMQ_SYS_TRANS_HALF_TOPIC        msgInner.setTopic(TransactionalMessageUtil.buildHalfTopic());        //默认都是第1个队列        msgInner.setQueueId(0);        msgInner.setPropertiesString(MessageDecoder.messageProperties2String(msgInner.getProperties()));        return msgInner;    }

在客户端发送事务消息 

首先需要实现一个事务消息监听器，实现TransactionListener接口，分别实现本地事务逻辑，检查本地事务状态的逻辑

public class TransactionListenerImpl implements TransactionListener {    private AtomicInteger transactionIndex = new AtomicInteger(0);​    private ConcurrentHashMap<String, Integer> localTrans = new ConcurrentHashMap<>();    //本地事务逻辑    @Override    public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {        int value = transactionIndex.getAndIncrement();        int status = value % 3;        localTrans.put(msg.getTransactionId(), status);        return LocalTransactionState.UNKNOW;    }    //查询本地事务    @Override    public LocalTransactionState checkLocalTransaction(MessageExt msg) {        System.out.println("check msg" + msg.getMsgId() +"===" + LocalDateTime.now().format( DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")));        Integer status = localTrans.get(msg.getTransactionId());        if (null != status) {            switch (status) {                case 0:                    return LocalTransactionState.UNKNOW;                case 1:                    return LocalTransactionState.COMMIT_MESSAGE;                case 2:                    return LocalTransactionState.ROLLBACK_MESSAGE;                default:                    return LocalTransactionState.COMMIT_MESSAGE;            }        }        return LocalTransactionState.COMMIT_MESSAGE;    }}

然后通过事务消息发送器发送消息

public class TransactionProducer {    public static void main(String[] args) throws MQClientException, InterruptedException {        TransactionMQProducer producer = new TransactionMQProducer("please_rename_unique_group_name");        ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000), new ThreadFactory());        producer.setExecutorService(executorService);        //指定事务消息发送监听器        TransactionListener transactionListener = new TransactionListenerImpl();        producer.setTransactionListener(transactionListener);        producer.start();        Message msg =  new Message("test", "TAG", "KEY",                            ("Hello RocketMQ ").getBytes(RemotingHelper.DEFAULT_CHARSET));            SendResult sendResult = producer.sendMessageInTransaction(msg, null);        producer.shutdown();    }}

在发送事务消息流程图如下

事务消息流程总结

1、客户端使用同步发送半消息,broker将半消息存储到内置的事务消息topic和队列，这个时候事务消息不能被消费者消费到。

2、客户端接收发送消息返回，执行本地事务，执行成功，发送本地事务执行结果到broker

3、broker如果发现成功，将半消息转移到真实topic和队列，删除半消息，这个时候事务消息可用被消费者消费，如果回滚，直接删除半消息

4、broker启用一个线程，扫描事务消息topic里的队列里面的消息，判断是否需要检查事务状态(最大检查15次)

5、通过oneway方式向客户端发起查询事务状态请求，客户端查询状态，客户端通过oneway发送事务状态到broker,broker执行第3步骤。 

本文介绍了消息发送和broker处理消息发送请求的实现，得出结论

1、生产者发送消息是会发送到指定的topic队列，默认采用轮训算法实现发送的负载均衡

2、发送消息类型有3种，分别是同步，异步，单次(oneway)，其中同步会重试3次。

3、broker存储消息采用mmap机制，刷盘机制支持同步刷盘和异步刷盘

4、broker主从复制模式支持同步复制

5、事务消息采用内置topic+消息回查机制实现本地事务和发送逻辑的事务一致性。