RocketMQ-Request-Reply特性

源码版本号:版本号:4.9.4

使用场景

随着服务规模的扩大，单机服务无法满足性能和容量的要求，此时需要将服务拆分为更小粒度的服务或者部署多个服务实例构成集群来提供服务。在分布式场景下，RPC是最常用的联机调用的方式。

在构建分布式应用时，有些领域，例如金融服务领域，常常使用消息队列来构建服务总线，实现联机调用的目的。消息队列的主要场景是解耦、削峰填谷，在联机调用的场景下，需要将服务的调用抽象成基于消息的交互，并增强同步调用的这种交互逻辑。为了更好地支持消息队列在联机调用场景下的应用，rocketmq-4.6.0推出了“Request-Reply”特性来支持RPC调用。

设计思路

在rocketmq中，整个同步调用主要包括两个过程：

（1）请求方生成消息，发送给响应方，并等待响应方回包；

（2）响应方收到请求消息后，消费这条消息，并发出一条响应消息给请求方。

整个过程实质上是两个消息收发过程的组合。所以这里最关键的问题是如何将异步的消息收发过程构建成一个同步的过程。其中主要有两个问题需要解决：

请求方如何同步等待回包

这个问题的解决方案中，一个关键的数据结构是RequestResponseFuture。

public class RequestResponseFuture {
    private final String correlationId;
    private final RequestCallback requestCallback;
    private final long beginTimestamp = System.currentTimeMillis();
    private final Message requestMsg = null;
    private long timeoutMillis;
    private CountDownLatch countDownLatch = new CountDownLatch(1);
    private volatile Message responseMsg = null;
    private volatile boolean sendRequestOk = true;
    private volatile Throwable cause = null;

    // 通过CountDownLatch机制实现同步等待
    public Message waitResponseMessage(final long timeout) throws InterruptedException {
        this.countDownLatch.await(timeout, TimeUnit.MILLISECONDS);
        return this.responseMsg;
    }
    
    public void putResponseMessage(final Message responseMsg) {
        this.responseMsg = responseMsg;
        this.countDownLatch.countDown();
    }
}

RequestResponseFuture中，利用correlationId来标识一个请求。如下图所示，Producer发送request时创建一个RequestResponseFuture，以correlationId为key，RequestResponseFuture为value存入map，同时请求中带上RequestResponseFuture中的correlationId，收到回包后根据correlationId拿到对应的RequestResponseFuture，并设置回包内容。

consumer消费消息后，如何准确回包

（1）producer在发送消息的时候，会给每条消息生成唯一的标识符，同时还带上了producer的clientId。当consumer收到并消费消息后，从消息中取出消息的标识符correlationId和producer的标识符clientId，放入响应消息，用来确定此响应消息是哪条请求消息的回包，以及此响应消息应该发给哪个producer。同时响应消息中设置了消息的类型以及响应消息的topic，然后consumer将消息发给broker，如下图所示。

（2）broker收到响应消息后，需要将消息发回给指定的producer。Broker如何知道发回给哪个producer？因为消息中包含了producer的标识符clientId，在ProducerManager中，维护了标识符和channel信息的对应关系，通过这个对应关系，就能把回包发给对应的producer。

响应消息发送和一般的消息发送流程区别在于，响应消息不需要producer拉取，而是由broker直接推给producer。同时选择broker的策略也有变化：请求消息从哪个broker发过来，响应消息也发到对应的broker上。

Producer收到响应消息后，根据消息中的唯一标识符，从RequestResponseFuture的map中找到对应的RequestResponseFuture结构，设置响应消息，同时计数器减一，解除等待状态，使请求方收到响应消息。

broker回调到发送者的代码如下

// client包
// org.apache.rocketmq.client.impl.ClientRemotingProcessor
public class ClientRemotingProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
    private void processReplyMessage(MessageExt replyMsg) {
        final String correlationId = replyMsg.getUserProperty(MessageConst.PROPERTY_CORRELATION_ID);
        final RequestResponseFuture requestResponseFuture = RequestFutureHolder.getInstance().getRequestFutureTable().get(correlationId);
        if (requestResponseFuture != null) {
            requestResponseFuture.putResponseMessage(replyMsg);

            RequestFutureHolder.getInstance().getRequestFutureTable().remove(correlationId);

            if (requestResponseFuture.getRequestCallback() != null) {
                requestResponseFuture.getRequestCallback().onSuccess(replyMsg);
            }
        } else {
            String bornHost = replyMsg.getBornHostString();
            log.warn(String.format("receive reply message, but not matched any request, CorrelationId: %s , reply from host: %s",
                    correlationId, bornHost));
        }
    }
}

使用方法

消费者

public class RRConsumer {
    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQProducer replyProducer = new DefaultMQProducer("producerGroupTest");
        replyProducer.setNamesrvAddr("127.0.0.1:9876");
        // 启动Producer实例
        replyProducer.start();

        // 实例化消费者
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumerGroupNameTest");
        // 设置NameServer的地址
        consumer.setNamesrvAddr("localhost:9876");
        // 多个tag之间用||分隔，* 代表所有
        consumer.subscribe("TopicTest001", "*");

        // 注册回调实现类来处理从broker拉取回来的消息
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                for (MessageExt msg : msgs) {
                    try {
                        System.out.printf("handle message: %s", msg.toString());

                        // 消费者收到消息后 发送响应消息
                        String replyTo = MessageUtil.getReplyToClient(msg);
                        byte[] replyContent = "reply message contents.".getBytes();
                        // create reply message with given util, do not create reply message by yourself
                        Message replyMessage = MessageUtil.createReplyMessage(msg, replyContent);
                        // send reply message with producer
                        SendResult replyResult = replyProducer.send(replyMessage, 3000);
                        System.out.printf("reply to %s , %s %n", replyTo, replyResult.toString());
                    } catch (MQClientException | RemotingException | MQBrokerException | InterruptedException e) {
                        e.printStackTrace();
                    }
                }
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        // 启动消费者实例
        consumer.start();
        System.out.printf("Consumer Started.%n");
    }
}

生产者同步调用

public class RRProducer {
    public static void main(String[] args) throws Exception {
        DefaultMQProducer producer = new DefaultMQProducer("producerGroupTest");
        producer.setNamesrvAddr("127.0.0.1:9876");
        // 启动Producer实例
        producer.start();
        byte[] bytes = ("消息内容" + 666).getBytes(RemotingHelper.DEFAULT_CHARSET);
        Message msg = new Message("TopicTest001","TagA", UUID.randomUUID().toString(), bytes);
        // 第一个参数为发送的消息 第二个参数为同步调用的超时时间
        Message message = producer.request(msg, 6000);
        System.out.println("body=" + new String(message.getBody()) + " -> " + message);
        producer.shutdown();
    }
}

运行代码后发现，打印的内容并不是消费者返回的内容，而是发送的消息内容。

下面分析一下request方法看看原因

内部调用的是DefaultMQProducerImpl.request(Message, long)

// org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl
public class DefaultMQProducerImpl implements MQProducerInner {
    // 找到1366行
    public Message request(final Message msg,
                           long timeout) throws RequestTimeoutException, MQClientException, RemotingException, MQBrokerException, InterruptedException {
        long beginTimestamp = System.currentTimeMillis();
        // 里面会生成 correlationId 和 requestClientId
        prepareSendRequest(msg, timeout);
        final String correlationId = msg.getProperty(MessageConst.PROPERTY_CORRELATION_ID);

        try {
            // 构造RequestResponseFuture对象
            final RequestResponseFuture requestResponseFuture = new RequestResponseFuture(correlationId, timeout, null);
            /**
             * 将RequestResponseFuture对象放入到Map中, key为correlationId
             * 后面broker回调的时候会通过correlationId找到RequestResponseFuture并将响应结果设置进去
             * 回调方法在:ClientRemotingProcessor#processReplyMessage
             */
            RequestFutureHolder.getInstance().getRequestFutureTable().put(correlationId, requestResponseFuture);

            long cost = System.currentTimeMillis() - beginTimestamp;
            // 发送消息并设置回调方法
            this.sendDefaultImpl(msg, CommunicationMode.ASYNC, new SendCallback() {
                @Override
                public void onSuccess(SendResult sendResult) {
                    /**
                     * 消息发送成功后调用回调方法
                     * 这里直接调用putResponseMessage方法并把当前发送的消息作为参数
                     */
                    requestResponseFuture.setSendRequestOk(true);
                    requestResponseFuture.putResponseMessage(msg);
                }

                @Override
                public void onException(Throwable e) {
                    requestResponseFuture.setSendRequestOk(false);
                    requestResponseFuture.putResponseMessage(null);
                    requestResponseFuture.setCause(e);
                }
            }, timeout - cost);
            /**
             * 因为消息发送成功后直接调用了RequestResponseFuture的putResponseMessage方法
             * 所以这里就直接返回了
             */
            return waitResponse(msg, timeout, requestResponseFuture, cost);
        } finally {
            /**
             * 移除RequestResponseFuture对象
             * 所以当broker回调ClientRemotingProcessor的processReplyMessage方法时也找不到对应的信息
             */
            RequestFutureHolder.getInstance().getRequestFutureTable().remove(correlationId);
        }
    }
}

查看源代码后可以发现，虽然RocketMQ提供了RPC调用的能力，但是好像并不能直接用它提供的方法来打到同步调用的效果。

生产者异步调用

public class RRProducer {
    public static void main(String[] args) throws Exception {
        DefaultMQProducer producer = new DefaultMQProducer("producerGroupTest");
        producer.setNamesrvAddr("127.0.0.1:9876");
        // 启动Producer实例
        producer.start();
        byte[] bytes = ("消息内容" + 666).getBytes(RemotingHelper.DEFAULT_CHARSET);
        Message msg = new Message("RRTopicTest010","TagA", UUID.randomUUID().toString(), bytes);
        // 第一个参数为发送的消息 第二个参数为同步调用的超时时间
        producer.request(msg, new RequestCallback() {
            @Override
            public void onSuccess(Message message) {
                if (message == null) {
                    System.out.println("message is null");
                } else {
                    System.out.println("body=" + new String(message.getBody()) + " -> " + message);
                }
            }
            @Override
            public void onException(Throwable e) {
                e.printStackTrace();
            }
        }, 6000);
        Thread.sleep(60*1000);
        producer.shutdown();
    }
}

运行代码发现，回调方法的onSuccess会被调用两次，第一次回调时消息为null，第二次回调时才是消费者返回的响应信息。

查看源码可以发现，消息发送成功时会被调用一次，而这个时候broker还没有来得及将相应信息设置到中，所以返回null。

当broker回调时，会将消费者的响应信息设置进去，所以第二次是有值的。

// org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl
public class DefaultMQProducerImpl implements MQProducerInner {
    // 找到1398行
    public void request(Message msg, final RequestCallback requestCallback, long timeout)
            throws RemotingException, InterruptedException, MQClientException, MQBrokerException {
        long beginTimestamp = System.currentTimeMillis();
        prepareSendRequest(msg, timeout);
        final String correlationId = msg.getProperty(MessageConst.PROPERTY_CORRELATION_ID);

        final RequestResponseFuture requestResponseFuture = new RequestResponseFuture(correlationId, timeout, requestCallback);
        RequestFutureHolder.getInstance().getRequestFutureTable().put(correlationId, requestResponseFuture);

        long cost = System.currentTimeMillis() - beginTimestamp;
        this.sendDefaultImpl(msg, CommunicationMode.ASYNC, new SendCallback() {
            @Override
            public void onSuccess(SendResult sendResult) {
                requestResponseFuture.setSendRequestOk(true);
                // 消息发送成功时直接调用回调方法
                requestResponseFuture.executeRequestCallback();
            }

            @Override
            public void onException(Throwable e) {
                requestResponseFuture.setCause(e);
                requestFail(correlationId);
            }
        }, timeout - cost);
    }
}

基于RocketMQ提供的方法自己实现

import org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl;
import org.apache.rocketmq.client.producer.*;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageAccessor;
import org.apache.rocketmq.common.message.MessageConst;
import org.apache.rocketmq.common.utils.CorrelationIdUtil;

public class LzcProducer extends DefaultMQProducer {
    public LzcProducer(String producerGroup) {
        super(producerGroup);
    }
    /**
     * 异步调用
     * @param msg
     * @param callback
     * @param timeout
     */
    void rpcRequest(final Message msg, RequestCallback callback, long timeout) {
        long beginTimestamp = System.currentTimeMillis();
        prepareSendRequest(msg, timeout);
        final String correlationId = msg.getProperty(MessageConst.PROPERTY_CORRELATION_ID);
        try {
            final RequestResponseFuture requestResponseFuture = new RequestResponseFuture(correlationId, timeout, null);
            RequestFutureHolder.getInstance().getRequestFutureTable().put(correlationId, requestResponseFuture);
            super.defaultMQProducerImpl.send(msg, new SendCallback() {
                @Override
                public void onSuccess(SendResult sendResult) {
                    requestResponseFuture.setSendRequestOk(true);
                    // 消息发送成功
                    if (callback != null) {
                        try {
                            requestResponseFuture.waitResponseMessage(timeout - (System.currentTimeMillis() - beginTimestamp));
                            callback.onSuccess(requestResponseFuture.getResponseMsg());
                        } catch (InterruptedException e) {
                            callback.onException(e);
                        }
                    }
                }
                @Override
                public void onException(Throwable e) {
                    requestResponseFuture.setCause(e);
                }
            }, timeout);
        } catch (Exception e){
            throw  new RuntimeException(e);
        }
    }

    /**
     * 同步调用
     * @param msg
     * @param timeout
     * @return
     */
    Message rpcRequest(final Message msg, final long timeout) {
        prepareSendRequest(msg, timeout);
        final String correlationId = msg.getProperty(MessageConst.PROPERTY_CORRELATION_ID);
        try {
            final RequestResponseFuture requestResponseFuture = new RequestResponseFuture(correlationId, timeout, null);
            RequestFutureHolder.getInstance().getRequestFutureTable().put(correlationId, requestResponseFuture);

//            SendResult sendResult = super.defaultMQProducerImpl.send(msg, timeout);
//            if (SendStatus.SEND_OK == sendResult.getSendStatus()) {
//                requestResponseFuture.setSendRequestOk(true);
//                Message message = requestResponseFuture.waitResponseMessage(requestResponseFuture.getTimeoutMillis());
//                if (requestResponseFuture.isSendRequestOk() && requestResponseFuture.isTimeout()) {
//                    // 消息发送成功但是超时......
//                }
//                return message;
//            }
//            // 发送失败
//            return null;

            super.defaultMQProducerImpl.send(msg, new SendCallback() {
                @Override
                public void onSuccess(SendResult sendResult) {
                    requestResponseFuture.setSendRequestOk(true);
                }
                @Override
                public void onException(Throwable e) {
                    requestResponseFuture.setCause(e);
                }
            }, timeout);
            Message message = requestResponseFuture.waitResponseMessage(requestResponseFuture.getTimeoutMillis());
            if (requestResponseFuture.isSendRequestOk() && requestResponseFuture.isTimeout()) {
                // 消息发送成功但是超时......
            }
            return message;
        } catch (Exception e){
            throw  new RuntimeException(e);
        } finally {
            RequestFutureHolder.getInstance().getRequestFutureTable().remove(correlationId);
        }
    }

    private void prepareSendRequest(final Message msg, long timeout) {
        DefaultMQProducerImpl defaultMQProducerImpl = super.getDefaultMQProducerImpl();
        String correlationId = CorrelationIdUtil.createCorrelationId();
        String requestClientId = defaultMQProducerImpl.getMqClientFactory().getClientId();
        MessageAccessor.putProperty(msg, MessageConst.PROPERTY_CORRELATION_ID, correlationId);
        MessageAccessor.putProperty(msg, MessageConst.PROPERTY_MESSAGE_REPLY_TO_CLIENT, requestClientId);
        MessageAccessor.putProperty(msg, MessageConst.PROPERTY_MESSAGE_TTL, String.valueOf(timeout));
    }
}