6.rocketmq源代码学习----客户端怎么与服务端通信

我们通过发送消息的方法，来分析客户端与服务端是怎么通信的：

public SendResult sendMessage(//
            final String addr,// 1
            final String brokerName,// 2
            final Message msg,// 3
            final SendMessageRequestHeader requestHeader,// 4
            final long timeoutMillis,// 5
            final CommunicationMode communicationMode,// 6
            final SendCallback sendCallback// 7
    ) throws RemotingException, MQBrokerException, InterruptedException {
		//....................
        switch (communicationMode) {
        //1.单向发送，不需要知道响应结果
        case ONEWAY:
            this.remotingClient.invokeOneway(addr, request, timeoutMillis);
            return null;
        //2. 异步发送，可设置一个回调方法
        case ASYNC:
            this.sendMessageAsync(addr, brokerName, msg, timeoutMillis, request, sendCallback);
            return null;
        //3. 同步发送，同步获取发送结果
        case SYNC:
            return this.sendMessageSync(addr, brokerName, msg, timeoutMillis, request);
        default:
            assert false;
            break;
        }

        return null;
    }

看到这段代码，我心中有2个疑问：

• 同步发送结果，是怎么同步获取到响应的？
• 异步发送时，回调方法是在哪里调用的？

2.从发送消息的关键代码开始分析：NettyRemotingAbstract.invokeSyncImpl()

public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
            final long timeoutMillis) throws InterruptedException, RemotingSendRequestException,
            RemotingTimeoutException {
        try {
           //request.getOpaque()，发送请求的标识，请求、响应均会包含该字段，
           //用于接收异步响应时，根据Opaque 找到对应的回调函数或者设置返回结果。
           //（NettyRemotingAbstract.processResponseCommand() ）
            final ResponseFuture responseFuture =
                    new ResponseFuture(request.getOpaque(), timeoutMillis, null, null);
            //根据请求id，记录下回调函数
            this.responseTable.put(request.getOpaque(), responseFuture);
            //发送请求
            channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture f) throws Exception {
                    if (f.isSuccess()) {
                        responseFuture.setSendRequestOK(true);
                        return;
                    }
                    else {
                        responseFuture.setSendRequestOK(false);
                    }

                    responseTable.remove(request.getOpaque());
                    responseFuture.setCause(f.cause());
                    responseFuture.putResponse(null);
                    plog.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
                    plog.warn(request.toString());
                }
            });
	   //阻塞等待响应结果，如果等待timeoutMillis后，将自动唤醒
            RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
            if (null == responseCommand) {
                if (responseFuture.isSendRequestOK()) {
                    throw new RemotingTimeoutException(RemotingHelper.parseChannelRemoteAddr(channel),
                        timeoutMillis, responseFuture.getCause());
                }
                else {
                    throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel),
                        responseFuture.getCause());
                }
            }

            return responseCommand;
        }
        finally {
            this.responseTable.remove(request.getOpaque());
        }
    }

=注意：这里用到了java.util.concurrent包中的，CountDownLatch，来阻塞等待；当收到响应结果后，自动解锁返回；或者当阻塞超时后，自动解锁。=

3.ResponseFuture.waitResponse()

public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException {
        this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
        return this.responseCommand;
    }

• await()方法在哪里唤醒呢？接着往下看
• 这里自始至终没有看到responseCommand在哪里赋值，那是在哪里赋值的呢？

4.NettyRemotingAbstract.processResponseCommand()
通过调用链找一下responseCommand在哪里赋值了。

通过上图，可知，是NettyClientHandler中处理的，那么NettyClientHandler是做什么的呢？先来看下NettyClientHandler的关键方法，可以知道NettyClientHandler是来接收服务端的响应，处理返回结果：

NettyClientHandler.channelRead0()–>NettyRemotingAbstract.processMessageReceived()–>NettyRemotingAbstract.processResponseCommand()

public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
       //根据请求标识找到之前放到responseTable的ResponseFuture
        final ResponseFuture responseFuture = responseTable.get(cmd.getOpaque());
        if (responseFuture != null) {
           //在这里设置response... 并调用了请求时设置的countDownLatch.countdown()方法，以此来唤醒发送时的阻塞await()
            responseFuture.setResponseCommand(cmd);

            responseFuture.release();

            if (responseFuture.getInvokeCallback() != null) {
                boolean runInThisThread = false;
                ExecutorService executor = this.getCallbackExecutor();
                if (executor != null) {
                    try {
                        executor.submit(new Runnable() {
                            @Override
                            public void run() {
                                try {
                                    responseFuture.executeInvokeCallback();
                                }
                                catch (Throwable e) {
                                    plog.warn("excute callback in executor exception, and callback throw", e);
                                }
                            }
                        });
                    }
                    catch (Exception e) {
                        runInThisThread = true;
                        plog.warn("excute callback in executor exception, maybe executor busy", e);
                    }
                }
                else {
                    runInThisThread = true;
                }

                if (runInThisThread) {
                    try {
                        responseFuture.executeInvokeCallback();
                    }
                    catch (Throwable e) {
                        plog.warn("executeInvokeCallback Exception", e);
                    }
                }
            }
            else {
                responseFuture.putResponse(cmd);
            }
        }
        else {
            plog.warn("receive response, but not matched any request, "
                    + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
            plog.warn(cmd.toString());
        }

        responseTable.remove(cmd.getOpaque());
    }

上面最关键的代码：调用responseFuture.putResponse()设置response、调用回调方法responseFuture.executeInvokeCallback();

5.responseFuture.putResponse(cmd);

public void putResponse(final RemotingCommand responseCommand) {
        this.responseCommand = responseCommand;
        this.countDownLatch.countDown();
    }

在这里我们上面的谜题解开了，countDown()是在收到服务端响应后调用的，唤醒invokeSyncImpl()处的 waitResponse()阻塞。

疑问又来了，那么NettyClientHandler是在哪里设置的呢？

通过NettyClientHandler我们又找到了NettyRemotingClient：

又通过NettyRemotingClient的启动方法：
通过调用链可知，在客户端消息消费、消息发送时，都会调用NettyRemotingClient的start()

NettyRemotingClient.start()

 @Override
    public void start() {
        this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(//
            nettyClientConfig.getClientWorkerThreads(), //
            new ThreadFactory() {

                private AtomicInteger threadIndex = new AtomicInteger(0);

                @Override
                public Thread newThread(Runnable r) {
                    //线程名称
                    return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
                }
            });

        Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)//
            //
            .option(ChannelOption.TCP_NODELAY, true)
            //
            .option(ChannelOption.SO_KEEPALIVE, false)
            //
            .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
            //
            .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
            //
            .handler(new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline().addLast(//
                        defaultEventExecutorGroup, //
                        //消息编码器，将RemotingCommand转换为字节数组
                        new NettyEncoder(), //
                        //消息解码器，将ByteBuffer转换为RemotingCommand
                        new NettyDecoder(), //
                        new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),//
                        new NettyConnetManageHandler(), //
                        //继承SimpleChannelInboundHandler，统一的接收响应数据处理器
                        new NettyClientHandler());
                }
            });

        this.timer.scheduleAtFixedRate(new TimerTask() {

            @Override
            public void run() {
                try {
                    //异步发送消息的回调处理
                    NettyRemotingClient.this.scanResponseTable();
                }
                catch (Exception e) {
                    log.error("scanResponseTable exception", e);
                }
            }
        }, 1000 * 3, 1000);

        if (this.channelEventListener != null) {
            this.nettyEventExecuter.start();
        }
    }

到这里一切都真相大白了，=rocketMq是通过NettyRemotingClient 启动netty的监听，并且设置了NettyClientHandler处理响应结果。=

rocketmq对netty的调用全部都封装到remoting包中：其中最主要的几个类如图：

NettyRemotingClient：对客户端通信做封装
NettyRemotingServer：对服务端通信做封装
RemotingCommand：请求、响应对象封装

总结，在rocketmq服务端与客户端通信过程中，有几个关键点：
1、请求与响应的RemotingCommand中，有一个关键的opaque，用于标识一次请求。并通过opaque获取ResponseFuture，设置响应结果
2、rocketmq通过CountDownLatch阻塞，并等待响应结果
3、通过NettyClientHandler接收服务端响应，设置响应结果，并通过countdown()唤醒发送时的阻塞