rocketmq学习笔记 ---- NettyRemotingServer和NettyRemotingClient

Netty是目前java最流行的网络框架，RocketMQ也使用Netty作为网络通讯底层框架。

NettyRemotingServer实现Netty服务器端功能，接受数据包，在服务器端处理后发送给客户端。NettyRemotingClient实现Netty客户端功能。

NettyRemotingServer

1.start() 方法

start方法主要启动Netty服务器，并在绑定端口后阻塞主线程。这里主要看看Netty服务器端装配了哪些ChannelHandler：

ch.pipeline()
    .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, new HandshakeHandler(TlsSystemConfig.tlsMode))
    .addLast(defaultEventExecutorGroup,
        new NettyEncoder(),
        new NettyDecoder(),
        new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
        new NettyConnectManageHandler(),
        new NettyServerHandler()
);

• HandshakeHandler：检测传输的包体是否使用TLS协议（传输层安全性协议，Transport Layer Security）传输 ，如果包体使用TSL协议，将会在Pipeline中加入处理TSL协议握手的Handler.

ctx.pipeline()
    .addAfter(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, TLS_HANDLER_NAME, sslContext.newHandler(ctx.channel().alloc()))
    .addAfter(defaultEventExecutorGroup, TLS_HANDLER_NAME, FILE_REGION_ENCODER_NAME, new FileRegionEncoder());

• NettyEncoder/NettyDecoder：RocketMQ自定义的编解码Handler，其中编码器将RemotingCommand(RocketMQ的服务器端和客户端交互的数据结构)序列化，其中序列化的方式有json或者二进制，具体编解码方式这里不讨论了。而解码器NettyDecoder继承LengthFieldBasedFrameDecoder，基于长度编解码方式，将二进制反序列化为RemotingCommand。
• IdleStateHandler：Netty包中定义的心跳检测包。读写超时时间由NettyServerConfig.serverChannelMaxIdleTimeSeconds变量控制，默认时间120s。
• NettyConnectManageHandler：Channel连接的管理handler，当发生channel连接的激活、失效、超时和异常时，NettyRemotingServer会生成一个Netty事件，管理连接的组件相应的会处理事件，实现方法后面结合MqNameSrv服务器管理broker服务器路由的实现过程继续探讨。

NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.CONNECT, remoteAddress, ctx.channel()));
NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.CLOSE, remoteAddress, ctx.channel()));
NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.IDLE, remoteAddress, ctx.channel()));
NettyRemotingServer.this.putNettyEvent(new NettyEvent(NettyEventType.EXCEPTION, remoteAddress, ctx.channel()));

• NettyServerHandler：处理RemotingCommand消息，并且返回相应的处理结果。具体实现如下：

class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {
    @Override
    protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
        processMessageReceived(ctx, msg);
    }
}

而processMessageReceived()方法在NettyRemotingServer的父类NettyRemotingAbstract中实现：

public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
        final RemotingCommand cmd = msg;
        if (cmd != null) {
            switch (cmd.getType()) {
                case REQUEST_COMMAND:
                    processRequestCommand(ctx, cmd);
                    break;
                case RESPONSE_COMMAND:
                    processResponseCommand(ctx, cmd);
                    break;
                default:
                    break;
            }
        }
    }

在看processRequestCommand和processResponseCommand这两个方法之前，先了解一下RemotingCommand这个对象：

public class RemotingCommand {
    private static SerializeType serializeTypeConfigInThisServer = SerializeType.JSON;
    static {
        // 获取配置的序列化方式
        final String protocol = System.getProperty(SERIALIZE_TYPE_PROPERTY, System.getenv(SERIALIZE_TYPE_ENV));
        if (!isBlank(protocol)) {
            try {
                serializeTypeConfigInThisServer = SerializeType.valueOf(protocol);
            } catch (IllegalArgumentException e) {
                throw new RuntimeException("parser specified protocol error. protocol=" + protocol, e);
            }
        }
    }

    private int code;  // 请求类型
    private LanguageCode language = LanguageCode.JAVA;
    private int version = 0; // RocketMQ版本编号
    private int opaque = requestId.getAndIncrement(); // 请求序号
    private int flag = 0;  // 标记请求是普通请求，还是无回应的请求
    private String remark; // 失败提示
    private HashMap<String, String> extFields;  // 参数字段的数值
    private transient CommandCustomHeader customHeader;  // 参数的类型

    private SerializeType serializeTypeCurrentRPC = serializeTypeConfigInThisServer;

    private transient byte[] body;  // 解码时缓存的字节流
}

2.processRequestCommand()

public void processRequestCommand(final ChannelHandlerContext ctx, final RemotingCommand cmd) {
        final Pair<NettyRequestProcessor, ExecutorService> matched = this.processorTable.get(cmd.getCode());
        final Pair<NettyRequestProcessor, ExecutorService> pair = null == matched ? this.defaultRequestProcessor : matched;
        final int opaque = cmd.getOpaque(); // 请求序号

        if (pair != null) {
            Runnable run = new Runnable() {
                @Override
                public void run() {
                    try {
                        doBeforeRpcHooks(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd);
                        final RemotingCommand response = pair.getObject1().processRequest(ctx, cmd);
                        doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(ctx.channel()), cmd, response);

                        if (!cmd.isOnewayRPC()) {
                            if (response != null) {
                                response.setOpaque(opaque); // 返回消息中标记请求序号，告诉客户端是哪个请求对应的返回消息
                                response.markResponseType(); // 标记response为返回消息
                                try {
                                    ctx.writeAndFlush(response);
                                } catch (Throwable e) {
                                    log.error("process request over, but response failed", e);
                                    log.error(cmd.toString());
                                    log.error(response.toString());
                                }
                            } else {

                            }
                        }
                    } catch (Throwable e) {
                        // error process
                    }
                }
            };
            // 请求过多，拒绝策略拒绝请求
            if (pair.getObject1().rejectRequest()) {
                // 返回RemotingSysResponseCode.SYSTEM_BUSY消息
                return;
            }

            try {// 提交任务
                final RequestTask requestTask = new RequestTask(run, ctx.channel(), cmd);
                pair.getObject2().submit(requestTask);
            } catch (RejectedExecutionException e) {
                // 返回RemotingSysResponseCode.SYSTEM_BUSY消息
            }
        } else {
            // 返回RemotingSysResponseCode.REQUEST_CODE_NOT_SUPPORTED消息
        }
    }

NettyRemotingServer可以针对每个code，在processorTable中定义一个特定的NettyRequestProcessor，而一般使用defaultRequestProcessor。这里调用NettyRequestProcessor.processRequest()方法处理command信息。

在NameSrvController中注册defaultRequestProcessor时设置的NettyRequestProcessor为DefaultRequestProcessor：

this.remotingServer.registerDefaultProcessor(new DefaultRequestProcessor(this), this.remotingExecutor);

而DefaultRequestProcessor的processRequest()方法实现如下：

    public RemotingCommand processRequest(ChannelHandlerContext ctx,
        RemotingCommand request) throws RemotingCommandException {
        if (ctx != null) {
            log.debug("receive request, {} {} {}", request.getCode(),
                RemotingHelper.parseChannelRemoteAddr(ctx.channel()), request);
        }

        switch (request.getCode()) {
            case RequestCode.PUT_KV_CONFIG:
                return this.putKVConfig(ctx, request);
            case RequestCode.GET_KV_CONFIG:
                return this.getKVConfig(ctx, request);
            case RequestCode.DELETE_KV_CONFIG:
                return this.deleteKVConfig(ctx, request);
            case RequestCode.QUERY_DATA_VERSION:
                return queryBrokerTopicConfig(ctx, request);
            case RequestCode.REGISTER_BROKER:
                Version brokerVersion = MQVersion.value2Version(request.getVersion());
                if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
                    return this.registerBrokerWithFilterServer(ctx, request);
                } else {
                    return this.registerBroker(ctx, request);
                }
            case RequestCode.UNREGISTER_BROKER:
                return this.unregisterBroker(ctx, request);
            case RequestCode.GET_ROUTEINTO_BY_TOPIC:
                return this.getRouteInfoByTopic(ctx, request);
            case RequestCode.GET_BROKER_CLUSTER_INFO:
                return this.getBrokerClusterInfo(ctx, request);
            // ......
            default:
                break;
        }
        return null;
    }

从code变量定义的取名来看，都是一些请求namesrv的操作。选择其中一个操作的实现方法，看看他的执行流程。

    public RemotingCommand putKVConfig(ChannelHandlerContext ctx,
        RemotingCommand request) throws RemotingCommandException {
        // 创建返回消息
        final RemotingCommand response = RemotingCommand.createResponseCommand(null);
        // 装配请求参数
        final PutKVConfigRequestHeader requestHeader =
            (PutKVConfigRequestHeader) request.decodeCommandCustomHeader(PutKVConfigRequestHeader.class);
        // 执行请求
        this.namesrvController.getKvConfigManager().putKVConfig(
            requestHeader.getNamespace(),
            requestHeader.getKey(),
            requestHeader.getValue()
        );

        response.setCode(ResponseCode.SUCCESS);
        response.setRemark(null);
        return response;
    }

其中RemotingCommand的decodeCommandCustomHeader方法，将RemotingCommand的extFields中的值置入PutKVConfigRequestHeader的变量中。

另外，从下面的方法中可以发现，作为CommandCustomHeader的变量都必须只能是基础对象类型，并且只能是String，Integer/int，Long/long，Double/double，Boolean/boolean之一。

public CommandCustomHeader decodeCommandCustomHeader(
        Class<? extends CommandCustomHeader> classHeader) throws RemotingCommandException {
        CommandCustomHeader objectHeader;
        try {
            objectHeader = classHeader.newInstance();
        } catch (InstantiationException e) {
            return null;
        } catch (IllegalAccessException e) {
            return null;
        }

        if (this.extFields != null) {

            Field[] fields = getClazzFields(classHeader);
            for (Field field : fields) {
                if (!Modifier.isStatic(field.getModifiers())) {
                    String fieldName = field.getName();
                    if (!fieldName.startsWith("this")) {
                        try {
                            String value = this.extFields.get(fieldName);
                            if (null == value) {
                                if (!isFieldNullable(field)) {
                                    throw new RemotingCommandException("the custom field <" + fieldName + "> is null");
                                }
                                continue;
                            }

                            field.setAccessible(true);
                            String type = getCanonicalName(field.getType());
                            Object valueParsed;

                            if (type.equals(STRING_CANONICAL_NAME)) {
                                valueParsed = value;
                            } else if (type.equals(INTEGER_CANONICAL_NAME_1) || type.equals(INTEGER_CANONICAL_NAME_2)) {
                                valueParsed = Integer.parseInt(value);
                            } else if (type.equals(LONG_CANONICAL_NAME_1) || type.equals(LONG_CANONICAL_NAME_2)) {
                                valueParsed = Long.parseLong(value);
                            } else if (type.equals(BOOLEAN_CANONICAL_NAME_1) || type.equals(BOOLEAN_CANONICAL_NAME_2)) {
                                valueParsed = Boolean.parseBoolean(value);
                            } else if (type.equals(DOUBLE_CANONICAL_NAME_1) || type.equals(DOUBLE_CANONICAL_NAME_2)) {
                                valueParsed = Double.parseDouble(value);
                            } else {
                                throw new RemotingCommandException("the custom field <" + fieldName + "> type is not supported");
                            }

                            field.set(objectHeader, valueParsed);

                        } catch (Throwable e) {
                            log.error("Failed field [{}] decoding", fieldName, e);
                        }
                    }
                }
            }

            objectHeader.checkFields();
        }

        return objectHeader;
    }

3.processResponseCommand()

该方法的用处是在发送请求命令时，设置一个回调，等到请求执行完毕返回执行结果时，执行回调，实现如下：

    public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
        final int opaque = cmd.getOpaque();
        final ResponseFuture responseFuture = responseTable.get(opaque);
        if (responseFuture != null) {
            responseFuture.setResponseCommand(cmd);

            responseTable.remove(opaque);

            if (responseFuture.getInvokeCallback() != null) {
                executeInvokeCallback(responseFuture);
            } else {
                responseFuture.putResponse(cmd);
                responseFuture.release();
            }
        } else {
            log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
            log.warn(cmd.toString());
        }
    }

下面分别看一下执行同步方法、异步方法以及单向请求方法的分别实现过程：

• 发送同步请求

public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
        final long timeoutMillis)
        throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
        final int opaque = request.getOpaque();

        try {
            final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
            this.responseTable.put(opaque, responseFuture);
            final SocketAddress addr = channel.remoteAddress();
            channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture f) throws Exception {
                    if (f.isSuccess()) {
                        responseFuture.setSendRequestOK(true);
                        return;
                    } else {
                        responseFuture.setSendRequestOK(false);
                    }

                    responseTable.remove(opaque);
                    responseFuture.setCause(f.cause());
                    responseFuture.putResponse(null);
                    log.warn("send a request command to channel <" + addr + "> failed.");
                }
            });

            RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
            if (null == responseCommand) {
                if (responseFuture.isSendRequestOK()) {
                    throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
                        responseFuture.getCause());
                } else {
                    throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
                }
            }

            return responseCommand;
        } finally {
            this.responseTable.remove(opaque);
        }
    }

1.记录请求id。

2.生成一个不带回调方法的ResponseFuture，将其放入responseTable。

3.发送请求RemotingCommand。

4.使用CountDownLatch等待请求执行结果RemotingCommand response。

    public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException {
        this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
        return this.responseCommand;
    }

在processResponseCommand()方法中，如果返回没有回调的时候，执行putResponse()方法，取消CountDownLatch的等待。

    public void putResponse(final RemotingCommand responseCommand) {
        this.responseCommand = responseCommand;
        this.countDownLatch.countDown();
    }

• 执行异步请求

执行异步请求的时候会设置一个异步回调。在这个异步方法的执行过程中，信号量semaphoreAsync控制最大并发异步请求数量，信号量的上限由NettyServerConfig.serverAsyncSemaphoreValue设置， 默认值为64。

    public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
        final InvokeCallback invokeCallback)
        throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
        long beginStartTime = System.currentTimeMillis();
        final int opaque = request.getOpaque();
        boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
        if (acquired) {
            final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
            long costTime = System.currentTimeMillis() - beginStartTime;
            if (timeoutMillis < costTime) {
                once.release();
                throw new RemotingTimeoutException("invokeAsyncImpl call timeout");
            }

            final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
            this.responseTable.put(opaque, responseFuture);
            try {
                channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture f) throws Exception {
                        if (f.isSuccess()) {
                            responseFuture.setSendRequestOK(true);
                            return;
                        }
                        requestFail(opaque);
                        log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
                    }
                });
            } catch (Exception e) {
                responseFuture.release();
                log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
                throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
            }
        } else {
            if (timeoutMillis <= 0) {
                throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
            } else {
                String info =
                    String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
                        timeoutMillis,
                        this.semaphoreAsync.getQueueLength(),
                        this.semaphoreAsync.availablePermits()
                    );
                log.warn(info);
                throw new RemotingTimeoutException(info);
            }
        }
    }

• 异步单向请求

执行单向异步请求比起同步请求和异步请求相对简单。在单向异步方法的执行过程中，信号量semaphoreOneway控制最大并发请求数量，信号量的上限由NettyServerConfig.serverOnewaySemaphoreValue 设置， 默认值为256。

    public void invokeOnewayImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
        throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
        request.markOnewayRPC();
        boolean acquired = this.semaphoreOneway.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
        if (acquired) {
            final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreOneway);
            try {
                channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture f) throws Exception {
                        once.release();
                        if (!f.isSuccess()) {
                            log.warn("send a request command to channel <" + channel.remoteAddress() + "> failed.");
                        }
                    }
                });
            } catch (Exception e) {
                once.release();
                log.warn("write send a request command to channel <" + channel.remoteAddress() + "> failed.");
                throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
            }
        } else {
            if (timeoutMillis <= 0) {
                throw new RemotingTooMuchRequestException("invokeOnewayImpl invoke too fast");
            } else {
                String info = String.format(
                    "invokeOnewayImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
                    timeoutMillis,
                    this.semaphoreOneway.getQueueLength(),
                    this.semaphoreOneway.availablePermits()
                );
                log.warn(info);
                throw new RemotingTimeoutException(info);
            }
        }
    }

NettyRemotingClient

NettyRemotingClient的start()方法与NettyRemotingServer类似，在添加ChannelHandler处理包的handler是NettyClientHandler，其功能与NettyServerHandler一样。

• invokeSync

    public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
        throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
        long beginStartTime = System.currentTimeMillis();
        final Channel channel = this.getAndCreateChannel(addr);
        if (channel != null && channel.isActive()) {
            try {
                doBeforeRpcHooks(addr, request);
                long costTime = System.currentTimeMillis() - beginStartTime;
                if (timeoutMillis < costTime) {
                    throw new RemotingTimeoutException("invokeSync call timeout");
                }
                RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
                doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
                return response;
            } catch (Exception e) {
                // .....
                this.closeChannel(addr, channel);
            }
        } else {
            this.closeChannel(addr, channel);
            throw new RemotingConnectException(addr);
        }
    }

invokeSync方法根据发送命令的地址取出或创建一个Channel，然后调用父类的invokeSyncImpl()方法，阻塞等待返回结果。

    private Channel getAndCreateChannel(final String addr) throws InterruptedException {
        if (null == addr) {
            return getAndCreateNameserverChannel();
        }

        ChannelWrapper cw = this.channelTables.get(addr);
        if (cw != null && cw.isOK()) {
            return cw.getChannel();
        }

        return this.createChannel(addr);
    }

当addr为null时，取出连接到namesrv的channek。也就是说，当调用NettyRemotingClient.invokeSync(null, request, 3000)时，请求会发送到namesrv。getAndCreateNameserverChannel()方法，从定时检测刷新的namesrvAddrList中按序取出自己绑定的nameserver的地址。然后从缓存中取出channel，如果缓存中的channel不存在或失活，那么重新连接。

类似的，createChannel()方法，主要逻辑实现连接到目标服务器，并将生成的channel放入到channelTables缓存中，下一次发送命令时，如果channel依然存活，那么从缓存中取出channel使用。

总结

RocketMQ中Netty服务器的逻辑大概如图所示：

1.客户端执行invokeSyncImpl、invokeAsyncImpl方法的时候，在responseTable中放入一个requestId对应的ResponseFuture，然后发送请求导NettyRemotingServer上。

2.服务器经过解析后，生成对应的请求对象，经过NettyServerHandler处理，生成一个RemotingCommand的Response通过WriteAndFlush将结果返回给客户端。

3.客户端从responseTable中通过requestId取出对应的ResponseFuture，如果有回调，即执行回调；否则取消等待返回结果的线程阻塞。