netty学习之五 netty在rocketmq中的使用

Rocketmq 的通信层是基于通信框架 netty，下面来看看rocketmq底层继承图。

NettyRemotingAbstract是NettyRemotingClient和NettyRemotingServer的抽象父类，对发送和接收的公共部分进行了处理

一 . 首先在数据结构方面使用了responseFuture模式

1.保存了RPC处理器 ，Broker 接收请求将 opaque 直接把这个值设置回响应对象，客户端接收到这个响应，通过 opaque 从缓存查找对应的 ResponseFuture 对象

protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
          new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);

当服务端接受客户端响应的时候,会调用NettyServerHandler处理客户端的请求。

 ServerBootstrap childHandler = //
                this.serverBootstrap.group(this.eventLoopGroupBoss,     
this.eventLoopGroupSelector).channel(NioServerSocketChannel.class)
                        .option(ChannelOption.SO_BACKLOG, 1024)
                       
                        .option(ChannelOption.SO_REUSEADDR, true)
                       
                        .option(ChannelOption.SO_KEEPALIVE, false)
                       
                        .childOption(ChannelOption.TCP_NODELAY, true)
                       
                        .option(ChannelOption.SO_SNDBUF, nettyServerConfig.getServerSocketSndBufSize())
                        //
                        .option(ChannelOption.SO_RCVBUF, nettyServerConfig.getServerSocketRcvBufSize())
                        //
                        .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
                        .childHandler(new ChannelInitializer<SocketChannel>() {
                            @Override
                            public void initChannel(SocketChannel ch) throws Exception {
                                ch.pipeline().addLast(
                                        defaultEventExecutorGroup, //
                                        new NettyEncoder(), //
                                        new NettyDecoder(), //
                                        new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()), //
                                        new NettyConnetManageHandler(), //
                                        new NettyServerHandler());
                            }
                        });

NettyServerHandler处理channelRead的时候调用processMessageRecevived

   class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {
        @Override
        protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
            processMessageReceived(ctx, msg);
        }
    }

processMesssageReceived方法就会判断当前是作为Server端，接收的消息是请求，那么调用processTable对应的事件进行处理，如果作为Client端，接收的消息是回复，即接收到Server端的回复，那么从responseTable中，首先获取opaque对应的ResponseFuture，如果这个response是异步回调，则有InvokeCallback，那么调用invokeBack函数，然后将Response塞入ResponseFuture后返回；

public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
        final RemotingCommand cmd = msg;
        if (cmd != null) {
            switch (cmd.getType()) {
                case REQUEST_COMMAND:
                    processRequestCommand(ctx, cmd);
                    break;
                case RESPONSE_COMMAND:
                    processResponseCommand(ctx, cmd);
                    break;
                default:
                    break;
            }
        }
    }

2.其次还保存了对外请求

protected final ConcurrentHashMap<Integer /* opaque */, ResponseFuture> responseTable =
            new ConcurrentHashMap<Integer, ResponseFuture>(256);

在服务端启动 start() 方法的时候调用线程启动扫描将超时的responseFuture直接删除掉

 public void start() {
    .............................
     this.timer.scheduleAtFixedRate(new TimerTask() {
            [@Override](https://my.oschina.net/u/1162528)
            public void run() {
                try {
                    NettyRemotingServer.this.scanResponseTable();
                } catch (Exception e) {
                    log.error("scanResponseTable exception", e);
                }
            }
        }, 1000 * 3, 1000);
}

扫描方法如下,判断哪些future超时：

 public void scanResponseTable() {
        final List<ResponseFuture> rfList = new LinkedList<ResponseFuture>();
        Iterator<Entry<Integer, ResponseFuture>> it = this.responseTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<Integer, ResponseFuture> next = it.next();
            ResponseFuture rep = next.getValue();
            if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
                rep.release();
                it.remove();
                rfList.add(rep);
                plog.warn("remove timeout request, " + rep);
            }
        }
        for (ResponseFuture rf : rfList) {
            try {
                rf.executeInvokeCallback();
            } catch (Throwable e) {
                plog.warn("scanResponseTable, operationComplete Exception", e);
            }
        }
    }

二. 上面看了基本的数据结构和数据处理方式，下面看下发送方信息的发送逻辑。

1. invokeSyncImpl: 同步发送，发送时，生成ResponseFuture，放入responseTable中；然后发送后等待设置的timeout(3s)时间，如果对应的ResponseFuture为空，则报错；否则返回RemoteCommand进行业务逻辑处理；发送失败设置 ResponseFuture 发送失败，并且从缓存中移除 ResponseFuture（没有响 应过来，就用不到缓存中的 ResponseFuturel）

public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
            throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
        final int opaque = request.getOpaque();
        try {
            final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, null, null);
            this.responseTable.put(opaque, responseFuture);
            final SocketAddress addr = channel.remoteAddress();
            channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture f) throws Exception {
                    if (f.isSuccess()) {
                        responseFuture.setSendRequestOK(true);
                        return;
                    } else {
                        responseFuture.setSendRequestOK(false);
                    }
                    responseTable.remove(opaque);
                    responseFuture.setCause(f.cause());
                    responseFuture.putResponse(null);
                    plog.warn("send a request command to channel <" + addr + "> failed.");
                }
            });

            RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
            if (null == responseCommand) {
                if (responseFuture.isSendRequestOK()) {
                    throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
                            responseFuture.getCause());
                } else {
                    throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
                }
            }
            return responseCommand;
        } finally {
            this.responseTable.remove(opaque);
        }
    }

2 invokeAsyncImpl：异步发送，发送时，生成ResponseFuture,设置 opaque， callback， once，超时时间等值，并放入缓存集合，放入responseTable中；如果超过scanResponseTable的timeout (30s)，则报错；否则调用注册的invokeCallback进行回调处理；异步一般链路耗时比较长， 为了防止本地缓存的 netty 请求过多， 使用信号量控制上,限默认 2048 个,发送成功 responseFuture.setSendRequestOK(true); 发送失败 responseFuture.setSendRequestOK(false), 信号量通过 once 释放， 删除缓存 Netty 接收 server 端响应，根据 opaque 从缓存获取 responseFuture，调用回调方法即接 口 InvokeCallback 实现

public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis)
           throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
       final int opaque = request.getOpaque();
       try {
           final ResponseFuture responseFuture = new ResponseFuture(opaque, timeoutMillis, null, null);
           this.responseTable.put(opaque, responseFuture);
           final SocketAddress addr = channel.remoteAddress();
           channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
               @Override
               public void operationComplete(ChannelFuture f) throws Exception {
                   if (f.isSuccess()) {
                       responseFuture.setSendRequestOK(true);
                       return;
                   } else {
                       responseFuture.setSendRequestOK(false);
                   }
                   responseTable.remove(opaque);
                   responseFuture.setCause(f.cause());
                   responseFuture.putResponse(null);
                   plog.warn("send a request command to channel <" + addr + "> failed.");
               }
           });

           RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
           if (null == responseCommand) {
               if (responseFuture.isSendRequestOK()) {
                   throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
                           responseFuture.getCause());
               } else {
                   throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
               }
           }
           return responseCommand;
       } finally {
           this.responseTable.remove(opaque);
       }
   }

rocketmq大概的通信流程基本如此，具体还是要去github上下载源码研读，学下源代码对自身的提高确实蛮大的。

转载于:https://my.oschina.net/u/1792341/blog/906965