深度解析RocketMq源码-远程通信组件_rocketmq netty连接源码-CSDN博客

本文链接：https://blog.csdn.net/zhifou123456/article/details/138367681

1.前言

rocketmq的各个组件之间的通信都是用rocketmq-remoting这个组件实现，而这个组件底层采用的是netty来进行通信的，所以学习这个组件我们可以学到netty的使用。

2.netty基本介绍

在介绍这个组件之前，我们来了解一下什么是netty。

2.1 NIO

2.1.1 传统AIO的不足

传统AIO是，每次请求到来的时候，会启动一个线程处理请求，并且每个线程在该请求处理完成后，才会处理下一个请求，所以AIO也叫同步阻塞IO，当请求量大的时候，会创建大量请求。

2.2.2 NIO

NIO是同步非阻塞IO，他会启动一个selector线程来接收请求，当没有请求的时候selector会阻塞。如果有读写请求的时候，会交给线程池来处理，这就是同步非阻塞IO。

2.2.3 举例

下面是nio的server的一个例子：

public class SimpleServer {

    public static void main(String[] args) throws IOException {
        Logger logger = LoggerFactory.getLogger(SimpleServer.class);

        //创建服务端channel
        ServerSocketChannel serverSocketChannel = ServerSocketChannel.open();
        //设置channel非阻塞
        serverSocketChannel.configureBlocking(false);
        //获得selector
        Selector selector = Selector.open();
        //把channel注册到selector上,现在还没有给key设置感兴趣的事件
        SelectionKey selectionKey = serverSocketChannel.register(selector, 0, serverSocketChannel);
        //给key设置感兴趣的事件
        selectionKey.interestOps(SelectionKey.OP_ACCEPT);
        //绑定端口号
        serverSocketChannel.bind(new InetSocketAddress(8080));
        //然后开始接受连接,处理事件,整个处理都在一个死循环之中
        while (true) {
            //当没有事件到来的时候，这里是阻塞的,有事件的时候会自动运行
            selector.select();
            //如果有事件到来，这里可以得到注册到该selector上的所有的key，每一个key上都有一个channel
            Set<SelectionKey> selectionKeys = selector.selectedKeys();
            //得到集合的迭代器
            Iterator<SelectionKey> keyIterator = selectionKeys.iterator();
            while (keyIterator.hasNext()) {
                //得到每一个key
                SelectionKey key = keyIterator.next();
                //首先要从集合中把key删除，否则会一直报告该key
                keyIterator.remove();
                //接下来就要处理事件，判断selector轮询到的是什么事件，并根据事件作出回应
                //如果是连接事件
                if (key.isAcceptable()) {
                    //得到服务端的channel,这里有两种方式获得服务端的channel，一种是直接获得,一种是通过attachment获得
                    //因为之前把服务端channel注册到selector上时，同时把serverSocketChannel放进去了
                    ServerSocketChannel channel = (ServerSocketChannel)key.channel();
                    //ServerSocketChannel attachment = (ServerSocketChannel)key.attachment();
                    //得到客户端的channel
                    SocketChannel socketChannel = channel.accept();
                    socketChannel.configureBlocking(false);
                    //接下来就要管理客户端的channel了，和服务端的channel的做法相同，客户端的channel也应该被注册到selector上
                    //通过一次次的轮询来接受并处理channel上的相关事件
                    //把客户端的channel注册到之前已经创建好的selector上
                    SelectionKey socketChannelKey = socketChannel.register(selector, 0, socketChannel);
                    //给客户端的channel设置可读事件
                    socketChannelKey.interestOps(SelectionKey.OP_READ);
                    logger.info("客户端连接成功！");
                    //连接成功之后，用客户端的channel写回一条消息
                    socketChannel.write(ByteBuffer.wrap("我发送成功了".getBytes()));
                    logger.info("向客户端发送数据成功！");
                }
                //如果接受到的为可读事件，说明要用客户端的channel来处理
                if (key.isReadable()) {
                    //同样有两种方式得到客户端的channel，这里只列出一种
                    SocketChannel channel = (SocketChannel)key.channel();
                    //分配字节缓冲区来接受客户端传过来的数据
                    ByteBuffer buffer = ByteBuffer.allocate(1024);
                        //向buffer写入客户端传来的数据
                        int len = channel.read(buffer);
                        logger.info("读到的字节数：" + len);
                        if (len == -1) {
                            channel.close();
                            break;
                        }else{
                        //切换buffer的读模式
                        buffer.flip();
                        logger.info(Charset.defaultCharset().decode(buffer).toString());
                    }
                }
            }
        }
    }
}

2.2 netty的线程模型

上面是netty经典的线程模型图，可以看出netty包含了两个NioEventLoop（两个线程池），分别是bossGroup和workerGroup。workerGroup主要用来封装请求，bossGroup用来处理请求，从图中可以得出Netty工作步骤如下:

1.client发送请求到达bossGroup，bossGroup将请求注册到selector上面这一动作封装成一个runnableTask，并且交给NioEventLoop里面的线程池执行。这里线程池其实是每个NioEventloopGroup包含多个NipEventLoop，每个NioEventLoop包含一个线程，所以可以理解为NioEventLoopGroup包含一个线程池。

2.selector收到请求过后，会判断请求是要读、写或者是建立连接，并且将其封装成一个runnableTask，并且交给workGorup线程池执行。workGroup线程池执行对于的IO操作。

3.worker线程池执行对于的IO操作的时候，会将请求发送给pipeline，pipeline其实就是一个拦截器链，会完成请求的编解码。我们对netty进行业务处理的时候，其实也是去实现ChannelHandler，使其成为拦截器的一个节点。

为什么Netty采用这种线程模型可以提升吞吐量？

其实思想就是将一个完整的业务链路分别拆分，然后并且交给不同的线程执行。在现实生活中，我们去吃饭的时候，假设老板只有一个人，他需要招待我们点餐->炒菜->上菜这几个步骤过后，才能去招待下一桌客人，这其实就是AIO。但实际生活中，是点餐人员来招待点餐，点好过后，就把菜单给厨师，厨师开始炒菜，这个时候点餐人员就可以招待下一桌人员了，这其实就是NIO。所以思想源于生活。

2.3 netty的pipeline

在NIO中，客户端的连接会被抽象为socketchannel，服务端的连接被抽象为serversocketchannel，而在netty里面的连接可以用channel来表示。

netty有两种处理器，分别是入站处理器和出站处理器，入站处理器，处理读消息，

3.rocketmq是如何利用Netty通信

3.1 rocketmq的核心组件

3.1.1 NettyRemotingAbstract的基本组成和作用

1.有哪些组件构成

NettyRemotingAbstract的主要作用是接收Netty的请求，根据请求找到一个响应或者请求的processor，然后调用对应的客户端实现的processor来处理结果。组成如下：

    /**
     * Semaphore to limit maximum number of on-going one-way requests, which protects system memory footprint.
     */
    //oneway请求的信号量，通过它来控制同时发送oneway请求的并发数
    protected final Semaphore semaphoreOneway;

    /**
     * Semaphore to limit maximum number of on-going asynchronous requests, which protects system memory footprint.
     */
    //异步请求的信号量，通过它来控制同时发送异步请求的并发数
    protected final Semaphore semaphoreAsync;

    /**
     * This map caches all on-going requests.
     */
    //存储请求的opaque 处理请求的future
    protected final ConcurrentMap<Integer /* opaque */, ResponseFuture> responseTable =
        new ConcurrentHashMap<Integer, ResponseFuture>(256);

    /**
     * This container holds all processors per request code, aka, for each incoming request, we may look up the
     * responding processor in this map to handle the request.
     */
    //享元模式的体现，这个是存储对应的请求类型和响应处理器的一个table，key:RequestCode里面 value:一个处理器pair，其实就是真正的对消息
    //做业务逻辑处理的地方，为了增大存储，采用线程池进行异步处理。NettyRequestProcessor有不同的子类，来处理不同的消息，比如DefaultRequestProcessor
    //就是处理Namesrv与broker消息之间通信的processor
    protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
        new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);

    /**
     * Executor to feed netty events to user defined {@link ChannelEventListener}.
     */
    //处理Netty事件的一个线程池，比如根据Netty传入的空闲，连接事件等进行处理,比如传入一个Close事件，以Namesrve为例子，表示
    //断开Namesrv和broker之间的连接，所以需要清除Namesrv里面routinfo的关于broker的信息
    protected final NettyEventExecutor nettyEventExecutor = new NettyEventExecutor();

    /**
     * The default request processor to use in case there is no exact match in {@link #processorTable} per request code.
     */
    //如果通过请求的唯一id没有找到对应processor，就用这个defaultRequestProcessor来处理请求
    protected Pair<NettyRequestProcessor, ExecutorService> defaultRequestProcessor;

    /**
     * SSL context via which to create {@link SslHandler}.
     */
    //ssl加密通信的组件
    protected volatile SslContext sslContext;

    /**
     * custom rpc hooks
     */
    //在请求处理的前后会加入一些拦截器，这里拦截器就是RPCHook
    protected List<RPCHook> rpcHooks = new ArrayList<RPCHook>();

2.rocketmq是如何处理响应的 - responseTable和ResponseFuture

rocketmq每个请求会有一个唯一id，opqueueId的，并且在发送请求过后会将opaque

和一个响应理器:ResponseFuture加入到responseTable中。

public class ResponseFuture {
    //请求的唯一id
    private final int opaque;
    //对应的channel
    private final Channel processChannel;
    //请求的超时时间
    private final long timeoutMillis;
    //在收到响应的时候，设置处理该请求的回调函数,交由调用方实现
    private final InvokeCallback invokeCallback;
    private final long beginTimestamp = System.currentTimeMillis();
    //同步请求阻塞请求线程，等待处理结果
    private final CountDownLatch countDownLatch = new CountDownLatch(1);
    //做请求的并发限制
    private final SemaphoreReleaseOnlyOnce once;

    private final AtomicBoolean executeCallbackOnlyOnce = new AtomicBoolean(false);
    //接收到的command响应
    private volatile RemotingCommand responseCommand;
    private volatile boolean sendRequestOK = true;
    private volatile Throwable cause;

    public ResponseFuture(Channel channel, int opaque, long timeoutMillis, InvokeCallback invokeCallback,
        SemaphoreReleaseOnlyOnce once) {
        this.opaque = opaque;
        this.processChannel = channel;
        this.timeoutMillis = timeoutMillis;
        this.invokeCallback = invokeCallback;
        this.once = once;
    }

    /**
     * 执行回调方法
     */
    public void executeInvokeCallback() {
        if (invokeCallback != null) {
            //通过AtomicBoolean保证回调方法只会执行一次
            if (this.executeCallbackOnlyOnce.compareAndSet(false, true)) {
                invokeCallback.operationComplete(this);
            }
        }
    }

    public void release() {
        if (this.once != null) {
            this.once.release();
        }
    }

    public boolean isTimeout() {
        long diff = System.currentTimeMillis() - this.beginTimestamp;
        return diff > this.timeoutMillis;
    }
    
    //通过同步方法，阻塞线程，保证同步调用接收到请求
    public RemotingCommand waitResponse(final long timeoutMillis) throws InterruptedException {
        this.countDownLatch.await(timeoutMillis, TimeUnit.MILLISECONDS);
        return this.responseCommand;
    }

    //当同步调用的响应来时，调用putResponse方法，会通过countDownLatch的countDown方法，释放掉调用方的阻塞线程
    public void putResponse(final RemotingCommand responseCommand) {
        this.responseCommand = responseCommand;
        this.countDownLatch.countDown();
    }
 //异步请求通过setResponseCommand设置响应
    public void setResponseCommand(RemotingCommand responseCommand) {
        this.responseCommand = responseCommand;
    }
}

额外补充一点，下面是SemaphoreReleaseOnlyOnce 的标准用法，因为信号量在释放的时候，可能也会有并发问题，所以需要用AtomicBoolean 来保证释放的并发安全。详情可参考：java高并发高频面试题:Sempahore的使用场景与常见误区_java semaphore坑-CSDN博客

public class SemaphoreReleaseOnlyOnce {
    private final AtomicBoolean released = new AtomicBoolean(false);
    private final Semaphore semaphore;

    public SemaphoreReleaseOnlyOnce(Semaphore semaphore) {
        this.semaphore = semaphore;
    }

    public void release() {
        if (this.semaphore != null) {
            if (this.released.compareAndSet(false, true)) {
                this.semaphore.release();
            }
        }
    }

    public Semaphore getSemaphore() {
        return semaphore;
    }
}

3.rocketmq是如何处理请求的 - processorTable

NettyRemotingAbstract里面有一个组件processorTable，记录了请求类型和processor的映射关系，根据request code判断出是哪种processor，然后通过processor调用processRequest方法执行处理逻辑。

protected final HashMap<Integer/* request code */, Pair<NettyRequestProcessor, ExecutorService>> processorTable =
        new HashMap<Integer, Pair<NettyRequestProcessor, ExecutorService>>(64);

public interface NettyRequestProcessor {
    RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request)
        throws Exception;

    boolean rejectRequest();

}

真正的处理逻辑在processor的实现类里面，也就是每个调用法实现的子类。

3.1.2 NettyRemotingServer的基本组成

1.有哪些组件构成

NettyRemotingServer其实就是一个Netty的server端。

public class NettyRemotingServer extends NettyRemotingAbstract implements RemotingServer {
    private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);
    // netty的启动器
    private final ServerBootstrap serverBootstrap;
    // 接收客户端的连接请求，并且将处理完的请求交给workGroup
    private final EventLoopGroup eventLoopGroupSelector;
    //netty的workGroup，负责连接过后的读写请求
    private final EventLoopGroup eventLoopGroupBoss;
    
    //netty的相关的config
    private final NettyServerConfig nettyServerConfig;

    //执行callback操作的线程池
    private final ExecutorService publicExecutor;
    //监听channel的断开或者连接事件，并且根据事件类型执行对应的操作
    private final ChannelEventListener channelEventListener;
    
    //定时任务调度器，主要用来扫描连接是否超时
    private final Timer timer = new Timer("ServerHouseKeepingService", true);
    //ChannelEventListener监听到各种事件后，通过各种事件交由该线程池执行
    private DefaultEventExecutorGroup defaultEventExecutorGroup;


    private int port = 0;
    
    //ssl需要加入的各种handler
    private static final String HANDSHAKE_HANDLER_NAME = "handshakeHandler";
    private static final String TLS_HANDLER_NAME = "sslHandler";
    private static final String FILE_REGION_ENCODER_NAME = "fileRegionEncoder";

    // sharable handlers
    //ssl需要handshake的handler
    private HandshakeHandler handshakeHandler;
    //编码器
    private NettyEncoder encoder;
    //连接处理器
    private NettyConnectManageHandler connectionManageHandler;
    //处理请求和响应的处理器
    private NettyServerHandler serverHandler;
}

接下来我们先看一下这个nettyserver是如何启动起来的。对应的就是NettyRemotingServer中的start方法：

  public void start() {
		//启动对连接事件的处理器
        this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
            nettyServerConfig.getServerWorkerThreads(),
            new ThreadFactory() {

                private AtomicInteger threadIndex = new AtomicInteger(0);

                @Override
                public Thread newThread(Runnable r) {
                    return new Thread(r, "NettyServerCodecThread_" + this.threadIndex.incrementAndGet());
                }
            });
		//构造netty的pipeline链条中的各个handler
        prepareSharableHandlers();
		
		//构造netty的启动器
        ServerBootstrap childHandler =
            this.serverBootstrap.group(this.eventLoopGroupBoss, this.eventLoopGroupSelector)
                .channel(useEpoll() ? EpollServerSocketChannel.class : NioServerSocketChannel.class)
                .option(ChannelOption.SO_BACKLOG, nettyServerConfig.getServerSocketBacklog())
                .option(ChannelOption.SO_REUSEADDR, true)
                .option(ChannelOption.SO_KEEPALIVE, false)
                .childOption(ChannelOption.TCP_NODELAY, true)
                .localAddress(new InetSocketAddress(this.nettyServerConfig.getListenPort()))
                .childHandler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    public void initChannel(SocketChannel ch) throws Exception {
                        ch.pipeline()
							//添加ssl需要的handShakeHandler
                            .addLast(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, handshakeHandler)
                            .addLast(defaultEventExecutorGroup,
								//rocketmq自定义通信协议解编码器
                                encoder,
								//rocketmq自定义通信协议解码器
                                new NettyDecoder(),
								//netty自带的空闲处理器
                                new IdleStateHandler(0, 0, nettyServerConfig.getServerChannelMaxIdleTimeSeconds()),
								//rocketmq的连接处理器
                                connectionManageHandler,
								//接收到请求后真正做逻辑处理的地方
                                serverHandler
                            );
                    }
                });
       
        try {
			//绑定端口后，启动netty
            ChannelFuture sync = this.serverBootstrap.bind().sync();
            InetSocketAddress addr = (InetSocketAddress) sync.channel().localAddress();
            this.port = addr.getPort();
        } catch (InterruptedException e1) {
            throw new RuntimeException("this.serverBootstrap.bind().sync() InterruptedException", e1);
        }

        if (this.channelEventListener != null) {
            this.nettyEventExecutor.start();
        }
		//启动扫描响应的定时任务
        this.timer.scheduleAtFixedRate(new TimerTask() {

            @Override
            public void run() {
                try {
                    NettyRemotingServer.this.scanResponseTable();
                } catch (Throwable e) {
                    log.error("scanResponseTable exception", e);
                }
            }
        }, 1000 * 3, 1000);
    }

其实现在我们已经大概清除rocketmq在收到请求后，会经过哪些handler处理了。

接下来我们来看看netty的各个组件的作用:

2 ssl连接的handshakeHandler

这个是用来判断服务端和客户端是否要通过ssl对连接进行加密，如果需要则会想netty的pipeline中加入sslHandler和fileRegionEncoder，请求在经过这借个handler过后，会自动与客户端建立ssl连接。

class HandshakeHandler extends SimpleChannelInboundHandler<ByteBuf> {

        private final TlsMode tlsMode;

        private static final byte HANDSHAKE_MAGIC_CODE = 0x16;

        HandshakeHandler(TlsMode tlsMode) {
            this.tlsMode = tlsMode;
        }

        @Override
        protected void channelRead0(ChannelHandlerContext ctx, ByteBuf msg) throws Exception {

            // mark the current position so that we can peek the first byte to determine if the content is starting with
            // TLS handshake
            msg.markReaderIndex();

            byte b = msg.getByte(0);

            if (b == HANDSHAKE_MAGIC_CODE) {
                switch (tlsMode) {
                    case DISABLED:
                        ctx.close();
                        log.warn("Clients intend to establish an SSL connection while this server is running in SSL disabled mode");
                        break;
                    case PERMISSIVE:
                    case ENFORCING:
                        if (null != sslContext) {
                            ctx.pipeline()
                                .addAfter(defaultEventExecutorGroup, HANDSHAKE_HANDLER_NAME, TLS_HANDLER_NAME, sslContext.newHandler(ctx.channel().alloc()))
                                .addAfter(defaultEventExecutorGroup, TLS_HANDLER_NAME, FILE_REGION_ENCODER_NAME, new FileRegionEncoder());
                            log.info("Handlers prepended to channel pipeline to establish SSL connection");
                        } else {
                            ctx.close();
                            log.error("Trying to establish an SSL connection but sslContext is null");
                        }
                        break;

                    default:
                        log.warn("Unknown TLS mode");
                        break;
                }
            } else if (tlsMode == TlsMode.ENFORCING) {
                ctx.close();
                log.warn("Clients intend to establish an insecure connection while this server is running in SSL enforcing mode");
            }

            // reset the reader index so that handshake negotiation may proceed as normal.
            msg.resetReaderIndex();

            try {
                // Remove this handler
                ctx.pipeline().remove(this);
            } catch (NoSuchElementException e) {
                log.error("Error while removing HandshakeHandler", e);
            }

            // Hand over this message to the next .
            ctx.fireChannelRead(msg.retain());
        }
    }

3.解码用的NettyEncoder和rocketmq的协议RemotingCommand

public class RemotingCommand { 
   //响应或者请求码
    private int code;
    //采用的语言
    private LanguageCode language = LanguageCode.JAVA;
    private int version = 0;
    //请求id，每个请求有个唯一id
    private int opaque = requestId.getAndIncrement();
    private int flag = 0;
    //备注
    private String remark;
    //扩展字段信息
    private HashMap<String, String> extFields;
    //请求头，一般每种请求有自己对应的请求头，比如发送给namesrv删除topic的请求会放DeleteTopicRequestHeader请求头
    private transient CommandCustomHeader customHeader;
    //序列化方式，默认采用ROCKETMQ
    private SerializeType serializeTypeCurrentRPC = serializeTypeConfigInThisServer;
    //请求体
    private transient byte[] body;
}

其实一般框架在采用netty通信的时候，都会自定义自己的协议，比如定义一个魔数做校验，顶一个请求头传输协议信息，然后是请求体等。在编码的时候，一般采用的方式是，字段长度+字段内容的方式形成一个字节流。而在解码的时候，首先会先读取字段长度，然后根据长度获取字段的内容，完成解码。rocketmq的编解码方式也是这样处理的。

 public static RemotingCommand decode(final ByteBuf byteBuffer) throws RemotingCommandException {
        //读取字段长度
        int length = byteBuffer.readableBytes();
        int oriHeaderLen = byteBuffer.readInt();
        int headerLength = getHeaderLength(oriHeaderLen);
        if (headerLength > length - 4) {
            throw new RemotingCommandException("decode error, bad header length: " + headerLength);
        }
        //读取内容
        RemotingCommand cmd = headerDecode(byteBuffer, headerLength, getProtocolType(oriHeaderLen));

        int bodyLength = length - 4 - headerLength;
        byte[] bodyData = null;
        if (bodyLength > 0) {
            bodyData = new byte[bodyLength];
            byteBuffer.readBytes(bodyData);
        }
        cmd.body = bodyData;

        return cmd;
    }

而nettyEnocder的解码其实就是调用的就是RemotingCommand 的解码

public class NettyDecoder extends LengthFieldBasedFrameDecoder {
    private static final InternalLogger log = InternalLoggerFactory.getLogger(RemotingHelper.ROCKETMQ_REMOTING);

    private static final int FRAME_MAX_LENGTH =
        Integer.parseInt(System.getProperty("com.rocketmq.remoting.frameMaxLength", "16777216"));

    public NettyDecoder() {
        super(FRAME_MAX_LENGTH, 0, 4, 0, 4);
    }

    @Override
    public Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
        ByteBuf frame = null;
        try {
            frame = (ByteBuf) super.decode(ctx, in);
            if (null == frame) {
                return null;
            }
            return RemotingCommand.decode(frame);
        } catch (Exception e) {
            log.error("decode exception, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()), e);
            RemotingUtil.closeChannel(ctx.channel());
        } finally {
            if (null != frame) {
                frame.release();
            }
        }

        return null;
    }
}

4.如何让客户端,比如Namesrv在监听到连接断开后做自己逻辑处理-NettyConnectManageHandler

比如Namesrv在监听到channel断开后需要清除broker的routeInfo信息，而broker在监听到channel断开后，也需要做自己的处理。这个是如何实现的呢？

NettyRomotingServer在Netty的channelpipeline中加入了NettyConnectManageHandler，这个组件会监听到channel所触发的事件，比如是Close事件。便会将其打包成一个NettyEvent，并且放到NettyEventExecutor中的一个LinkedBlockingQueue中，并且启动该线程池一直消费该事件，来调用注册到这里面的ChannelEventListener方法中对应的onChannelClose进行处理，每次构造一个NettyRemotingServer的时候，会传入ChannelEventListener，比如Namesrv就是BrokerHousekeepingService。

//接收channel事件，并且启动线程池来根据事件来分发,调用listener来处理时事件 
class NettyEventExecutor extends ServiceThread {
    //一个阻塞队列来接收channel相关事件
        private final LinkedBlockingQueue<NettyEvent> eventQueue = new LinkedBlockingQueue<NettyEvent>();
        private final int maxSize = 10000;

        public void putNettyEvent(final NettyEvent event) {
            int currentSize = this.eventQueue.size();
            if (currentSize <= maxSize) {
                this.eventQueue.add(event);
            } else {
                log.warn("event queue size [{}] over the limit [{}], so drop this event {}", currentSize, maxSize, event.toString());
            }
        }

        @Override
        public void run() {
            log.info(this.getServiceName() + " service started");

            final ChannelEventListener listener = NettyRemotingAbstract.this.getChannelEventListener();

            while (!this.isStopped()) {
                try {
                    NettyEvent event = this.eventQueue.poll(3000, TimeUnit.MILLISECONDS);
                    if (event != null && listener != null) {
                        switch (event.getType()) {
                            case IDLE:
                                //根据对应事件，调用对应evetentListener来做处理
                                listener.onChannelIdle(event.getRemoteAddr(), event.getChannel());
                                break;
                            case CLOSE:
                                listener.onChannelClose(event.getRemoteAddr(), event.getChannel());
                                break;
                            case CONNECT:
                                listener.onChannelConnect(event.getRemoteAddr(), event.getChannel());
                                break;
                            case EXCEPTION:
                                listener.onChannelException(event.getRemoteAddr(), event.getChannel());
                                break;
                            default:
                                break;

                        }
                    }
                } catch (Exception e) {
                    log.warn(this.getServiceName() + " service has exception. ", e);
                }
            }

            log.info(this.getServiceName() + " service end");
        }

        @Override
        public String getServiceName() {
            return NettyEventExecutor.class.getSimpleName();
        }
    }

//获取到当前remotingServer中的eventListener    
public abstract ChannelEventListener getChannelEventListener();

//将事件放入到eventExecutor中
 public void putNettyEvent(final NettyEvent event) {
        this.nettyEventExecutor.putNettyEvent(event);
 }

5.netty的基本配置-NettyServerConfig

netty的基本配置，可以提供使用方配置nettyRemotingServer

public class NettyServerConfig implements Cloneable {
    //namesrv监听端口号
    private int listenPort = 8888;
    //netty的worker线程数量
    private int serverWorkerThreads = 8;
    private int serverCallbackExecutorThreads = 0;
    //netty中bossGroup数量
    private int serverSelectorThreads = 3;
    //最多同时运行发送多少个oneWay请求
    private int serverOnewaySemaphoreValue = 256;
    //最多同时发送多少个异步请求
    private int serverAsyncSemaphoreValue = 64;
    //netty长连接最大的空闲时间
    private int serverChannelMaxIdleTimeSeconds = 120;

    //netty发送方的bytebuffer的大小
    private int serverSocketSndBufSize = NettySystemConfig.socketSndbufSize;
    //netty接收方的bytebuffer的大小
    private int serverSocketRcvBufSize = NettySystemConfig.socketRcvbufSize;
    //netty的流量控制参数，当超过高水位时，channel将不可写，直到地域低水位时，channel才会变成可写状态
    private int writeBufferHighWaterMark = NettySystemConfig.writeBufferHighWaterMark;
    private int writeBufferLowWaterMark = NettySystemConfig.writeBufferLowWaterMark;
    private int serverSocketBacklog = NettySystemConfig.socketBacklog;
    //在netty中是否开启池化，开启池化后，当bytebuffer的数据被写完过后，不会立刻释放，而达到内存重复利用
    private boolean serverPooledByteBufAllocatorEnable = true;
}

6.服务端如何处理器请求和连接-NettyServerHandler

  class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {

        @Override
        protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
            processMessageReceived(ctx, msg);
        }
    }

NettyServerHandler是pipeline的最后一个handler，是做逻辑处理的地方。它其实调用的就是NettyRemotingAbstract的processMessageReceived方法:

 public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
        final RemotingCommand cmd = msg;
        if (cmd != null) {
            switch (cmd.getType()) {
                case REQUEST_COMMAND:
                    processRequestCommand(ctx, cmd);
                    break;
                case RESPONSE_COMMAND:
                    processResponseCommand(ctx, cmd);
                    break;
                default:
                    break;
            }
        }
    }

可以看出它其实就是对收到的信息进行判断，看是请求还是响应，netty是一个双向的通信软件，比如broker向namesrv请求删除某个topic，namesrv收到的就是一个请求，namesrv删除后会给broker一个答复，此时broker收到的就是一个响应。

如果收到的是请求：

 public void processRequestCommand(final ChannelHandlerContext ctx, final RemotingCommand cmd) {
        //根据请求类型获取到请求的processor
        final Pair<NettyRequestProcessor, ExecutorService> matched = this.processorTable.get(cmd.getCode());
        final Pair<NettyRequestProcessor, ExecutorService> pair = null == matched ? this.defaultRequestProcessor : matched;
        final int opaque = cmd.getOpaque();

        if (pair != null) {
            Runnable run = new Runnable() {
                @Override
                public void run() {
                    try {
                        String remoteAddr = RemotingHelper.parseChannelRemoteAddr(ctx.channel());
                        //执行请求前的拦截器
                        doBeforeRpcHooks(remoteAddr, cmd);
                        //这个是该请求在服务端处理完过后，需要执行的拦截器，切实就是将opqueue和响应类型放入到response里面
                        final RemotingResponseCallback callback = new RemotingResponseCallback() {
                            @Override
                            public void callback(RemotingCommand response) {
                                doAfterRpcHooks(remoteAddr, cmd, response);
                                if (!cmd.isOnewayRPC()) {
                                    if (response != null) {
                                        response.setOpaque(opaque);
                                        response.markResponseType();
                                        response.setSerializeTypeCurrentRPC(cmd.getSerializeTypeCurrentRPC());
                                        try {
                                            ctx.writeAndFlush(response);
                                        } catch (Throwable e) {
                                            log.error("process request over, but response failed", e);
                                            log.error(cmd.toString());
                                            log.error(response.toString());
                                        }
                                    } else {
                                    }
                                }
                            }
                        };
                        //异步请求执行该逻辑
                        if (pair.getObject1() instanceof AsyncNettyRequestProcessor) {
                            AsyncNettyRequestProcessor processor = (AsyncNettyRequestProcessor)pair.getObject1();
                            //调用客户端的processRequest方法，执行对请求的响应
                            processor.asyncProcessRequest(ctx, cmd, callback);
                        } else {
                            //同步请求执行该逻辑
                            NettyRequestProcessor processor = pair.getObject1();
                            RemotingCommand response = processor.processRequest(ctx, cmd);
                            callback.callback(response);
                        }
                    } 
				}
			}
		}
		
	}

如果收到的是响应：

 public void processResponseCommand(ChannelHandlerContext ctx, RemotingCommand cmd) {
        final int opaque = cmd.getOpaque();
        //根据响应获取到发送请求时塞入的回调逻辑
        final ResponseFuture responseFuture = responseTable.get(opaque);
        if (responseFuture != null) {
            //将响应加入到responseCommand中
            responseFuture.setResponseCommand(cmd);

            responseTable.remove(opaque);

            if (responseFuture.getInvokeCallback() != null) {
                //如果是异步请求，需要执行回调逻辑，即调用方的业务逻辑
                executeInvokeCallback(responseFuture);
            } else {   
                //同步请求，通过countDownLatch唤醒发起调用的线程
                responseFuture.putResponse(cmd);
                responseFuture.release();
            }
        } else {
            log.warn("receive response, but not matched any request, " + RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
            log.warn(cmd.toString());
        }
    }

 //其实就是异步调用，调用方在调用的时候，会塞入回调函数，当接收到服务端响应过后，利用在callBackExecutor中执行回调逻辑
    private void executeInvokeCallback(final ResponseFuture responseFuture) {
        boolean runInThisThread = false;
        ExecutorService executor = this.getCallbackExecutor();
        if (executor != null) {
            try {
                executor.submit(new Runnable() {
                    @Override
                    public void run() {
                        try {
                            responseFuture.executeInvokeCallback();
                        } catch (Throwable e) {
                            log.warn("execute callback in executor exception, and callback throw", e);
                        } finally {
                            responseFuture.release();
                        }
                    }
                });
            } catch (Exception e) {
                runInThisThread = true;
                log.warn("execute callback in executor exception, maybe executor busy", e);
            }
        } else {
            runInThisThread = true;
        }

        if (runInThisThread) {
            try {
                responseFuture.executeInvokeCallback();
            } catch (Throwable e) {
                log.warn("executeInvokeCallback Exception", e);
            } finally {
                responseFuture.release();
            }
        }
    }

7.调用方如何里利用NettyRomotingServer发起调用

NettyRomotingServer提供了三种调用模式分别是同步调用、异步调用、onway调用。

同步调用:

    public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
        final long timeoutMillis)
        throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
        //获取请求的唯一id
        final int opaque = request.getOpaque();

        try {
            //设置responseFuture，便是在响应回来的时候，交给ResponseFutre
            final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
            //放入到responseTable中
            this.responseTable.put(opaque, responseFuture);
            final SocketAddress addr = channel.remoteAddress();
            
            channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                @Override
                public void operationComplete(ChannelFuture f) throws Exception {
                    if (f.isSuccess()) {
                        responseFuture.setSendRequestOK(true);
                        return;
                    } else {
                        responseFuture.setSendRequestOK(false);
                    }
                    //当响应完成过后，设置响应结果
                    responseTable.remove(opaque);
                    responseFuture.setCause(f.cause());
                    responseFuture.putResponse(null);
                    log.warn("send a request command to channel <" + addr + "> failed.");
                }
            });
            //同步调用的核心，会阻塞当前线程，一直到结果返回
            RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
            if (null == responseCommand) {
                if (responseFuture.isSendRequestOK()) {
                    throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
                        responseFuture.getCause());
                } else {
                    throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
                }
            }

            return responseCommand;
        } finally {
            this.responseTable.remove(opaque);
        }
    }

异步调用:

public void invokeAsyncImpl(final Channel channel, final RemotingCommand request, final long timeoutMillis,
        final InvokeCallback invokeCallback)
        throws InterruptedException, RemotingTooMuchRequestException, RemotingTimeoutException, RemotingSendRequestException {
        long beginStartTime = System.currentTimeMillis();
        final int opaque = request.getOpaque();
        //异步调用会通过信号量限制请求数量，先获取到信号量
        boolean acquired = this.semaphoreAsync.tryAcquire(timeoutMillis, TimeUnit.MILLISECONDS);
        if (acquired) {
            final SemaphoreReleaseOnlyOnce once = new SemaphoreReleaseOnlyOnce(this.semaphoreAsync);
            long costTime = System.currentTimeMillis() - beginStartTime;
            //如果获取时间超过异步调用的超时时间，会释放信号量，并且抛出异常
            if (timeoutMillis < costTime) {
                once.release();
                throw new RemotingTimeoutException("invokeAsyncImpl call timeout");
            }
            //设置responseFuture,这里会给responseFuture中设置等待该调用返回的超时时间，会启动一个定时线程去扫描，如果超时会抛出异常
            final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis - costTime, invokeCallback, once);
            this.responseTable.put(opaque, responseFuture);
            try {
                //设置调用成功的回调函数
                channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture f) throws Exception {
                        if (f.isSuccess()) {
                            //设置responseFuture的状态为OK
                            responseFuture.setSendRequestOK(true);
                            return;
                        }
                        requestFail(opaque);
                        log.warn("send a request command to channel <{}> failed.", RemotingHelper.parseChannelRemoteAddr(channel));
                    }
                });
            } catch (Exception e) {
                responseFuture.release();
                log.warn("send a request command to channel <" + RemotingHelper.parseChannelRemoteAddr(channel) + "> Exception", e);
                throw new RemotingSendRequestException(RemotingHelper.parseChannelRemoteAddr(channel), e);
            }
        } else {
            if (timeoutMillis <= 0) {
                throw new RemotingTooMuchRequestException("invokeAsyncImpl invoke too fast");
            } else {
                String info =
                    String.format("invokeAsyncImpl tryAcquire semaphore timeout, %dms, waiting thread nums: %d semaphoreAsyncValue: %d",
                        timeoutMillis,
                        this.semaphoreAsync.getQueueLength(),
                        this.semaphoreAsync.availablePermits()
                    );
                log.warn(info);
                throw new RemotingTimeoutException(info);
            }
        }
    }

oneway调用：就是发送请求后，不关心结果

8.如果调用超过了设置的超时时间，怎么处理?

nettyRemotingServer在start的时候有这样一行代码:其实就是开启一个定时任务去扫描responeTable，并且启动线程去执行reponseFuture中的回调逻辑。由于此时响应还没有回来，所以响应码并不是成功，在执行executeInvokeCallback会多错误进行处理。

this.timer.scheduleAtFixedRate(new TimerTask() {

            @Override
            public void run() {
                try {
                    NettyRemotingServer.this.scanResponseTable();
                } catch (Throwable e) {
                    log.error("scanResponseTable exception", e);
                }
            }
        }, 1000 * 3, 1000);

   public void scanResponseTable() {
        final List<ResponseFuture> rfList = new LinkedList<ResponseFuture>();
        //变量responseTable
        Iterator<Entry<Integer, ResponseFuture>> it = this.responseTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<Integer, ResponseFuture> next = it.next();
            ResponseFuture rep = next.getValue();
            //计算调用开始时间+超时时间是否是超过当前时间，如果没有超时，便会从当前responseTable中移除，同时执行调用方的响应逻辑
            if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
                rep.release();
                it.remove();
                rfList.add(rep);
                log.warn("remove timeout request, " + rep);
            }
        }

        for (ResponseFuture rf : rfList) {
            try {
                //同时一会执行调用方的回调方法（也即调用方的响应逻辑）
                executeInvokeCallback(rf);
            } catch (Throwable e) {
                log.warn("scanResponseTable, operationComplete Exception", e);
            }
        }
    }

3.2 调用法如何利用NettyRemotingServer进行网络连接

3.2.1 构造一个NettyRemotingServer，并且启动该服务器

   //1.构NettyRemotingServer
        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);

3.2.2 实现processor，并且注册到remotingServer中

public class DefaultRequestProcessor extends AsyncNettyRequestProcessor implements NettyRequestProcessor {
    private static InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);

    protected final NamesrvController namesrvController;

    protected Set<String> configBlackList = new HashSet<>();

    public DefaultRequestProcessor(NamesrvController namesrvController) {
        this.namesrvController = namesrvController;
        initConfigBlackList();
    }
}

//启动remotingServer

this.remotingServer.start();

 private void registerProcessor() {
        if (namesrvConfig.isClusterTest()) {

            this.remotingServer.registerDefaultProcessor(new ClusterTestRequestProcessor(this, namesrvConfig.getProductEnvName()),
                this.remotingExecutor);
        } else {

            this.remotingServer.registerDefaultProcessor(new DefaultRequestProcessor(this), this.remotingExecutor);
        }
    }

上面其实可以在processor中实现接收到请求后对请求进行处理。

3.2.3 利用remotingServer的invokeSync方法发送请求

  public RemotingCommand callClient(final Channel channel,
                                      final RemotingCommand request
    ) throws RemotingSendRequestException, RemotingTimeoutException, InterruptedException {
        return this.brokerController.getRemotingServer().invokeSync(channel, request, 10000);
    }