Netty通讯

1.当服务端启动之后，监听端口的线程NioEventLoop在持续运行，当客户端的Channel channel = bootstrap.connect(host, port).sync().channel();-》socketChannel.connect(remoteAddress);请求到达服务端时，客户端连接的请求对于服务端就是SelectionKey.OP_ACCEPT，在NioEventLoop线程的run方法的processSelectedKey方法中

 if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }

NioServerSocketChannel的父类AbstractNioMessageChannel的内部类NioMessageUnsafe，判断是否网络配置NioServerSocketChannelConfig是否自动读以及准备读的标志readPending是否为true

public void read() {
            assert eventLoop().inEventLoop();
            final ChannelConfig config = config();
            if (!config.isAutoRead() && !isReadPending()) {
                // ChannelConfig.setAutoRead(false) was called in the meantime
                removeReadOp();
                return;
            }

            final int maxMessagesPerRead = config.getMaxMessagesPerRead();
            final ChannelPipeline pipeline = pipeline();
            boolean closed = false;
            Throwable exception = null;
            try {
                try {
                    for (;;) {
                        int localRead = doReadMessages(readBuf);
                        if (localRead == 0) {
                            break;
                        }
                        if (localRead < 0) {
                            closed = true;
                            break;
                        }

                        // stop reading and remove op
                        if (!config.isAutoRead()) {
                            break;
                        }

                        if (readBuf.size() >= maxMessagesPerRead) {
                            break;
                        }
                    }
                } catch (Throwable t) {
                    exception = t;
                }
                setReadPending(false);
                int size = readBuf.size();
                for (int i = 0; i < size; i ++) {
                    pipeline.fireChannelRead(readBuf.get(i));
                }

                readBuf.clear();
                pipeline.fireChannelReadComplete();
            } 
        }

当握手连接成功之后，为该通道SocketChannel分配一个NioSocketChannel单独的线程对应该客户端的后续请求，doReadMessages第二次循环就会返回，因为这时没有别的客户端连接请求，默认的一次处理的最大量是maxMessagesPerRead=16

 protected int doReadMessages(List<Object> buf) throws Exception {
        SocketChannel ch = SocketUtils.accept(javaChannel());

        try {
            if (ch != null) {
                buf.add(new NioSocketChannel(this, ch));
                return 1;
            }
        } 
        return 0;
    }

管道开始触发fireChannelRead，过程基本上都是类似，DefaultChannelPipeline-》AbstractChannelHandlerContext.invokeChannelRead-》HeadContext-》

public final ChannelPipeline fireChannelRead(Object msg) {
        AbstractChannelHandlerContext.invokeChannelRead(head, msg);
        return this;
    }

AbstractChannelHandlerContext，这里的findContextInbound()是ServerBootstrapAcceptor

 public ChannelHandlerContext fireChannelRead(final Object msg) {
        invokeChannelRead(findContextInbound(), msg);
        return this;
    }

ServerBootstrapAcceptor中会给该通道的处理管道上加上服务端main函数里配置的childHandler，childOptions，childAttrs

 public void channelRead(ChannelHandlerContext ctx, Object msg) {
            final Channel child = (Channel) msg;

            child.pipeline().addLast(childHandler);

            setChannelOptions(child, childOptions, logger);

            for (Entry<AttributeKey<?>, Object> e: childAttrs) {
                child.attr((AttributeKey<Object>) e.getKey()).set(e.getValue());
            }

            try {
                childGroup.register(child).addListener(new ChannelFutureListener() {
                    @Override
                    public void operationComplete(ChannelFuture future) throws Exception {
                        if (!future.isSuccess()) {
                            forceClose(child, future.cause());
                        }
                    }
                });
            } catch (Throwable t) {
                forceClose(child, t);
            }
        }

然后把这个通道注册到子线程池childGroup，然后后续与该客户端的读写请求都通过该线程来处理。注册过程和之前的类似。

2.当服务端处理完客户端的链接请求之后，客户端的processSelectedKey的readyOps = k.readyOps() = 8也就是链接请求。客户端处理连接成功请求pipeline().fireChannelActive();等等，.sync().channel()就会返回建立的通道。

if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }

3.客户端王服务写请求 channel.writeAndFlush(msg).sync();-》AbstractChannel-》

 public ChannelFuture writeAndFlush(Object msg) {
        return pipeline.writeAndFlush(msg);
    }

DefaultChannelPipeline-》

public final ChannelFuture writeAndFlush(Object msg) {
        return tail.writeAndFlush(msg);
    }

AbstractChannelHandlerContext， write(msg, true, promise);

public ChannelFuture writeAndFlush(Object msg) {
        return writeAndFlush(msg, newPromise());
    }

这个时候当前线程是main主线程，next 为name为encoder的StringEncoder也就是客户端示例程序的处理器

 private void write(Object msg, boolean flush, ChannelPromise promise) {
        AbstractChannelHandlerContext next = findContextOutbound();
        EventExecutor executor = next.executor();
        if (executor.inEventLoop()) {
            if (flush) {
                next.invokeWriteAndFlush(msg, promise);
            } else {
                next.invokeWrite(msg, promise);
            }
        } else {
            AbstractWriteTask task;
            if (flush) {
                task = WriteAndFlushTask.newInstance(next, msg, promise);
            }  else {
                task = WriteTask.newInstance(next, msg, promise);
            }
            safeExecute(executor, task, promise, msg);
        }
    }

把返回值，消息，管道处理器包装成一个WriteAndFlushTask，这里Recycler<WriteAndFlushTask>()充当一个可循环利用的对象，netty的优化

private static WriteAndFlushTask newInstance(
                AbstractChannelHandlerContext ctx, Object msg,  ChannelPromise promise) {
            WriteAndFlushTask task = RECYCLER.get();
            init(task, ctx, msg, promise);
            return task;
        }

构造ChannelOutboundBuffer，检查要写的消息长度，检验是否超过限制等等，然后把该任务放入处理器所在的这个线程中处理executor.execute(runnable);

if (ESTIMATE_TASK_SIZE_ON_SUBMIT) {
                ChannelOutboundBuffer buffer = ctx.channel().unsafe().outboundBuffer();

                // Check for null as it may be set to null if the channel is closed already
                if (buffer != null) {
                    task.size = ctx.pipeline.estimatorHandle().size(msg) + WRITE_TASK_OVERHEAD;
                    buffer.incrementPendingOutboundBytes(task.size);
                } else {
                    task.size = 0;
                }
            } 

4.整个写入过程是非阻塞的，主线程开启另一个线程进行和服务器端通讯，这个过程会触发这个线程第一次启动，也就是NioEventLoop的父类SingleThreadEventExecutor的thread.start();线程的run函数中有runAllTasks(ioTime * (100 - ioRatio) / ioRatio);用来执行这些除了与服务端交互的哪几种事件任务外的其他io事情。提交任务的时候会把任务放进队列中taskQueue.offer(task);然后根据条件决定是否触发 selector.wakeup();

public void execute(Runnable task) {
        boolean inEventLoop = inEventLoop();
        if (inEventLoop) {
            addTask(task);
        } else {
            startThread();
            addTask(task);
            if (isShutdown() && removeTask(task)) {
                reject();
            }
        }

        if (!addTaskWakesUp && wakesUpForTask(task)) {
            wakeup(inEventLoop);
        }
    }

5.当每次需要执行任务时，就会从定时任务队列中取出需要执行的任务fetchFromScheduledTaskQueue，然后放入需要执行的任务队列，如果设置了执行时间的比率，则需要计算task执行的时间，因为nanoTime操作获取比较昂贵，所以没64次计算一次是否超时，决定是否退出执行task，转而执行网络事件请求。

 protected boolean runAllTasks(long timeoutNanos) {
        fetchFromScheduledTaskQueue();
        Runnable task = pollTask();
        if (task == null) {
            return false;
        }

        final long deadline = ScheduledFutureTask.nanoTime() + timeoutNanos;
        long runTasks = 0;
        long lastExecutionTime;
        for (;;) {
            try {
                task.run();
            } catch (Throwable t) {
                logger.warn("A task raised an exception.", t);
            }

            runTasks ++;

            // Check timeout every 64 tasks because nanoTime() is relatively expensive.
            // XXX: Hard-coded value - will make it configurable if it is really a problem.
            if ((runTasks & 0x3F) == 0) {
                lastExecutionTime = ScheduledFutureTask.nanoTime();
                if (lastExecutionTime >= deadline) {
                    break;
                }
            }

            task = pollTask();
            if (task == null) {
                lastExecutionTime = ScheduledFutureTask.nanoTime();
                break;
            }
        }

        this.lastExecutionTime = lastExecutionTime;
        return true;
    }

6.AbstractChannelHandlerContext的内部类AbstractWriteTask，针对输出参数大小做一些简单的处理，写完之后进行资源回收，清空。

public final void run() {
            try {
                ChannelOutboundBuffer buffer = ctx.channel().unsafe().outboundBuffer();
                // Check for null as it may be set to null if the channel is closed already
                if (ESTIMATE_TASK_SIZE_ON_SUBMIT && buffer != null) {
                    buffer.decrementPendingOutboundBytes(size);
                }
                write(ctx, msg, promise);
            } finally {
                // Set to null so the GC can collect them directly
                ctx = null;
                msg = null;
                promise = null;
                recycle(handle);
            }
        }

WriteAndFlushTask-》ctx.invokeWrite(msg, promise);

 public void write(AbstractChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
            super.write(ctx, msg, promise);
            ctx.invokeFlush();
        }

private void invokeWrite(Object msg, ChannelPromise promise) {
        if (invokeHandler()) {
            invokeWrite0(msg, promise);
        } else {
            write(msg, promise);
        }
    }

-》((ChannelOutboundHandler) handler()).write(this, msg, promise);--》MessageToMessageEncoder，进行消息匹配，-》StringEncoder的out.add(ByteBufUtil.encodeString(ctx.alloc(), CharBuffer.wrap(msg), charset));-》ctx.write(out.get(0), promise);

public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
        CodecOutputList out = null;
        try {
            if (acceptOutboundMessage(msg)) {
                out = CodecOutputList.newInstance();
                @SuppressWarnings("unchecked")
                I cast = (I) msg;
                try {
                    encode(ctx, cast, out);
                } 
            } 
        }finally {
            if (out != null) {
                final int sizeMinusOne = out.size() - 1;
                if (sizeMinusOne == 0) {
                    ctx.write(out.get(0), promise);
                } 
                out.recycle();
            }
        }
    }

-》AbstractChannelHandlerContext的write(msg, false, promise);-》这时执行线程就是当前线程，所以直接执行 next.invokeWrite(msg, promise);-》invokeWrite0(msg, promise);-》((ChannelOutboundHandler) handler()).write(this, msg, promise);-》MessageToByteEncoder分配内存super.allocateBuffer(ctx, msg, preferDirect).order(byteOrder);，这里会决定使用堆外直接内存还是堆内存。

public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
        ByteBuf buf = null;
        try {
            if (acceptOutboundMessage(msg)) {
                @SuppressWarnings("unchecked")
                I cast = (I) msg;
                buf = allocateBuffer(ctx, cast, preferDirect);
                try {
                    encode(ctx, cast, buf);
                } finally {
                    ReferenceCountUtil.release(cast);
                }

                if (buf.isReadable()) {
                    ctx.write(buf, promise);
                } else {
                    buf.release();
                    ctx.write(Unpooled.EMPTY_BUFFER, promise);
                }
                buf = null;
            } else {
                ctx.write(msg, promise);
            }
        } catch (EncoderException e) {
            throw e;
        } catch (Throwable e) {
            throw new EncoderException(e);
        } finally {
            if (buf != null) {
                buf.release();
            }
        }
    }

往内存ByteBuf中写入消息 out.writeBytes(msg, msg.readerIndex(), msg.readableBytes());-》ctx.write(buf, promise);-》HeadContext的 unsafe.write(msg, promise);-》AbstractChannel的 outboundBuffer.addMessage(msg, size, promise);消息写入内存完成

public final void write(Object msg, ChannelPromise promise) {
            assertEventLoop();

            ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
            if (outboundBuffer == null) {
            int size;
            try {
                msg = filterOutboundMessage(msg);
                size = pipeline.estimatorHandle().size(msg);
                if (size < 0) {
                    size = 0;
                }
            } catch (Throwable t) {
                safeSetFailure(promise, t);
                ReferenceCountUtil.release(msg);
                return;
            }

            outboundBuffer.addMessage(msg, size, promise);
        }

7.开始把消息刷入通道ctx.invokeFlush();-》 invokeFlush0();-》((ChannelOutboundHandler) handler()).flush(this);-》ChannelOutboundHandlerAdapter的ctx.flush();-》AbstractChannelHandlerContext的 next.invokeFlush();-》((ChannelOutboundHandler) handler()).flush(this);-》最后到达HeadContext的unsafe.flush();-》AbstractChannel内部类AbstractUnsafe的flush0()-》 doWrite(outboundBuffer);-》

 public final void flush() {
            assertEventLoop();

            ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
            if (outboundBuffer == null) {
                return;
            }

            outboundBuffer.addFlush();
            flush0();
        }

NioSocketChannel，获取通道javaChannel()，正式写入通道ch.write(nioBuffer);释放内存 in.removeBytes(writtenBytes);

protected void doWrite(ChannelOutboundBuffer in) throws Exception {
        for (;;) {
            int size = in.size();
            if (size == 0) {
                // All written so clear OP_WRITE
                clearOpWrite();
                break;
            }
            long writtenBytes = 0;
            boolean done = false;
            boolean setOpWrite = false;

            // Ensure the pending writes are made of ByteBufs only.
            ByteBuffer[] nioBuffers = in.nioBuffers();
            int nioBufferCnt = in.nioBufferCount();
            long expectedWrittenBytes = in.nioBufferSize();
            SocketChannel ch = javaChannel();

            // Always us nioBuffers() to workaround data-corruption.
            // See https://github.com/netty/netty/issues/2761
            switch (nioBufferCnt) {
                case 0:
                    // We have something else beside ByteBuffers to write so fallback to normal writes.
                    super.doWrite(in);
                    return;
                case 1:
                    // Only one ByteBuf so use non-gathering write
                    ByteBuffer nioBuffer = nioBuffers[0];
                    for (int i = config().getWriteSpinCount() - 1; i >= 0; i --) {
                        final int localWrittenBytes = ch.write(nioBuffer);
                        if (localWrittenBytes == 0) {
                            setOpWrite = true;
                            break;
                        }
                        expectedWrittenBytes -= localWrittenBytes;
                        writtenBytes += localWrittenBytes;
                        if (expectedWrittenBytes == 0) {
                            done = true;
                            break;
                        }
                    }
                    break;
            }

            // Release the fully written buffers, and update the indexes of the partially written buffer.
            in.removeBytes(writtenBytes);
        }
    }

8.服务端读消息NioEventLoop的processSelectedKey-》AbstractNioByteChannel的NioByteUnsafe的 pipeline.fireChannelRead(byteBuf);

 if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }

开始进入管道处理-》DefaultChannelPipeline-》AbstractChannelHandlerContext的 next.invokeChannelRead(msg);

public final ChannelPipeline fireChannelRead(Object msg) {
        AbstractChannelHandlerContext.invokeChannelRead(head, msg);
        return this;
    }

然后依次通过处理器处理 ((ChannelInboundHandler) handler()).channelRead(this, msg);-》HeadContext-》LengthFieldBasedFrameDecoder-》StringDecoder->TcpServerHandler。释放回收资源，整个过程就结束了。