netty源码分析(二)－处理请求

上一篇对netty的启动过程做了分析，netty源码分析(一)-启动.

本篇将对netty处理请求的主要过程进行源码层面分析。

根据上一篇的最后部分内容我们知道，netty启动后会不断循环accept请求

        public void run() {
            final Thread currentThread = Thread.currentThread();

            channel.shutdownLock.lock();
            try {
                for (;;) {
                    try {
                        if (selector.select(1000) > 0) {
                            selector.selectedKeys().clear();
                        }

                        SocketChannel acceptedSocket = channel.socket.accept();
                        if (acceptedSocket != null) {
                            registerAcceptedChannel(acceptedSocket, currentThread);
                        }
                    } catch (SocketTimeoutException e) {
                        ......
                    }
                }
            } finally {
                channel.shutdownLock.unlock();
                closeSelector();
            }
        }

        private void registerAcceptedChannel(SocketChannel acceptedSocket, Thread currentThread) {
            try {
                ChannelPipeline pipeline =
                    channel.getConfig().getPipelineFactory().getPipeline();
                NioWorker worker = nextWorker();
                worker.register(new NioAcceptedSocketChannel(
                        channel.getFactory(), pipeline, channel,
                        NioServerSocketPipelineSink.this, acceptedSocket,
                        worker, currentThread), null);
            } catch (Exception e) {
                logger.warn(
                        "Failed to initialize an accepted socket.", e);
                try {
                    acceptedSocket.close();
                } catch (IOException e2) {
                    logger.warn(
                            "Failed to close a partially accepted socket.",
                            e2);
                }
            }
        }

        private void closeSelector() {
            channel.selector = null;
            try {
                selector.close();
            } catch (Exception e) {
                logger.warn("Failed to close a selector.", e);
            }
        }
    }

如上，接受到的socket连接请求会扔给registerAcceptedChannel()方法处理。

registerAcceptedChannel内具体做什么呢？

首先，轮询指定一个NioWorker，NioWorker的数量在初始化的时候由用户指定，默认是处理器个数＊2。

然后，有请求进来，自然需要有对应接收处理请求的channel，于是根据serversocketchannel，pipeline，acceptsocket等参数创建NioAcceptedSocketChannel的对象，注意此时serversocketchannel是NioAcceptedSocketChannel对象的parent channel。同时触发NioAcceptedSocketChannel的fireChannelOpen事件，通知所有的upstream handler完成channelOpen操作。

最后，将这个NioAcceptedSocketChannel对象注册到前面选定的nioworker进行处理。

    void register(NioSocketChannel channel, ChannelFuture future) {

        boolean server = !(channel instanceof NioClientSocketChannel);
        Runnable registerTask = new RegisterTask(channel, future, server);
        Selector selector;

        synchronized (startStopLock) {

            if (!started) {
                // Open a selector if this worker didn't start yet.
                try {
                    this.selector = selector = Selector.open();
                } catch (Throwable t) {
                    throw new ChannelException(
                            "Failed to create a selector.", t);
                }

                ......
                try {
                    //这里启动nioworker线程
                    DeadLockProofWorker.start(
                            executor, new ThreadRenamingRunnable(this, threadName));
                    success = true;
                } finally {
                    ......
                }
            } else {
                // Use the existing selector if this worker has been started.
                selector = this.selector;
            }

            assert selector != null && selector.isOpen();

            started = true;
            boolean offered = registerTaskQueue.offer(registerTask);
            assert offered;
        }

        if (wakenUp.compareAndSet(false, true)) {
            selector.wakeup();
        }
    }

acceptsocketchannel注册到nioworker线程的过程：

先生成RegisterTask对象，并添加到registerTaskQueue的队列中，注册只做这一个事情；如果注册时发现nioworker的线程执行没有启动，就会同时启动nioworker线程。启动过程也是线程以ThreadLocal的方式获取worker线程池启动线程，关于ThreadLocal的理解，可以参考这里

注册过程完成了，Boss线程就继续accept新的请求，剩下的任务就由nioworker线程处理了。

NioWorker处理请求过程：

主要是下面run()部分代码：

    public void run() {
        thread = Thread.currentThread();

        boolean shutdown = false;
        Selector selector = this.selector;
        for (;;) {
            ......

            try {

                SelectorUtil.select(selector);
                ......

                cancelledKeys = 0;
                processRegisterTaskQueue();
                processWriteTaskQueue();
                processSelectedKeys(selector.selectedKeys());

                ......
            } catch (Throwable t) {
                ......
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException e) {
                    // Ignore.
                }
            }
        }
    }

processRegisterTaskQueue()是从registerTaskQueue阻塞队列中获取RegisterTask对象并执行task。

那RegisterTask对象是干嘛用的呢？

    private final class RegisterTask implements Runnable {
        ......
        public void run() {
            SocketAddress localAddress = channel.getLocalAddress();
            SocketAddress remoteAddress = channel.getRemoteAddress();
            if (localAddress == null || remoteAddress == null) {
                if (future != null) {
                    future.setFailure(new ClosedChannelException());
                }
                close(channel, succeededFuture(channel));
                return;
            }

            try {
                if (server) {
                    channel.socket.configureBlocking(false);
                }

                synchronized (channel.interestOpsLock) {
                     //channel注册到selector
                    channel.socket.register(
                            selector, channel.getRawInterestOps(), channel);
                }
                if (future != null) {
                    channel.setConnected();
                    future.setSuccess();
                }
            } catch (IOException e) {
                .....
            }

            if (server || !((NioClientSocketChannel) channel).boundManually) {
                fireChannelBound(channel, localAddress);
            }
            fireChannelConnected(channel, remoteAddress);
        }
    }

1.将channel注册到nioworker的selector中；2.fireChannelConnected触发通知到所有channel pipeline的upstream handler。

processWriteTaskQueue()是执行写response任务的，这里先略过。

processSelectedKeys()是最终处理channel注册到selector感兴趣的事件就绪的处理过程。

    private void processSelectedKeys(Set<SelectionKey> selectedKeys) throws IOException {
        for (Iterator<SelectionKey> i = selectedKeys.iterator(); i.hasNext();) {
            SelectionKey k = i.next();
            i.remove();
            try {
                int readyOps = k.readyOps();
                if ((readyOps & SelectionKey.OP_READ) != 0 || readyOps == 0) {
                    if (!read(k)) {
                        // Connection already closed - no need to handle write.
                        continue;
                    }
                }
                if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                    writeFromSelectorLoop(k);
                }
            } catch (CancelledKeyException e) {
                close(k);
            }

            if (cleanUpCancelledKeys()) {
                break; // break the loop to avoid ConcurrentModificationException
            }
        }
    }

当注册到selector中的channel的read事件就绪时，处理对应channel的read操作

    private boolean read(SelectionKey k) {
        final SocketChannel ch = (SocketChannel) k.channel();
        final NioSocketChannel channel = (NioSocketChannel) k.attachment();

        final ReceiveBufferSizePredictor predictor =
            channel.getConfig().getReceiveBufferSizePredictor();
        final int predictedRecvBufSize = predictor.nextReceiveBufferSize();

        int ret = 0;
        int readBytes = 0;
        boolean failure = true;

        ByteBuffer bb = recvBufferPool.acquire(predictedRecvBufSize);
        try {
            while ((ret = ch.read(bb)) > 0) {
                readBytes += ret;
                if (!bb.hasRemaining()) {
                    break;
                }
            }
            failure = false;
        } catch (ClosedChannelException e) {
            // Can happen, and does not need a user attention.
        } catch (Throwable t) {
            fireExceptionCaught(channel, t);
        }

        if (readBytes > 0) {
            bb.flip();

            final ChannelBufferFactory bufferFactory =
                channel.getConfig().getBufferFactory();
            final ChannelBuffer buffer = bufferFactory.getBuffer(readBytes);
            buffer.setBytes(0, bb);
            buffer.writerIndex(readBytes);

            recvBufferPool.release(bb);

            // Update the predictor.
            predictor.previousReceiveBufferSize(readBytes);

            // Fire the event.
            fireMessageReceived(channel, buffer);
        } else {
            recvBufferPool.release(bb);
        }

        if (ret < 0 || failure) {
            k.cancel(); // Some JDK implementations run into an infinite loop without this.
            close(channel, succeededFuture(channel));
            return false;
        }

        return true;
    }

从SelectionKey中获取就绪的channel，读取channel中的byte到bytebuffer，再将bytebuffer的内容拷贝到channelbuffer，并调用fireMessageReceived触发UpstreamMessageEvent事件，由channel的pipeline中的handler处理。

在netty启动的入口，我们添加了下面这三个handler

            pipeline.addLast("decoder", new HttpRequestDecoder());
            pipeline.addLast("encoder", new HttpResponseEncoder());
            pipeline.addLast("handler", new YourHandler());

UpstreamMessageEvent在这三个handler的处理过程是，先执行upstream handler,这里是HttpRequestDecoder和YourHandler，（YourHandler正常来说是对请求的处理，也是upstream），HttpRequestDecoder根据http协议解析请求内容，生成对应的httprequest对象，交给YourHandler处理；YourHandler处理完请求之后，调用channel.write(httpresponse)向channel中写入处理的httpresponse返回给请求方。

    public static ChannelFuture write(Channel channel, Object message) {
        return write(channel, message, null);
    }

    public static ChannelFuture write(Channel channel, Object message, SocketAddress remoteAddress) {
        ChannelFuture future = future(channel);
        channel.getPipeline().sendDownstream(
                new DownstreamMessageEvent(channel, future, message, remoteAddress));
        return future;
    }

httpresponse被包装在DownstreamMessageEvent，然后在downstream handler中传递，在此例中只有HttpResponseEncoder，经过HttpResponseEncoder处理后，再向下传递，prev会为null，最后传递给NioServerSocketPipelineSink的eventSunk方法处理。

        public void sendDownstream(ChannelEvent e) {
            DefaultChannelHandlerContext prev = getActualDownstreamContext(this.prev);
            if (prev == null) {
                try {
                    getSink().eventSunk(DefaultChannelPipeline.this, e);
                } catch (Throwable t) {
                    notifyHandlerException(e, t);
                }
            } else {
                DefaultChannelPipeline.this.sendDownstream(prev, e);
            }
        }

由于channel是NioAcceptSocketChannel，传递的消息类型是DownstreamMessageEvent，所以最后在eventSunk中的处理是调用handleAcceptedSocket()方法。

    public void eventSunk(
            ChannelPipeline pipeline, ChannelEvent e) throws Exception {
        Channel channel = e.getChannel();
        if (channel instanceof NioServerSocketChannel) {
            handleServerSocket(e);
        } else if (channel instanceof NioSocketChannel) {
            handleAcceptedSocket(e);
        }
    }

    private void handleAcceptedSocket(ChannelEvent e) {
        if (e instanceof ChannelStateEvent) {
            ......
        } else if (e instanceof MessageEvent) {
            MessageEvent event = (MessageEvent) e;
            NioSocketChannel channel = (NioSocketChannel) event.getChannel();
            boolean offered = channel.writeBuffer.offer(event);
            assert offered;
            channel.worker.writeFromUserCode(channel);
        }
    }

handleAcceptedSocket()方法将DownstreamMessageEvent添加到对应channel的writeBuffer队列中，writeFromUserCode()的处理，分两种情况：

1.由于handler中有可能会添加executorhandler实现异步，对于异步的线程请求处理，会将DownstreamMessageEvent的写任务添加到当前nioworker的writeTaskQueue队列中，这个队列的处理就是在前面nioworker的run()方法中调用的processWriteTaskQueue()方法处理的，最终调用的是write0(channel)

2.没有异步的情况下，直接调用write0(channel) write数据到对应的socketchannel中

    private void write0(NioSocketChannel channel) {
        boolean open = true;
        boolean addOpWrite = false;
        boolean removeOpWrite = false;

        long writtenBytes = 0;

        final SocketSendBufferPool sendBufferPool = this.sendBufferPool;
        final SocketChannel ch = channel.socket;
        final Queue<MessageEvent> writeBuffer = channel.writeBuffer;
        final int writeSpinCount = channel.getConfig().getWriteSpinCount();
        synchronized (channel.writeLock) {
            channel.inWriteNowLoop = true;
            for (;;) {
                MessageEvent evt = channel.currentWriteEvent;
                SendBuffer buf;
                if (evt == null) {
                    if ((channel.currentWriteEvent = evt = writeBuffer.poll()) == null) {
                        removeOpWrite = true;
                        channel.writeSuspended = false;
                        break;
                    }

                    channel.currentWriteBuffer = buf = sendBufferPool.acquire(evt.getMessage());
                } else {
                    buf = channel.currentWriteBuffer;
                }

                ChannelFuture future = evt.getFuture();
                try {
                    long localWrittenBytes = 0;
                    for (int i = writeSpinCount; i > 0; i --) {
                        localWrittenBytes = buf.transferTo(ch);//这里完成channel.write(buf)
                        if (localWrittenBytes != 0) {
                            writtenBytes += localWrittenBytes;
                            break;
                        }
                        if (buf.finished()) {
                            break;
                        }
                    }

                    ......
                } catch (AsynchronousCloseException e) {
                    // Doesn't need a user attention - ignore.
                } catch (Throwable t) {
                    buf.release();
                    channel.currentWriteEvent = null;
                    channel.currentWriteBuffer = null;
                    buf = null;
                    evt = null;
                    future.setFailure(t);
                    fireExceptionCaught(channel, t);
                    if (t instanceof IOException) {
                        open = false;
                        close(channel, succeededFuture(channel));
                    }
                }
            }
            channel.inWriteNowLoop = false;

            ......
        }

        fireWriteComplete(channel, writtenBytes);
    }

完成response消息写入到channel的操作后，调用fireWriteComplete触发消息写入后的upstream DefaultWriteCompletionEvent事件。

至此，整个请求的处理流程结束。