Netty 源码之NioEventLoopGroup

在我们写一个netty服务端程序的时候，一般都会定义两个NioEventLoopGroup，这篇博客就来分析分析NioEventLoopGroup的构成和作用。

先看一看类结构图

在新建一个NioEventLoopGroup的时候，调用的是MultithreadEventExecutorGroup父类的构造方法

方法中最重要的代码如下，新建一个ThreadPerTaskExecutor，传入一个DefaultThreadFactory

//当前没有executor，就创建一个默认的ThreadPerTaskExecutor 传入DefaultThreadFactory
if (executor == null) {
    executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
}
//EventLoop数组，数量对应线程数
children = new EventExecutor[nThreads];

ThreadPerTaskExecutor的execute方法逻辑就是通过DefaultThreadFactory将Runable构造一个Thread，并且直接start启动。总之就是看着接口好像用了什么线程池，实际上并没有

 @Override
    public void execute(Runnable command) {
        threadFactory.newThread(command).start();
    }

 再来看创建NioEventLoop，就是上面的children元素的逻辑，就是通过构造方法创建

  @Override
    protected EventLoop newChild(Executor executor, Object... args) throws Exception {
        EventLoopTaskQueueFactory queueFactory = args.length == 4 ? (EventLoopTaskQueueFactory) args[3] : null;
        return new NioEventLoop(this, executor, (SelectorProvider) args[0],
            ((SelectStrategyFactory) args[1]).newSelectStrategy(), (RejectedExecutionHandler) args[2], queueFactory);
    }

看一看NioEventLoop的类结构图  

构造方法基本没有太多的逻辑，大多数都是赋值

从构造方法可以看出，NioEventLoop中维护了一个任务队列

    NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
                 SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler,
                 EventLoopTaskQueueFactory queueFactory) {
        //parent就是存放当前NioEventLoop的NioEventLoopGroup
        //newTaskQueue 创建队列
        super(parent, executor, false, newTaskQueue(queueFactory), newTaskQueue(queueFactory),
                rejectedExecutionHandler);
        this.provider = ObjectUtil.checkNotNull(selectorProvider, "selectorProvider");
        //select策略
        this.selectStrategy = ObjectUtil.checkNotNull(strategy, "selectStrategy");
        //创建一个Selector用于监听事件java nio的方法
        final SelectorTuple selectorTuple = openSelector();
        this.selector = selectorTuple.selector;
        this.unwrappedSelector = selectorTuple.unwrappedSelector;
    }

而NioEventLoop的使用可以从NioEventLoopGroup的使用开始。

如果把NioEventLoopGroup当成一个线程池，NioEventLoop可以看成是线程池中的一个个线程，不过NioEventLoop对于线程做了许多封装

所以从NioEventLoopGroup的execute方法展开

next()方法是调用了NioEventLoopGroup中的选择器选择该任务交给哪个NioEventLoop执行

@Override
    public void execute(Runnable command) {
        next().execute(command);
    }

接下来就进入到了NioEventLoop的execute方法，这里可以看到，NioEventLoop执行任务是往任务队列中添加，而如果NioEventLoop中维护的那个线程没有启动的话，就会去启动这个线程

 public void execute(Runnable task) {
        ObjectUtil.checkNotNull(task, "task");
        //wakesUpForTask方法默认true
        execute(task, !(task instanceof LazyRunnable) && wakesUpForTask(task));
    }

 private void execute(Runnable task, boolean immediate) {
        //当前线程是不是EventLoop中的线程
        boolean inEventLoop = inEventLoop();
        //往任务队列中添加
        addTask(task);

        //如果当前线程不是EventLoop中的线程,尝试启动EventLoop
        if (!inEventLoop) {
            startThread();
            if (isShutdown()) {
                boolean reject = false;
                try {
                    if (removeTask(task)) {
                        reject = true;
                    }
                } catch (UnsupportedOperationException e) {
                    // The task queue does not support removal so the best thing we can do is to just move on and
                    // hope we will be able to pick-up the task before its completely terminated.
                    // In worst case we will log on termination.
                }
                if (reject) {
                    reject();
                }
            }
        }

        //addTaskWakesUp 表示执行addTask就会唤起当前线程 ，NioEventLoop中默认是false
        //immediate 表示是否立即执行
        //wakeup方法默认 和addTask方法类似，不过NioEventLoop这个子类重写了这个方法
        if (!addTaskWakesUp && immediate) {

            wakeup(inEventLoop);
        }
    }

   private void startThread() {
        //如果是未执行状态，就启动新线程，否则无视
        if (state == ST_NOT_STARTED) {
            if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {
                boolean success = false;
                try {
                    doStartThread();
                    success = true;
                } finally {
                    if (!success) {
                        STATE_UPDATER.compareAndSet(this, ST_STARTED, ST_NOT_STARTED);
                    }
                }
            }
        }
    }

这个线程中调用的具体方法在子类中

  //具体方法在子类中，如 NioEventLoop
SingleThreadEventExecutor.this.run();

那就来看看NioEventLoop中的具体逻辑

NioEventLoop的run方法中是一个死循环，这也就是为什么能够通过一个线程能够运行多个execute提交的任务，就是通过队列以及这个死循环实现的。

首先会判断任务队列当前有没有任务，有的话执行selectNow非阻塞方法返回事件，并执行任务，没有的话就会调用java nio的select阻塞方法等待事件

  protected void run() {
        int selectCnt = 0;
        for (;;) {
            try {
                int strategy;
                try {
                    //selectNowSupplier 方法是调用selector的selectNow方法，这个方法不阻塞，立即返回现有的事件，没有事件返回0
                    //如果当前任务队列中有任务，调用selectNowSupplier的非阻塞方法，否则，返回-1，接下来执行阻塞方法
                    //这是为了能即时调用任务队列中的任务
                    strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
                    switch (strategy) {
                    case SelectStrategy.CONTINUE:
                        continue;

                    case SelectStrategy.BUSY_WAIT:
                        // fall-through to SELECT since the busy-wait is not supported with NIO

                    case SelectStrategy.SELECT:
                        long curDeadlineNanos = nextScheduledTaskDeadlineNanos();
                        if (curDeadlineNanos == -1L) {
                            curDeadlineNanos = NONE; // nothing on the calendar
                        }
                        nextWakeupNanos.set(curDeadlineNanos);
                        try {
                            if (!hasTasks()) {
                                strategy = select(curDeadlineNanos);
                            }
                        } finally {
                            // This update is just to help block unnecessary selector wakeups
                            // so use of lazySet is ok (no race condition)
                            nextWakeupNanos.lazySet(AWAKE);
                        }
                        // fall through
                    default:
                    }
                } catch (IOException e) {
                    // If we receive an IOException here its because the Selector is messed up. Let's rebuild
                    // the selector and retry. https://github.com/netty/netty/issues/8566
                    rebuildSelector0();
                    selectCnt = 0;
                    handleLoopException(e);
                    continue;
                }

                selectCnt++;
                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                boolean ranTasks;
                //ioRatio io的占用的时间比例
                //当小于100的时候会得出一个异步任务的超时时间
                if (ioRatio == 100) {
                    try {
                        //strategy大于0的时候，表示当前有事件select到了
                        if (strategy > 0) {
                            processSelectedKeys();
                        }
                    } finally {
                        //运行 任务队列中的所有任务 ，会执行完所有的任务才返回
                        ranTasks = runAllTasks();
                    }
                } else if (strategy > 0) {
                    final long ioStartTime = System.nanoTime();
                    try {

                        processSelectedKeys();
                    } finally {
                        // Ensure we always run tasks.
                        final long ioTime = System.nanoTime() - ioStartTime;
                        ranTasks = runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }
                } else {
                    //运行 任务队列中的所有任务，一次最多运行64个任务
                    ranTasks = runAllTasks(0); // This will run the minimum number of tasks
                }

                if (ranTasks || strategy > 0) {
                    if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS && logger.isDebugEnabled()) {
                        logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.",
                                selectCnt - 1, selector);
                    }
                    selectCnt = 0;
                } else if (unexpectedSelectorWakeup(selectCnt)) { // Unexpected wakeup (unusual case)
                    selectCnt = 0;
                }
            } catch (CancelledKeyException e) {
                // Harmless exception - log anyway
                if (logger.isDebugEnabled()) {
                    logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?",
                            selector, e);
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
            // Always handle shutdown even if the loop processing threw an exception.
            try {
                if (isShuttingDown()) {
                    closeAll();
                    if (confirmShutdown()) {
                        return;
                    }
                }
            } catch (Throwable t) {
                handleLoopException(t);
            }
        }
    }

处理事件集合的方法是processSelectedKeys()，netty会自定义一个性能更高的集合存放key事件

  private void processSelectedKeys() {
        //selectedKeys这个集合初始化的时候通过反射放入selector中
        //对于nio selector返回key的那个Set集合的一个优化
        if (selectedKeys != null) {
            processSelectedKeysOptimized();
        } else {
            //处理selector中的事件
            processSelectedKeysPlain(selector.selectedKeys());
        }
    }

最后根据不同的事件分别处理

  private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
        if (!k.isValid()) {
            final EventLoop eventLoop;
            try {
                eventLoop = ch.eventLoop();
            } catch (Throwable ignored) {
                // If the channel implementation throws an exception because there is no event loop, we ignore this
                // because we are only trying to determine if ch is registered to this event loop and thus has authority
                // to close ch.
                return;
            }
            // Only close ch if ch is still registered to this EventLoop. ch could have deregistered from the event loop
            // and thus the SelectionKey could be cancelled as part of the deregistration process, but the channel is
            // still healthy and should not be closed.
            // See https://github.com/netty/netty/issues/5125
            if (eventLoop == this) {
                // close the channel if the key is not valid anymore
                unsafe.close(unsafe.voidPromise());
            }
            return;
        }

        try {
            int readyOps = k.readyOps();
            // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
            // the NIO JDK channel implementation may throw a NotYetConnectedException.

            //一下是对于OP_CONNECT导致Selector.select(..)的一个bug的解决
            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                // See https://github.com/netty/netty/issues/924
                int ops = k.interestOps();
                ops &= ~SelectionKey.OP_CONNECT;
                k.interestOps(ops);

                unsafe.finishConnect();
            }

            // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
            //如果是写出的事件
            if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                ch.unsafe().forceFlush();
            }

            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
            // to a spin loop
            //如果是读入的事件
            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                unsafe.read();
            }
        } catch (CancelledKeyException ignored) {
            unsafe.close(unsafe.voidPromise());
        }
    }

最后总结一下

NioEventLoopGroup可以说是一个线程池，其中存在多个NioEventLoop，每个NioEventLoop中只维护一个线程以及一个任务队列。

NioEventLoop的数量通过一开始传入的构造参数，或者默认根据cup核心数确定，调用NioEventLoopGroup的execute方法，就会从其中一个选择一个NioEventLoop放入任务队列，如果是NioEventLoop中线程放入的任务会直接执行。

NioEventLoop中的那个线程会在放入第一个任务的时候被启动，进入一个死循环，这个线程负责两件事，一个是从任务队列中拿任务执行，另一个就是从selector中监听事件，处理nio事件，一个NioEventLoop对应一个selector

而实际上一个Channel会通过registor注册的时候与一个NioEventLoop以及这个selector绑定，这在另外博客中详细讨论。