Netty学习——源码篇4 服务端Bootstrap(二)

接上面：Netty学习——源码篇3 服务端Bootstrap(一)

目录

1 Netty解决JDK空轮询Bug

---

        JDK空轮询bug原因是当Selector轮询结果为空时，没有进行wakeup或对新消息及时处理，导致发生了空轮询，CPU使用率达到100%。在部分Linux Kernel 2.6中，poll和epoll对于突然中断的socket连接会对返回的EventSet事件集合置为POLLHUP，也就是POLLERR，EventSet事件集合发生了变化，这就可能导致Selector会被唤醒。

        在Netty中最终的解决办法是：创建一个新的Selector，将可用事件重新注册到新的Selector中来终止空轮询，关键代码：

protected void run() {
        for (;;) {
            try {
                switch (selectStrategy.calculateStrategy(selectNowSupplier, hasTasks())) {
                    case SelectStrategy.CONTINUE:
                        continue;
                    case SelectStrategy.SELECT:
                        select(wakenUp.getAndSet(false));

                        if (wakenUp.get()) {
                            selector.wakeup();
                        }
                    default:
                        // fallthrough
                }

        前面提到select方法解决了JDK空轮询的bug，看一下select方法的源码。

private void select(boolean oldWakenUp) throws IOException {
        Selector selector = this.selector;

        try {
            int selectCnt = 0;
            long currentTimeNanos = System.nanoTime();
            long selectDeadLineNanos = currentTimeNanos + this.delayNanos(currentTimeNanos);

            while(true) {
                long timeoutMillis = (selectDeadLineNanos - currentTimeNanos + 500000L) / 1000000L;
                if (timeoutMillis <= 0L) {
                    if (selectCnt == 0) {
                        selector.selectNow();
                        selectCnt = 1;
                    }
                    break;
                }

                if (this.hasTasks() && this.wakenUp.compareAndSet(false, true)) {
                    selector.selectNow();
                    selectCnt = 1;
                    break;
                }

                int selectedKeys = selector.select(timeoutMillis);
                ++selectCnt;
                if (selectedKeys != 0 || oldWakenUp || this.wakenUp.get() || this.hasTasks() || this.hasScheduledTasks()) {
                    break;
                }

                if (Thread.interrupted()) {
                    if (logger.isDebugEnabled()) {
                        logger.debug("Selector.select() returned prematurely because Thread.currentThread().interrupt() was called. Use NioEventLoop.shutdownGracefully() to shutdown the NioEventLoop.");
                    }

                    selectCnt = 1;
                    break;
                }

                long time = System.nanoTime();
                if (time - TimeUnit.MILLISECONDS.toNanos(timeoutMillis) >= currentTimeNanos) {
                    selectCnt = 1;
                } else if (SELECTOR_AUTO_REBUILD_THRESHOLD > 0 && selectCnt >= SELECTOR_AUTO_REBUILD_THRESHOLD) {
                    logger.warn("Selector.select() returned prematurely {} times in a row; rebuilding Selector {}.", selectCnt, selector);
                    this.rebuildSelector();
                    selector = this.selector;
                    selector.selectNow();
                    selectCnt = 1;
                    break;
                }

                currentTimeNanos = time;
            }

            if (selectCnt > 3 && logger.isDebugEnabled()) {
                logger.debug("Selector.select() returned prematurely {} times in a row for Selector {}.", selectCnt - 1, selector);
            }
        } catch (CancelledKeyException var13) {
            if (logger.isDebugEnabled()) {
                logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector {} - JDK bug?", selector, var13);
            }
        }

    }

         从上面的代码可以看出，Selector每一次轮询都计数selectCnt++，开始轮询会将系统时间戳赋值给timeoutMillis，轮询完成后再将系统时间戳赋值给time，这两个时间会有一个时间差，而这个时间差就是每次轮询所消耗的时间。从上面的逻辑可以看出，如果每次轮询消耗的时间为0s，且重复次数超过512次，则调用rebuildSelector方法，即重构Selector，具体实现代码如下：

public void rebuildSelector() {
        if (!this.inEventLoop()) {
            this.execute(new Runnable() {
                public void run() {
                    NioEventLoop.this.rebuildSelector();
                }
            });
        } else {
            Selector oldSelector = this.selector;
            if (oldSelector != null) {
                Selector newSelector;
                try {
                    newSelector = this.openSelector();
                } catch (Exception var9) {
                    logger.warn("Failed to create a new Selector.", var9);
                    return;
                }

                int nChannels = 0;

                label69:
                while(true) {
                    try {
                        Iterator i$ = oldSelector.keys().iterator();

                        while(true) {
                            if (!i$.hasNext()) {
                                break label69;
                            }

                            SelectionKey key = (SelectionKey)i$.next();
                            Object a = key.attachment();

                            try {
                                if (key.isValid() && key.channel().keyFor(newSelector) == null) {
                                    int interestOps = key.interestOps();
                                    key.cancel();
                                    SelectionKey newKey = key.channel().register(newSelector, interestOps, a);
                                    if (a instanceof AbstractNioChannel) {
                                        ((AbstractNioChannel)a).selectionKey = newKey;
                                    }

                                    ++nChannels;
                                }
                            } catch (Exception var11) {
                                logger.warn("Failed to re-register a Channel to the new Selector.", var11);
                                if (a instanceof AbstractNioChannel) {
                                    AbstractNioChannel ch = (AbstractNioChannel)a;
                                    ch.unsafe().close(ch.unsafe().voidPromise());
                                } else {
                                    NioTask<SelectableChannel> task = (NioTask)a;
                                    invokeChannelUnregistered(task, key, var11);
                                }
                            }
                        }
                    } catch (ConcurrentModificationException var12) {
                    }
                }

                this.selector = newSelector;

                try {
                    oldSelector.close();
                } catch (Throwable var10) {
                    if (logger.isWarnEnabled()) {
                        logger.warn("Failed to close the old Selector.", var10);
                    }
                }

                logger.info("Migrated " + nChannels + " channel(s) to the new Selector.");
            }
        }
    }

        实际上，在rebuildSelector方法中，主要做了以下三件事。

        1、创建一个新的Selector。

        2、将原来Selector中注册的事件全部取消。

        3、将可用事件重新注册到新的Selector，并激活。

目录

2 Netty对Selector中KeySet的优化                

---

        Netty对Selector中存储SelectionKey的HashSet也做了优化，在前面的分析中，Netty对Selector有重构，创建一个新的Selector就会调用openSelector方法，代码如下：

private Selector openSelector() {
        final Selector selector;
        try {
            selector = provider.openSelector();
        } catch (IOException e) {
            throw new ChannelException("failed to open a new selector", e);
        }

        if (DISABLE_KEYSET_OPTIMIZATION) {
            return selector;
        }

        final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();

        Object maybeSelectorImplClass = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            @Override
            public Object run() {
                try {
                    return Class.forName(
                            "sun.nio.ch.SelectorImpl",
                            false,
                            PlatformDependent.getSystemClassLoader());
                } catch (ClassNotFoundException e) {
                    return e;
                } catch (SecurityException e) {
                    return e;
                }
            }
        });

        if (!(maybeSelectorImplClass instanceof Class) ||
                // ensure the current selector implementation is what we can instrument.
                !((Class<?>) maybeSelectorImplClass).isAssignableFrom(selector.getClass())) {
            if (maybeSelectorImplClass instanceof Exception) {
                Exception e = (Exception) maybeSelectorImplClass;
                logger.trace("failed to instrument a special java.util.Set into: {}", selector, e);
            }
            return selector;
        }

        final Class<?> selectorImplClass = (Class<?>) maybeSelectorImplClass;

        Object maybeException = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            @Override
            public Object run() {
                try {
                    Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys");
                    Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys");

                    selectedKeysField.setAccessible(true);
                    publicSelectedKeysField.setAccessible(true);

                    selectedKeysField.set(selector, selectedKeySet);
                    publicSelectedKeysField.set(selector, selectedKeySet);
                    return null;
                } catch (NoSuchFieldException e) {
                    return e;
                } catch (IllegalAccessException e) {
                    return e;
                } catch (RuntimeException e) {
                    // JDK 9 can throw an inaccessible object exception here; since Netty compiles
                    // against JDK 7 and this exception was only added in JDK 9, we have to weakly
                    // check the type
                    if ("java.lang.reflect.InaccessibleObjectException".equals(e.getClass().getName())) {
                        return e;
                    } else {
                        throw e;
                    }
                }
            }
        });

        if (maybeException instanceof Exception) {
            selectedKeys = null;
            Exception e = (Exception) maybeException;
            logger.trace("failed to instrument a special java.util.Set into: {}", selector, e);
        } else {
            selectedKeys = selectedKeySet;
            logger.trace("instrumented a special java.util.Set into: {}", selector);
        }

        return selector;
    }

        这个方法的主要功能就是利用反射机制，获取JDK底层的Selector的Class对象，用反射方法从Class对象中获得两个属性：selectedKeys和publicSelectedKeys，这两个属性就是用来存储已注册事件的。然后，将这两个对象重新赋值为Netty创建的SelectedSelectionKeySet。

        先看selectedKeys和publicSelectedKeys到底是什么类型，打开SelectorImpl的源码，其构造方法的代码如下：

public abstract class SelectorImpl
    extends AbstractSelector
{

    // The set of keys with data ready for an operation
    protected Set<SelectionKey> selectedKeys;

    // The set of keys registered with this Selector
    protected HashSet<SelectionKey> keys;

    // Public views of the key sets
    private Set<SelectionKey> publicKeys;             // Immutable
    private Set<SelectionKey> publicSelectedKeys;     // Removal allowed, but not addition

    protected SelectorImpl(SelectorProvider sp) {
        super(sp);
        keys = new HashSet<SelectionKey>();
        selectedKeys = new HashSet<SelectionKey>();
        if (Util.atBugLevel("1.4")) {
            publicKeys = keys;
            publicSelectedKeys = selectedKeys;
        } else {
            publicKeys = Collections.unmodifiableSet(keys);
            publicSelectedKeys = Util.ungrowableSet(selectedKeys);
        }
    }
}

        可以发现 selectedKeys和publicSelectedKeys  就是HashSet。看一下Netty创建的SelectedSelectionKesSet对象的源代码。

final class SelectedSelectionKeySet extends AbstractSet<SelectionKey> {

    private SelectionKey[] keysA;
    private int keysASize;
    private SelectionKey[] keysB;
    private int keysBSize;
    private boolean isA = true;

    SelectedSelectionKeySet() {
        keysA = new SelectionKey[1024];
        keysB = keysA.clone();
    }

    @Override
    public boolean add(SelectionKey o) {
        if (o == null) {
            return false;
        }

        if (isA) {
            int size = keysASize;
            keysA[size ++] = o;
            keysASize = size;
            if (size == keysA.length) {
                doubleCapacityA();
            }
        } else {
            int size = keysBSize;
            keysB[size ++] = o;
            keysBSize = size;
            if (size == keysB.length) {
                doubleCapacityB();
            }
        }

        return true;
    }

    private void doubleCapacityA() {
        SelectionKey[] newKeysA = new SelectionKey[keysA.length << 1];
        System.arraycopy(keysA, 0, newKeysA, 0, keysASize);
        keysA = newKeysA;
    }

    private void doubleCapacityB() {
        SelectionKey[] newKeysB = new SelectionKey[keysB.length << 1];
        System.arraycopy(keysB, 0, newKeysB, 0, keysBSize);
        keysB = newKeysB;
    }

    SelectionKey[] flip() {
        if (isA) {
            isA = false;
            keysA[keysASize] = null;
            keysBSize = 0;
            return keysA;
        } else {
            isA = true;
            keysB[keysBSize] = null;
            keysASize = 0;
            return keysB;
        }
    }

    @Override
    public int size() {
        if (isA) {
            return keysASize;
        } else {
            return keysBSize;
        }
    }

    @Override
    public boolean remove(Object o) {
        return false;
    }

    @Override
    public boolean contains(Object o) {
        return false;
    }

    @Override
    public Iterator<SelectionKey> iterator() {
        throw new UnsupportedOperationException();
    }
}

        SelectedSelectionKeySet同样继承了AbstractSet，因此赋值给selectedKeys和publicSelectedKeys不存在类型强制转换的问题。SelectedSelectionKeySet禁用了remove、contains和iterator方法，只保留的add方法。这样设计的主要目的是在轮询事件时的操作，不需要每次轮询都移除Key。

3 Handler添加过程

        服务端Handler的添加过程和客户端有些区别，跟EventLoopGroup一样，服务端的Handler也有两个：一个是通过handler方法设置的Handler，另一个是通过childHandler方法设置的childHandler。通过前面的bossGroup和workerGroup的分析，可以猜测：Handler与accept过程有关，即Handler负责处理客户端新连接的接入请求；而childHandler就是负责和客户端连接的I/O交互。在前面的介绍已经了解到ServerBootstrap重新了init方法，在这个方法中添加了Handler，代码如下：

void init(Channel channel) throws Exception {
        Map<ChannelOption<?>, Object> options = this.options0();
        synchronized(options) {
            channel.config().setOptions(options);
        }

        Map<AttributeKey<?>, Object> attrs = this.attrs0();
        synchronized(attrs) {
            Iterator i$ = attrs.entrySet().iterator();

            while(true) {
                if (!i$.hasNext()) {
                    break;
                }

                Map.Entry<AttributeKey<?>, Object> e = (Map.Entry)i$.next();
                AttributeKey<Object> key = (AttributeKey)e.getKey();
                channel.attr(key).set(e.getValue());
            }
        }

        ChannelPipeline p = channel.pipeline();
        final EventLoopGroup currentChildGroup = this.childGroup;
        final ChannelHandler currentChildHandler = this.childHandler;
        final Map.Entry[] currentChildOptions;
        synchronized(this.childOptions) {
            currentChildOptions = (Map.Entry[])this.childOptions.entrySet().toArray(newOptionArray(this.childOptions.size()));
        }

        final Map.Entry[] currentChildAttrs;
        synchronized(this.childAttrs) {
            currentChildAttrs = (Map.Entry[])this.childAttrs.entrySet().toArray(newAttrArray(this.childAttrs.size()));
        }

        p.addLast(new ChannelHandler[]{new ChannelInitializer<Channel>() {
            public void initChannel(Channel ch) throws Exception {
                final ChannelPipeline pipeline = ch.pipeline();
                ChannelHandler handler = ServerBootstrap.this.config.handler();
                if (handler != null) {
                    pipeline.addLast(new ChannelHandler[]{handler});
                }

                ch.eventLoop().execute(new Runnable() {
                    public void run() {
                        pipeline.addLast(new ChannelHandler[]{new ServerBootstrapAcceptor(currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs)});
                    }
                });
            }
        }});
    }

        在上面代码initChannel()方法中，首先通过handler方法获取一个Handler，如果获取的Handler不为空，则添加到Pipeline中，然后添加一个ServerBootsrapAccept的实例。这里的Handler方法返回的Handler属性，而这个属性是在服务端的启动代码中设置的。

bootstrap.group(boosGroup,workersGroup)

        这个时候，Pipeline中的Handler情况如下图所示。

        根据对原来客户端代码的分析，将Channel绑定到EventLoop（这里是指NioServerSocketChannel绑定到bossGroup）后，会在Pipeline中触发fireChannelRegister事件，接着会触发对ChannelInitializer的initChannel方法的调用。因此在绑定完成后，此时的Pipiline的内容如下图：

        在分析bossGroup和workerGroup时，已经知道ServerBootstrapAccept的channelRead方法会为新建的Channel设置Handler并注册到一个EventLoop中。 

        后续的步骤基本已经清楚了，在客户端连接Channel注册后，就会触发ChannelInitializer的initChannel方法的调用。最后总结一下服务端Handler与childHandler的区别和联系。

        1、在服务端NioServerSocketChannel对象的Pipiline中添加Handler对象和ServerBootstrapAccept对象。

        2、当有新的客户端连接请求时，会调用ServerBootsrapAccept的channelRead方法创建并连接对应的NioSocketChannel对象，并将childHandler添加到NioSocketChannel对应的Pipeline中，而且将此Channel绑定到workerGroup中的某个EventLoop。

        3、Handler对象只在accept阻塞阶段起作用，它主要处理客户端发送过来的连接请求。

        4、childHandler在客户端连接建立以后起作用，它负责客户端连接的I/O交互。