netty 并发 随笔 1-netty线程模型

0. 我们的故事不能忘,太多的情节要发展 ~

从 netty-example 入手真的很不戳,学到了

(其实写这个的东西的动力，有一份部分是想增进一下markdown书写规则的了解)

---

这里放一段org.netty.example.EchoServer的代码(客户端也大同小异)，
本文会基于这个用例代码着手源码走读

// Configure the server.
// EventLoopGroup 即集成了Channel(IO)能力的EventExecutorGroup
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
// nThread=0会取一个默认值(与机器的可用处理器相关)
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
    ServerBootstrap b = new ServerBootstrap();
    // volatile EventLoopGroup group
    // volatile EventLoopGroup childGroup
    // 构建的过程基本没有什么逻辑转换在里面，可以说只是赋值而已
    b.group(bossGroup, workerGroup)
            // 这里的channel.class包装到ReflectiveChannelFactory的内部属性中
            // 当需要使用到时,借助该工厂创建出ServerSocketChannel
            // ServerBootstrap.channelFactory = channelFactory
            .channel(NioServerSocketChannel.class)
            // Map<ChannelOption<?>, Object> options做了put
            .option(ChannelOption.SO_BACKLOG, 100)
            // volatile ChannelHandler handler
            // 入参类型:ChannelHandler handler
            .handler(new LoggingHandler(LogLevel.INFO))
            // volatile ChannelHandler childHandler
            // 类型也是:ChannelHandler childHandler
            .childHandler(new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    // 这个channel绑定了pipeline
                    // pipeline又由多个handler组成
                    ChannelPipeline p = ch.pipeline();
                    p.addLast(new LoggingHandler(LogLevel.INFO));
                    p.addLast(new EchoServerHandler());
                }
            });

    // 前面包装到 ReflectChannelFactory 在这里 doBind() 时初始化channel实例
    // bind() -> io.netty.bootstrap.AbstractBootstrap.initAndRegister -> 返回 ChannelFuture(channel产出)
    // sync() -> io.netty.util.concurrent.DefaultPromise.await() -> synchronized && waiters ++
    ChannelFuture f = b.bind(EchoConstants.PORT_SERVER).sync();

    // Wait until the server socket is closed.
    f.channel().closeFuture().sync();
} finally {
    // Shut down all event loops to terminate all threads.
    bossGroup.shutdownGracefully();
    workerGroup.shutdownGracefully();
}

• 至少，通过上面这个服务端代码可以大概地知道相关类有哪些。

• 建议将其中重要的类做个UML，因为netty是基于juc的线程模型+nio上封装而成，注定会引入一段作者的封装逻辑。

• 客户端的逻辑基本大同小异

1. 搞清楚相关类是怎么设计的

先来段免责声明:

编者默认读者了解jdk.juc的Executor、Thread、ThreadFactory等概念

编者默认读者了解jdk.nio的Buffer、Channel、Selector等概念

编者默认读者了解jdk.net的Socket、Connect等概念

1.1 先放大看个局比较简单的设计

---

通过翻阅他们api中出、入参类型的变化，再加上命名上的暗示，可以确定:

• 相比 juc.future,netty.future新增了 同步执行(await()、sync()都是synchronized(this),sync()多了rethrow)、监听器集成 的能力

• 相比 netty.future,netty.Promise暴露了 try/set 成功、失败状态 的接口
• 相比 netty.future,netty.ChannelFuture将很多相同的接口出、入参类型换了 ChannelFuture,埋下了nio的种子

• 相比 以上两者,netty.ChannelPromise并没有整出啥新活儿，但他是集大成者

1.2 再缩小来看集成整体的设计

---

值得注意的是:

• netty接入juc的设计:XxxExecutorGroup同时集成了juc.ExecutorService + Iterator(赋值了Group 群组维护的能力)
• netty接入nio的设计:XxxLoopGroup(图中蓝块) extends XxxExecutorGroup(图中红块),这俩命名接近的Group的区别:引入了ChannelPromise(nio)相关的API
• netty的内部逻辑设计: Xxx(SingleThread) extends XxxGroup(MultiThread),这意味这俩者还存在着职责/角色的差异

仅仅是了解这几个比较重要的类，还是不能做到通读源码的，我们还可以再补一课:
搞清楚NioEventLoopGroup的全参构造中的入参是干嘛的，以及他们的默认值
这有助我们从入口就开始跟着源码，跟紧源码。

2. 构造NioEventLoopGroup

搞清楚默认值，以及其接口方法是做什么的就行了

下面每个变量使用三行注释来解释:

• 名称
• 默认值
• 去处(主要用途)

public NioEventLoopGroup(
        // 任务线程数
        // 默认与当前机器的核心数相关 io.netty.channel.MultithreadEventLoopGroup.DEFAULT_EVENT_LOOP_THREADS
        // 用于构造 io.netty.util.concurrent.MultithreadEventExecutorGroup.children:EventExecutor[nThreads]
        int nThreads,
        
        // 线程池的实现类
        // 默认new ThreadPerTaskExecutor(newDefaultThreadFactory())
        // 也是用于构造 io.netty.util.concurrent.MultithreadEventExecutorGroup.children[i] = newChild(executor, args)
        Executor executor,

        // 任务线程的选择策略的工厂
        // 默认DefaultEventExecutorChooserFactory(任务队列没有任务，则阻塞)
        // 任务来的时候，eventLoopGroup将选出1个eventLoop执行这个任务(先add到这个ventLoop的任务队列中)
        EventExecutorChooserFactory chooserFactory,
    
        // 用于返回1个轮询连接状态的provider
        // 默认jdk的工厂(sun.nio.ch.DefaultSelectorProvider)
        // eventLoopGroup初始化时，借助selectProvider，获取并Open selector
        final SelectorProvider selectorProvider

        // 轮询连接状态的工厂
        // 默认io.netty.channel.DefaultSelectStrategyFactory
        // 用于 eventLoop尝试循环检出事件时，选择1个分支走向:检出、退出、等待、跳过
        final SelectStrategyFactory selectStrategyFactory,
    
        // 任务拒绝时的处理器
        // 默认抛异常(io.netty.util.concurrent.RejectedExecutionHandlers.REJECT)
        // 任务新增到队列失败时，走这个拒绝的处理逻辑
        final RejectedExecutionHandler rejectedExecutionHandler,
    
        // 任务阻塞队列的工厂
        // 默认null
        // 这个先过吧，我也没有注意过这个...
        final EventLoopTaskQueueFactory taskQueueFactory) 
    {
        super(nThreads, executor, chooserFactory, selectorProvider, selectStrategyFactory,
                rejectedExecutionHandler, taskQueueFactory);
    }

NioEventLoopGroup构造完成后的 ServerBootstrap初始化、构建过程(just赋值，没有啥逻辑的)，直接步进到ServerBootstrap.bind()
我错了，有逻辑

顺着NioEventLoopGroup(nThreads)往里狠狠的步进，掠过中间的无逻辑的地方

	// io.netty.util.concurrent.MultithreadEventExecutorGroup#MultithreadEventExecutorGroup(int, java.util.concurrent.Executor, io.netty.util.concurrent.EventExecutorChooserFactory, java.lang.Object...)
    protected MultithreadEventExecutorGroup(int nThreads, Executor executor,
                                            EventExecutorChooserFactory chooserFactory, Object... args) {
        if (nThreads <= 0) {
            throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));
        }

        if (executor == null) {
        	// 这里赋值默认的线程池(eventLoop的)
        	// 也就是说group下所有的eventLoop的eventExecutor都是这货
        	// 建议关注这个executor.execute()实现逻辑:public void execute(Runnable r) { new Thread(r).start(); }
        	// 任务执行即创建1个新的线程
        	// java.util.concurrent.CompletableFuture.ThreadPerTaskExecutor
            executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());
        }

        children = new EventExecutor[nThreads];

        for (int i = 0; i < nThreads; i ++) {
            boolean success = false;
            try {
            	// EventLoopGroup 创建 EventLoop
                children[i] = newChild(executor, args);
                success = true;
            } catch (Exception e) {
                // TODO: Think about if this is a good exception type
                throw new IllegalStateException("failed to create a child event loop", e);
            } finally {
                if (!success) {
                    for (int j = 0; j < i; j ++) {
                        children[j].shutdownGracefully();
                    }

                    for (int j = 0; j < i; j ++) {
                        EventExecutor e = children[j];
                        try {
                            while (!e.isTerminated()) {
                                e.awaitTermination(Integer.MAX_VALUE, TimeUnit.SECONDS);
                            }
                        } catch (InterruptedException interrupted) {
                            // Let the caller handle the interruption.
                            Thread.currentThread().interrupt();
                            break;
                        }
                    }
                }
            }
        }

        chooser = chooserFactory.newChooser(children);

        final FutureListener<Object> terminationListener = new FutureListener<Object>() {
            @Override
            public void operationComplete(Future<Object> future) throws Exception {
                if (terminatedChildren.incrementAndGet() == children.length) {
                    terminationFuture.setSuccess(null);
                }
            }
        };

        for (EventExecutor e: children) {
            e.terminationFuture().addListener(terminationListener);
        }

        Set<EventExecutor> childrenSet = new LinkedHashSet<EventExecutor>(children.length);
        Collections.addAll(childrenSet, children);
        readonlyChildren = Collections.unmodifiableSet(childrenSet);
    }

	// io.netty.channel.nio.NioEventLoopGroup#newChild
    @Override
    protected EventLoop newChild(Executor executor, Object... args) throws Exception {
        EventLoopTaskQueueFactory queueFactory = args.length == 4 ? (EventLoopTaskQueueFactory) args[3] : null;
 		// step into ...
        return new NioEventLoop(this, executor, (SelectorProvider) args[0],
            ((SelectStrategyFactory) args[1]).newSelectStrategy(), (RejectedExecutionHandler) args[2], queueFactory);
    }

	// io.netty.channel.nio.NioEventLoop#NioEventLoop
    NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,
                 SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler,
                 EventLoopTaskQueueFactory queueFactory) {
        // 1. 构造EventLoop父类        
        super(parent, executor, false, newTaskQueue(queueFactory), newTaskQueue(queueFactory),
                rejectedExecutionHandler);
        this.provider = ObjectUtil.checkNotNull(selectorProvider, "selectorProvider");
        this.selectStrategy = ObjectUtil.checkNotNull(strategy, "selectStrategy");
        // 2. jdk.selector.open()
        final SelectorTuple selectorTuple = openSelector();
        this.selector = selectorTuple.selector;
        this.unwrappedSelector = selectorTuple.unwrappedSelector;
    }

	// io.netty.channel.SingleThreadEventLoop#SingleThreadEventLoop(io.netty.channel.EventLoopGroup, java.util.concurrent.Executor, boolean, java.util.Queue<java.lang.Runnable>, java.util.Queue<java.lang.Runnable>, io.netty.util.concurrent.RejectedExecutionHandler)
	protected SingleThreadEventLoop(EventLoopGroup parent, Executor executor,
                                    boolean addTaskWakesUp, Queue<Runnable> taskQueue, Queue<Runnable> tailTaskQueue,
                                    RejectedExecutionHandler rejectedExecutionHandler) {
        // step into ...
        super(parent, executor, addTaskWakesUp, taskQueue, rejectedExecutionHandler);
        tailTasks = ObjectUtil.checkNotNull(tailTaskQueue, "tailTaskQueue");
    }

	// io.netty.util.concurrent.SingleThreadEventExecutor#SingleThreadEventExecutor(io.netty.util.concurrent.EventExecutorGroup, java.util.concurrent.Executor, boolean, java.util.Queue<java.lang.Runnable>, io.netty.util.concurrent.RejectedExecutionHandler)
    protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor,
                                        boolean addTaskWakesUp, Queue<Runnable> taskQueue,
                                        RejectedExecutionHandler rejectedHandler) {
        // just 传递 eventLoopGroup 引用而已 
        super(parent);
        this.addTaskWakesUp = addTaskWakesUp;
        this.maxPendingTasks = DEFAULT_MAX_PENDING_EXECUTOR_TASKS;
        // step into ...
        // 只是初始化了EventLoop的EventExecutor，此时线程1个没有
        this.executor = ThreadExecutorMap.apply(executor, this);
        this.taskQueue = ObjectUtil.checkNotNull(taskQueue, "taskQueue");
        this.rejectedExecutionHandler = ObjectUtil.checkNotNull(rejectedHandler, "rejectedHandler");
    }

	// io.netty.util.internal.ThreadExecutorMap#apply(java.util.concurrent.Executor, io.netty.util.concurrent.EventExecutor)
    public static Executor apply(final Executor executor, final EventExecutor eventExecutor) {
        ObjectUtil.checkNotNull(executor, "executor");
        ObjectUtil.checkNotNull(eventExecutor, "eventExecutor");
        return new Executor() {
            @Override
            public void execute(final Runnable command) {
                executor.execute(apply(command, eventExecutor));
            }
        };
    }

	// io.netty.util.internal.ThreadExecutorMap#apply(java.lang.Runnable, io.netty.util.concurrent.EventExecutor)
    public static Runnable apply(final Runnable command, final EventExecutor eventExecutor) {
        ObjectUtil.checkNotNull(command, "command");
        ObjectUtil.checkNotNull(eventExecutor, "eventExecutor");
        return new Runnable() {
            @Override
            public void run() {
            	// 设置当前的eventExecutor(刚才在外面默认赋值的)
                setCurrentEventExecutor(eventExecutor);
                try {
                    command.run();
                } finally {
                    setCurrentEventExecutor(null);
                }
            }
        };
    }

	// io.netty.channel.nio.NioEventLoop#openSelector
	// 2. jdk.selector.open()
    private SelectorTuple openSelector() {
        final Selector unwrappedSelector;
        try {
        	// open我们前面默认的jdk.Selector
            unwrappedSelector = provider.openSelector();
        } catch (IOException e) {
            throw new ChannelException("failed to open a new selector", e);
        }

        if (DISABLE_KEY_SET_OPTIMIZATION) {
            return new SelectorTuple(unwrappedSelector);
        }

        Object maybeSelectorImplClass = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            @Override
            public Object run() {
                try {
                    return Class.forName(
                            "sun.nio.ch.SelectorImpl",
                            false,
                            PlatformDependent.getSystemClassLoader());
                } catch (Throwable cause) {
                    return cause;
                }
            }
        });

        if (!(maybeSelectorImplClass instanceof Class) ||
            // ensure the current selector implementation is what we can instrument.
            !((Class<?>) maybeSelectorImplClass).isAssignableFrom(unwrappedSelector.getClass())) {
            if (maybeSelectorImplClass instanceof Throwable) {
                Throwable t = (Throwable) maybeSelectorImplClass;
                logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, t);
            }
            return new SelectorTuple(unwrappedSelector);
        }

        final Class<?> selectorImplClass = (Class<?>) maybeSelectorImplClass;
        final SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();

        Object maybeException = AccessController.doPrivileged(new PrivilegedAction<Object>() {
            @Override
            public Object run() {
                try {
                    Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys");
                    Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys");

                    if (PlatformDependent.javaVersion() >= 9 && PlatformDependent.hasUnsafe()) {
                        // Let us try to use sun.misc.Unsafe to replace the SelectionKeySet.
                        // This allows us to also do this in Java9+ without any extra flags.
                        long selectedKeysFieldOffset = PlatformDependent.objectFieldOffset(selectedKeysField);
                        long publicSelectedKeysFieldOffset =
                                PlatformDependent.objectFieldOffset(publicSelectedKeysField);

                        if (selectedKeysFieldOffset != -1 && publicSelectedKeysFieldOffset != -1) {
                            PlatformDependent.putObject(
                                    unwrappedSelector, selectedKeysFieldOffset, selectedKeySet);
                            PlatformDependent.putObject(
                                    unwrappedSelector, publicSelectedKeysFieldOffset, selectedKeySet);
                            return null;
                        }
                        // We could not retrieve the offset, lets try reflection as last-resort.
                    }

                    Throwable cause = ReflectionUtil.trySetAccessible(selectedKeysField, true);
                    if (cause != null) {
                        return cause;
                    }
                    cause = ReflectionUtil.trySetAccessible(publicSelectedKeysField, true);
                    if (cause != null) {
                        return cause;
                    }

                    selectedKeysField.set(unwrappedSelector, selectedKeySet);
                    publicSelectedKeysField.set(unwrappedSelector, selectedKeySet);
                    return null;
                } catch (NoSuchFieldException e) {
                    return e;
                } catch (IllegalAccessException e) {
                    return e;
                }
            }
        });

        if (maybeException instanceof Exception) {
            selectedKeys = null;
            Exception e = (Exception) maybeException;
            logger.trace("failed to instrument a special java.util.Set into: {}", unwrappedSelector, e);
            return new SelectorTuple(unwrappedSelector);
        }
        selectedKeys = selectedKeySet;
        logger.trace("instrumented a special java.util.Set into: {}", unwrappedSelector);
        return new SelectorTuple(unwrappedSelector,
                                 new SelectedSelectionKeySetSelector(unwrappedSelector, selectedKeySet));
    }

整理上面我们掌握的信息:

• EventLoop 初始化时，，其EventExecutor并没有调度任何线程，仍然是EventLoopGroup的初始化线程在执行
• selector 的作用域发生在 eventLoop
• 任务到来后会，evenLoopGroup会选出1个eventLoop，并先将任务add到该eventLoop的任务队列中，任务执行会在该eventLoop的selector将其检出后,如果满足select策略将启动1条新线程去执行他

---

想了想，还是分成多篇文章吧