Netty源码—2.Reactor线程模型一

大纲

1.关于NioEventLoop的问题整理

2.理解Reactor线程模型主要分三部分

3.NioEventLoop的创建

4.NioEventLoop的启动

 

1.关于NioEventLoop的问题整理

一.默认下Netty服务端起多少线程及何时启动？

答：默认是2倍CPU核数个线程。在调用EventExcutor的execute(task)方法时，会判断当前线程是否为Netty的Reactor线程，也就是判断当前线程是否为NioEventLoop对应的线程实体。如果是，则说明Netty的Reactor线程已经启动了。如果不是，则说明是外部线程调用EventExcutor的execute()方法。于是会先调用startThread()方法判断当前线程是否已被启动，如果还没有被启动就启动当前线程作为Netty的Reactor线程。

 

二.Netty是如何解决JDK空轮询的？

答：Netty会判断如果当前阻塞的一个Select()操作并没有花那么长时间，那么就说明此时有可能触发了空轮询Bug。默认情况下如果这个现象达到512次，那么就重建一个Selector，并且把之前Selector上所有的key重新移交到新Selector上。通过以上这种处理方式来避免JDK空轮询Bug。

 

三.Netty是如何保证异步串行无锁化的？

答：异步串行无锁化有两个场景。

场景一：拿到客户端一个Channel，不需要对该Channel进行同步，直接就可以多线程并发读写。

场景二：ChannelHandler里的所有操作都是线程安全的，不需要进行同步。

 

Netty在所有外部线程去调用EventLoop或者Channel的方法时，会通过inEventLoop()方法来判断出当前线程是外部线程(非NioEventLoop的线程实体)。在这种情况下，会把所有操作都封装成一个Task放入MPSC队列，然后在NioEventLoop的执行逻辑也就是run()方法里，这些Task会被逐个执行。

 

2.理解Reactor线程模型主要分三部分

一.NioEventLoop的创建

二.NioEventLoop的启动

三.NioEventLoop的执行

 

3.NioEventLoop的创建

(1)创建入口

(2)确定NioEventLoop的个数

(3)NioEventLoopGroup的创建流程

(4)创建线程执行器ThreadPerTaskExecutor

(5)创建NioEventLoop

(6)创建线程选择器EventExecutorChooser

(7)NioEventLoopGroup的创建总结

 

(1)创建入口

EventLoopGroup bossGroup = new NioEventLoopGroup();
EventLoopGroup workerGroup = new NioEventLoopGroup();

(2)确定NioEventLoop的个数

由NioEventLoopGroup的构造方法来确定NioEventLoop的个数。如果NioEventLoopGroup没有传递构造参数，那么NioEventLoop线程的个数为CPU核数的2倍。如果NioEventLoopGroup传递了参数n，那么NioEventLoop线程的个数就是n。

 

(3)NioEventLoopGroup的创建流程

NioEventLoopGroup的构造方法会触发创建流程。

一.创建线程执行器ThreadPerTaskExecutor

每次调用ThreadPerTaskExecutor.execute()方法时都会创建一个线程。

二.创建NioEventLoop

NioEventLoop对应NioEventLoopGroup线程池里的线程，NioEventLoopGroup的构造方法会用一个for循环通过调用newChild()方法来创建NioEventLoop线程。

三.创建线程选择器EventExecutorChooser

线程选择器的作用是用于给每个新连接分配一个NioEventLoop线程，也就是从NioEventLoopGroup线程池中选择一个NioEventLoop线程来处理新连接。

//MultithreadEventLoopGroup implementations which is used for NIO Selector based Channels.
public class NioEventLoopGroup extends MultithreadEventLoopGroup {//Create a new instance using the default number of threads, //the default ThreadFactory and the SelectorProvider which is returned by SelectorProvider#provider().public NioEventLoopGroup() {this(0);}//Create a new instance using the specified number of threads, //ThreadFactory and the SelectorProvider which is returned by SelectorProvider#provider().public NioEventLoopGroup(int nThreads) {this(nThreads, (Executor) null);}public NioEventLoopGroup(int nThreads, Executor executor) {this(nThreads, executor, SelectorProvider.provider());}public NioEventLoopGroup(int nThreads, Executor executor, final SelectorProvider selectorProvider) {this(nThreads, executor, selectorProvider, DefaultSelectStrategyFactory.INSTANCE);}public NioEventLoopGroup(int nThreads, Executor executor, final SelectorProvider selectorProvider, final SelectStrategyFactory selectStrategyFactory) { super(nThreads, executor, selectorProvider, selectStrategyFactory, RejectedExecutionHandlers.reject());}...
}//Abstract base class for EventLoopGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventLoopGroup extends MultithreadEventExecutorGroup implements EventLoopGroup {private static final InternalLogger logger = InternalLoggerFactory.getInstance(MultithreadEventLoopGroup.class);private static final int DEFAULT_EVENT_LOOP_THREADS;static {DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt("io.netty.eventLoopThreads", Runtime.getRuntime().availableProcessors() * 2));if (logger.isDebugEnabled()) logger.debug("-Dio.netty.eventLoopThreads: {}", DEFAULT_EVENT_LOOP_THREADS);}protected MultithreadEventLoopGroup(int nThreads, Executor executor, Object... args) {super(nThreads == 0 ? DEFAULT_EVENT_LOOP_THREADS : nThreads, executor, args);}...
}//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {private final EventExecutor[] children;private final EventExecutorChooserFactory.EventExecutorChooser chooser;...//Create a new instance.protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);}//Create a new instance.//@param nThreads，the number of threads that will be used by this instance.//@param executor，the Executor to use, or null if the default should be used.//@param chooserFactory，the EventExecutorChooserFactory to use.//@param args，arguments which will passed to each #newChild(Executor, Object...) callprotected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));//1.创建ThreadPerTaskExecutor线程执行器if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());//2.创建NioEventLoopchildren = new EventExecutor[nThreads];for (int i = 0; i < nThreads; i ++) {...//创建每一个NioEventLoop时，都会调用newChild()方法给每一个NioEventLoop配置一些核心参数//传入线程执行器executor去创建NioEventLoopchildren[i] = newChild(executor, args);}//3.创建线程选择器chooser = chooserFactory.newChooser(children);...}...
}

创建NioEventLoopGroup的脉络如下：

new NioEventLoopGroup() //线程组，线程个数默认为2 * CPU核数new ThreadPerTaskExecutor() //创建线程执行器，作用是负责创建NioEventLoop对应的线程for(...) { newChild() } //构造NioEventLoop，创建NioEventLoop线程组chooserFactory.newChooser() //线程选择器，用于给每个新连接分配一个NioEventLoop线程

(4)创建线程执行器ThreadPerTaskExecutor

ThreadPerTaskExecutor的作用是：每次调用它的execute()方法执行Runnable任务时，都会通过threadFactory.newThread()创建出一个线程，然后把要执行的Runnable任务传递进该线程进行执行。

 

其中成员变量threadFactory是在传参给ThreadPerTaskExecutor的构造方法时，由newDefaultThreadFactory()方法构建的，也就是一个DefaultThreadFactory对象。

 

所以线程执行器ThreadPerTaskExecutor在通过threadFactory.newThread()创建线程时，其实就是调用DefaultThreadFactory的newThread()方法。

 

而DefaultThreadFactory.newThread()方法创建出来的线程实体，是Netty经过优化之后的FastThreadLocalThread对象，这个线程实体在操作ThreadLocal时，要比JDK快。

 

ThreadPerTaskExecutor线程执行器总结：

一.每次执行ThreadPerTaskExecutor的execute()方法时，都会创建出一个FastThreadLocalThread的线程实体，所以Netty的线程实体都是由ThreadPerTaskExecutor创建的。

二.FastThreadLocalThread线程实体的命名规则是：nioEventLoop-自增的线程池编号-自增的线程数编号。

//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {private final EventExecutor[] children;private final EventExecutorChooserFactory.EventExecutorChooser chooser;...    //Create a new instance.protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));//1.创建ThreadPerTaskExecutor线程执行器if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());//2.创建NioEventLoopchildren = new EventExecutor[nThreads];for (int i = 0; i < nThreads; i ++) {...//创建每一个NioEventLoop时，都会调用newChild()方法给每一个NioEventLoop配置一些核心参数//传入线程执行器executor去创建NioEventLoopchildren[i] = newChild(executor, args);}//3.创建线程选择器chooser = chooserFactory.newChooser(children);...}protected ThreadFactory newDefaultThreadFactory() {//getClass()是获取该方法所属的对象类型，也就是NioEventLoopGroup类型//因为是通过NioEventLoopGroup的构造方法层层调用到这里的return new DefaultThreadFactory(getClass());}...
}public final class ThreadPerTaskExecutor implements Executor {private final ThreadFactory threadFactory;public ThreadPerTaskExecutor(ThreadFactory threadFactory) {if (threadFactory == null) throw new NullPointerException("threadFactory");this.threadFactory = threadFactory;}@Overridepublic void execute(Runnable command) {//调用DefaultThreadFactory的newThread()方法执行Runnable任务threadFactory.newThread(command).start();}
}//A ThreadFactory implementation with a simple naming rule.
public class DefaultThreadFactory implements ThreadFactory {private static final AtomicInteger poolId = new AtomicInteger();private final AtomicInteger nextId = new AtomicInteger();private final boolean daemon;private final int priority;protected final ThreadGroup threadGroup;...public DefaultThreadFactory(Class<?> poolType) {this(poolType, false, Thread.NORM_PRIORITY);}public DefaultThreadFactory(Class<?> poolType, boolean daemon, int priority) {//toPoolName()方法会把NioEventLoopGroup的首字母变成小写this(toPoolName(poolType), daemon, priority);}public DefaultThreadFactory(String poolName, boolean daemon, int priority) {this(poolName, daemon, priority, System.getSecurityManager() == null? Thread.currentThread().getThreadGroup() : System.getSecurityManager().getThreadGroup());}public DefaultThreadFactory(String poolName, boolean daemon, int priority, ThreadGroup threadGroup) {...//prefix用来标记线程名字的前缀prefix = poolName + '-' + poolId.incrementAndGet() + '-';this.daemon = daemon;this.priority = priority;this.threadGroup = threadGroup;}@Overridepublic Thread newThread(Runnable r) {Thread t = newThread(new DefaultRunnableDecorator(r), prefix + nextId.incrementAndGet());if (t.isDaemon()) {if (!daemon) t.setDaemon(false);} else {if (daemon) t.setDaemon(true);}if (t.getPriority() != priority) t.setPriority(priority);return t;}protected Thread newThread(Runnable r, String name) {return new FastThreadLocalThread(threadGroup, r, name);}...
}

(5)创建NioEventLoop

说明一：

由MultithreadEventExecutorGroup的构造方法可知，Netty会使用for循环 + newChild()方法来创建nThreads个NioEventLoop，而且一个NioEventLoop对应一个线程实体FastThreadLocalThread。

//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {private final EventExecutor[] children;private final EventExecutorChooserFactory.EventExecutorChooser chooser;...    //Create a new instance.protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));//1.创建ThreadPerTaskExecutor线程执行器if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());//2.创建NioEventLoopchildren = new EventExecutor[nThreads];for (int i = 0; i < nThreads; i ++) {...//创建每一个NioEventLoop时，都会调用newChild()方法给每一个NioEventLoop配置一些核心参数//传入线程执行器executor去创建NioEventLoopchildren[i] = newChild(executor, args);}//3.创建线程选择器chooser = chooserFactory.newChooser(children);...}//Create a new EventExecutor which will later then accessible via the #next() method.//This method will be called for each thread that will serve this MultithreadEventExecutorGroup.protected abstract EventExecutor newChild(Executor executor, Object... args) throws Exception;...
}

说明二：

MultithreadEventExecutorGroup的newChild()抽象方法是由NioEventLoopGroup实现的，所以在执行NioEventLoopGroup的默认构造方法时也会执行其newChild()方法。

 

NioEventLoopGroup的newChild()方法需要传递一个executor参数，该参数就是执行NioEventLoopGroup构造方法开始时创建的线程执行器，之后newChild()方法会返回一个新创建的NioEventLoop对象。

//MultithreadEventLoopGroup implementations which is used for NIO Selector based Channels.
public class NioEventLoopGroup extends MultithreadEventLoopGroup {...@Overrideprotected EventLoop newChild(Executor executor, Object... args) throws Exception {//executor是执行NioEventLoopGroup构造方法开始时创建的线程执行器ThreadPerTaskExecutor//this指的是NioEventLoopGroup，表示新创建的NioEventLoop对象归属哪个NioEventLoopGroupreturn new NioEventLoop(this, executor, (SelectorProvider) args[0],((SelectStrategyFactory) args[1]).newSelectStrategy(), (RejectedExecutionHandler) args[2]);}...
}

说明三：

创建NioEventLoop对象时，NioEventLoop的构造方法会通过调用其openSelector()方法来创建一个Selector，所以一个Selector就和一个NioEventLoop绑定了，而一个Selector可以将多个连接绑定在一起来负责监听这些连接的读写事件。

 

在NioEventLoop的openSelector()方法中，Netty会通过反射对Selector底层的数据结构进行优化(Hash Set => 数组)。

//SingleThreadEventLoop implementation which register the Channel's to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {//The NIO Selector.Selector selector;private final SelectorProvider provider;private final SelectStrategy selectStrategy;...NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider,SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler) {//调用其父类SingleThreadEventLoop的构造方法super(parent, executor, false, DEFAULT_MAX_PENDING_TASKS, rejectedExecutionHandler);if (selectorProvider == null) throw new NullPointerException("selectorProvider");if (strategy == null) throw new NullPointerException("selectStrategy");this.provider = selectorProvider;this.selector = openSelector();//创建一个Selectorthis.selectStrategy = strategy;}private Selector openSelector() {final Selector selector;try {selector = provider.openSelector();...} catch(IOException e) {...}...return selector;}...
}

说明四：

NioEventLoop的构造方法还会调用其父类的父类SingleThreadEventExecutor的构造方法。SingleThreadEventExecutor的构造方法里有两个关键的操作：一是把线程执行器保存起来，因为后面创建NioEventLoop对应的线程时要用到。二是创建一个MPSC任务队列，因为Netty中所有异步执行的本质都是通过该任务队列来协调完成的。

//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {private final Queue<Runnable> tailTasks;...protected SingleThreadEventLoop(EventLoopGroup parent, Executor executor, boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedExecutionHandler) {//调用其父类SingleThreadEventExecutor的构造方法super(parent, executor, addTaskWakesUp, maxPendingTasks, rejectedExecutionHandler);//调用父类SingleThreadEventExecutor的newTaskQueue()方法tailTasks = newTaskQueue(maxPendingTasks);}...
}//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {private final boolean addTaskWakesUp;private final Executor executor;private final int maxPendingTasks;private final Queue<Runnable> taskQueue;private final RejectedExecutionHandler rejectedExecutionHandler;...//Create a new instanceprotectedSingleThreadEventExecutor(EventExecutorGroup parent, ThreadFactory threadFactory,boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {this(parent, new ThreadPerTaskExecutor(threadFactory), addTaskWakesUp, maxPendingTasks, rejectedHandler);}//Create a new instance//@param parent，the EventExecutorGroup which is the parent of this instance and belongs to it//@param executor，the Executor which will be used for executing//@param addTaskWakesUp，true if and only if invocation of #addTask(Runnable) will wake up the executor thread//@param maxPendingTasks，the maximum number of pending tasks before new tasks will be rejected.//@param rejectedHandler，the RejectedExecutionHandler to use.protected SingleThreadEventExecutor(EventExecutorGroup parent, Executor executor,boolean addTaskWakesUp, int maxPendingTasks, RejectedExecutionHandler rejectedHandler) {super(parent);this.addTaskWakesUp = addTaskWakesUp;this.maxPendingTasks = Math.max(16, maxPendingTasks);//关键操作一：把线程执行器保存起来this.executor = ObjectUtil.checkNotNull(executor, "executor");//关键操作二：创建一个MPSC任务队列this.taskQueue = newTaskQueue(this.maxPendingTasks);this.rejectedExecutionHandler = ObjectUtil.checkNotNull(rejectedHandler, "rejectedHandler");}//Create a new Queue which will holds the tasks to execute.//NioEventLoop会重写这个newTaskQueue()方法protected Queue<Runnable> newTaskQueue(int maxPendingTasks) {return new LinkedBlockingQueue<Runnable>(maxPendingTasks);}...
}//SingleThreadEventLoop implementation which register the Channel's to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {...//创建一个MPSC任务队列@Overrideprotected Queue<Runnable> newTaskQueue(int maxPendingTasks) {//This event loop never calls takeTask()return PlatformDependent.newMpscQueue(maxPendingTasks);}...
}

MPSC队列也就是多生产者单消费者队列。单消费者是指某个NioEventLoop对应的线程(执行其run()方法的那个线程)。多生产者就是这个NioEventLoop对应的线程之外的线程，通常情况下就是我们的业务线程。比如，一些线程在调用writeAndFlush()方法时可以不用考虑线程安全而随意调用，那么这些线程就是多生产者。

 

MPSC队列是通过JCTools这个工具包来实现的，Netty的高性能很大程度上要归功于这个工具包。MPSC的全称是Muti Producer Single Consumer。Muti Producer对应的是外部线程，Single Consumer对应的是Netty的NioEventLoop线程。外部线程在执行Netty的一些任务时，如果判断不是由NioEventLoop对应的线程执行的，就会直接放入一个任务队列里，然后由一个NioEventLoop对应的线程去执行。

 

创建NioEventLoop总结：

NioEventLoopGroup的newChild()方法创建NioEventLoop时做了三项事情：一.创建一个Selector用于轮询注册到该NioEventLoop上的连接，二.创建一个MPSC任务队列，三.保存线程执行器到NioEventLoop。

 

(6)创建线程选择器EventExecutorChooser

说明一：

在传统的BIO编程中，一个新连接被创建后，通常需要给这个连接绑定一个Selector，之后这个连接的整个生命周期都由这个Selector管理。

 

说明二：

创建NioEventLoop时会创建一个Selector，所以一个Selector会对应一个NioEventLoop，一个NioEventLoop上会有一个Selector。线程选择器的作用就是为一个连接在NioEventLoopGroup中选择一个NioEventLoop，从而将该连接绑定到这个NioEventLoop的Selector上。

 

说明三：

根据MultithreadEventExecutorGroup的构造方法，会使用DefaultEventExecutorChooserFactory的newChooser()方法来创建线程选择器。创建好线程选择器EventExecutorChooser之后，便可以通过其next()方法获取一个NioEventLoop。

 

Netty通过判断NioEventLoopGroup中的NioEventLoop个数是否是2的幂来创建不同的线程选择器。但不管是哪一种选择器，最终效果都是从第一个NioEventLoop开始遍历到最后一个NioEventLoop，然后再从第一个NioEventLoop开始，如此循环。

//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {private final EventExecutor[] children;private final EventExecutorChooserFactory.EventExecutorChooser chooser;...//Create a new instance.protected MultithreadEventExecutorGroup(int nThreads, Executor executor, Object... args) {this(nThreads, executor, DefaultEventExecutorChooserFactory.INSTANCE, args);}//Create a new instance.//@param nThreads，the number of threads that will be used by this instance.//@param executor，the Executor to use, or null if the default should be used.//@param chooserFactory，the EventExecutorChooserFactory to use.//@param args，arguments which will passed to each #newChild(Executor, Object...) callprotected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));//1.创建ThreadPerTaskExecutor线程执行器if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());//2.创建NioEventLoopchildren = new EventExecutor[nThreads];for (int i = 0; i < nThreads; i ++) {...//创建每一个NioEventLoop时，都会调用newChild()方法给每一个NioEventLoop配置一些核心参数//传入线程执行器executor去创建NioEventLoopchildren[i] = newChild(executor, args);}//3.创建线程选择器，chooserFactory就是传入的DefaultEventExecutorChooserFactory实例chooser = chooserFactory.newChooser(children);...}...
}//Default implementation which uses simple round-robin to choose next EventExecutor.
@UnstableApi
public final class DefaultEventExecutorChooserFactory implements EventExecutorChooserFactory {public static final DefaultEventExecutorChooserFactory INSTANCE = new DefaultEventExecutorChooserFactory();private DefaultEventExecutorChooserFactory() { }@SuppressWarnings("unchecked")@Overridepublic EventExecutorChooser newChooser(EventExecutor[] executors) {if (isPowerOfTwo(executors.length)) {//如果NioEventLoop个数是2的幂，则进行位与运算return new PowerOfTowEventExecutorChooser(executors);} else {//如果NioEventLoop个数不是2的幂，则进行普通取模运算return new GenericEventExecutorChooser(executors);}}private static boolean isPowerOfTwo(int val) {return (val & -val) == val;}private static final class PowerOfTowEventExecutorChooser implements EventExecutorChooser {private final AtomicInteger idx = new AtomicInteger();private final EventExecutor[] executors;PowerOfTowEventExecutorChooser(EventExecutor[] executors) {this.executors = executors;}@Overridepublic EventExecutor next() {return executors[idx.getAndIncrement() & executors.length - 1];}}private static final class GenericEventExecutorChooser implements EventExecutorChooser {private final AtomicInteger idx = new AtomicInteger();private final EventExecutor[] executors;GenericEventExecutorChooser(EventExecutor[] executors) {this.executors = executors;}@Overridepublic EventExecutor next() {return executors[Math.abs(idx.getAndIncrement() % executors.length)];}}
}

说明四：

创建NioEventLoopGroup的最后一个步骤就是创建线程选择器chooser，创建线程选择器的流程如下：

chooserFactory.newChooser() //创建线程选择器的入口，chooser的作用就是为新连接绑定一个NioEventLoopDefaultEventExecutorChooserFactory.isPowerOfTwo() //判断NioEventLoop个数是否为2的幂PowerOfTowEventExecutorChooser //优化index++ & (length - 1) //位与运算GenericEventExecutorChooser //普通abs(index++ % length) //取模运算

(7)NioEventLoopGroup的创建总结

默认情况下，NioEventLoopGroup会创建2倍CPU核数个NioEventLoop。一个NioEventLoop和一个Selector以及一个MPSC任务队列一一对应。

 

NioEventLoop线程的命名规则是nioEventLoopGroup-xx-yy，其中xx表示全局第xx个NioEventLoopGroup线程池，yy表示这个NioEventLoop在这个NioEventLoopGroup中是属于第yy个。

 

线程选择器chooser的作用是为一个连接选择一个NioEventLoop，可通过线程选择器的next()方法返回一个NioEventLoop。如果NioEventLoop的个数为2的幂，则next()方法会使用位与运算进行优化。

 

一个NioEventLoopGroup会有一个线程执行器executor、一个线程选择器chooser、一个数组children存放2倍CPU核数个NioEventLoop。

 

4.NioEventLoop的启动

(1)启动NioEventLoop的两大入口

(2)判断当前线程是否是NioEventLoop线程

(3)创建一个线程并启动

(4)NioEventLoop的启动总结

 

(1)启动NioEventLoop的两大入口

入口一：服务端启动，注册服务端Channel到Selector时

入口二：新连接接入，通过chooser绑定一个NioEventLoop时

 

下面先看入口一：

 

调用ServerBootstrap的bind()方法其实会调用AbstractBootstrap的doBind()方法，然后会调用AbstractBootstrap的initAndRegister()方法，接着执行config().group().register(channel)注册服务端Channel。最后会逐层深入调用到AbstractChannel.AbstractUnsafe的register()方法，来启动一个NioEventLoop将服务端Channel注册到Selector上。

//Bootstrap sub-class which allows easy bootstrap of ServerChannel
public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {......
}//AbstractBootstrap is a helper class that makes it easy to bootstrap a Channel. 
//It support method-chaining to provide an easy way to configure the AbstractBootstrap.
//When not used in a ServerBootstrap context, the #bind() methods are useful for connectionless transports such as datagram (UDP).
public abstract class AbstractBootstrap<B extends AbstractBootstrap<B, C>, C extends Channel> implements Cloneable {...//Create a new Channel and bind it.public ChannelFuture bind(int inetPort) {//首先根据端口号创建一个InetSocketAddress对象，然后调用重载方法bind()return bind(new InetSocketAddress(inetPort));}//Create a new Channel and bind it.public ChannelFuture bind(SocketAddress localAddress) {//验证服务启动需要的必要参数validate();if (localAddress == null) throw new NullPointerException("localAddress");return doBind(ObjectUtil.checkNotNull(localAddress, "localAddress"));}private ChannelFuture doBind(final SocketAddress localAddress) {final ChannelFuture regFuture = initAndRegister();//1.初始化和注册Channelfinal Channel channel = regFuture.channel();...doBind0(regFuture, channel, localAddress, promise);//2.绑定服务端端口...return promise;}final ChannelFuture initAndRegister() {Channel channel = null;...//1.创建服务端Channelchannel = channelFactory.newChannel();//2.初始化服务端Channelinit(channel);...//3.注册服务端Channel，比如通过NioEventLoopGroup的register()方法进行注册ChannelFuture regFuture = config().group().register(channel);...return regFuture;}...
}//Bootstrap sub-class which allows easy bootstrap of ServerChannel
public class ServerBootstrap extends AbstractBootstrap<ServerBootstrap, ServerChannel> {private final ServerBootstrapConfig config = new ServerBootstrapConfig(this);...@Overridepublic final ServerBootstrapConfig config() {return config;}...
}public abstract class AbstractBootstrapConfig<B extends AbstractBootstrap<B, C>, C extends Channel> {protected final B bootstrap;...protected AbstractBootstrapConfig(B bootstrap) {this.bootstrap = ObjectUtil.checkNotNull(bootstrap, "bootstrap");}//Returns the configured EventLoopGroup or null if non is configured yet.public final EventLoopGroup group() {//比如返回一个NioEventLoopGroup对象return bootstrap.group();}...
}//MultithreadEventLoopGroup implementations which is used for NIO Selector based Channels.
public class NioEventLoopGroup extends MultithreadEventLoopGroup {......
}//Abstract base class for EventLoopGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventLoopGroup extends MultithreadEventExecutorGroup implements EventLoopGroup {...@Overridepublic ChannelFuture register(Channel channel) {//先通过next()方法获取一个NioEventLoop，然后通过NioEventLoop.register()方法注册服务端Channelreturn next().register(channel);}@Overridepublic EventLoop next() {return (EventLoop) super.next();}...
}//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {private final EventExecutorChooserFactory.EventExecutorChooser chooser;...@Overridepublic EventExecutor next() {//通过线程选择器chooser选择一个NioEventLoopreturn chooser.next();}    ...
}//SingleThreadEventLoop implementation which register the Channel's to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {......
}//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {...@Overridepublic ChannelFuture register(Channel channel) {return register(new DefaultChannelPromise(channel, this));}@Overridepublic ChannelFuture register(final ChannelPromise promise) {ObjectUtil.checkNotNull(promise, "promise");//调用AbstractUnsafe的register()方法promise.channel().unsafe().register(this, promise);return promise;}...
}//A skeletal Channel implementation.
public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {private volatile EventLoop eventLoop;...//Unsafe implementation which sub-classes must extend and use.protected abstract class AbstractUnsafe implements Unsafe {...@Overridepublic final void register(EventLoop eventLoop, final ChannelPromise promise) {...//绑定事件循环器，即绑定一个NioEventLoop到该Channel上AbstractChannel.this.eventLoop = eventLoop;//注册Selector，并启动一个NioEventLoopif (eventLoop.inEventLoop()) {register0(promise);} else {...//通过启动这个NioEventLoop线程来调用register0()方法将这个服务端Channel注册到Selector上//其实执行的是SingleThreadEventExecutor的execute()方法eventLoop.execute(new Runnable() {@Overridepublic void run() {register0(promise);}});...}}}...
}

(2)判断当前线程是否是NioEventLoop线程

调用NioEventLoop的inEventLoop()方法可以判断当前线程是否是Netty的Reactor线程，也就是NioEventLoop对应的线程实体。NioEventLoop的线程实体被创建之后，会将该线程实体保存到NioEventLoop父类的成员变量thread中。

 

服务端启动、注册服务端Channel到Selector，执行到AbstractUnsafe.register()方法中的eventLoop.inEventLoop()代码时，会将main方法对应的主线程传递进来与this.thread进行比较。由于this.thread此时并未赋值，所以为空，因此inEventLoop()方法返回false，于是便会执行eventLoop.execute()代码创建一个线程并启动。

//SingleThreadEventLoop implementation which register the Channel's 
//to a Selector and so does the multi-plexing of these in the event loop.
public final class NioEventLoop extends SingleThreadEventLoop {......
}//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {......
}//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {......
}//Abstract base class for EventExecutors that want to support scheduling.
public abstract class AbstractScheduledEventExecutor extends AbstractEventExecutor {......
}//Abstract base class for {@link EventExecutor} implementations.
public abstract class AbstractEventExecutor extends AbstractExecutorService implements EventExecutor {...@Overridepublic boolean inEventLoop() {//注册服务端Channel时是通过主线程进行注册的，Thread.currentThread()对应的就是main线程//调用SingleThreadEventExecutor.inEventLoop()方法return inEventLoop(Thread.currentThread());}...
}//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {private volatile Thread thread;...@Overridepublic boolean inEventLoop(Thread thread) {return thread == this.thread;//此时线程还没创建，this.thread为null}...
}

(3)创建一个线程并启动

AbstractUnsafe.register()方法准备将服务端Channel注册到Selector上时，首先在判断条件中执行eventLoop.inEventLoop()代码发现为false，于是便执行eventLoop.execute()代码创建一个线程并启动它去执行注册任务。而执行eventLoop.execute()代码其实就是调用SingleThreadEventExecutor的execute()方法。

//Abstract base class for EventLoops that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventLoop extends SingleThreadEventExecutor implements EventLoop {...@Overridepublic ChannelFuture register(Channel channel) {return register(new DefaultChannelPromise(channel, this));}@Overridepublic ChannelFuture register(final ChannelPromise promise) {ObjectUtil.checkNotNull(promise, "promise");//调用AbstractUnsafe的register()方法，并把NioEventLoop自己当作参数传入promise.channel().unsafe().register(this, promise);return promise;}...
}//A skeletal Channel implementation.
public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {private volatile EventLoop eventLoop;...//Unsafe implementation which sub-classes must extend and use.protected abstract class AbstractUnsafe implements Unsafe {...@Overridepublic final void register(EventLoop eventLoop, final ChannelPromise promise) {...//绑定事件循环器，即绑定一个NioEventLoop到该Channel上AbstractChannel.this.eventLoop = eventLoop;//注册Selector，并启动一个NioEventLoopif (eventLoop.inEventLoop()) {register0(promise);} else {...//通过启动这个NioEventLoop线程来调用register0()方法将这个服务端Channel注册到Selector上//其实执行的是SingleThreadEventExecutor的execute()方法eventLoop.execute(new Runnable() {@Overridepublic void run() {register0(promise);}});...}}}...
}//Abstract base class for OrderedEventExecutor's that execute all its submitted tasks in a single thread.
public abstract class SingleThreadEventExecutor extends AbstractScheduledEventExecutor implements OrderedEventExecutor {private volatile Thread thread;//创建NioEventLoop时会通过构造方法传入NioEventLoopGroup的线程执行器executorprivate final Executor executor;...@Overridepublic void execute(Runnable task) {if (task == null) throw new NullPointerException("task");boolean inEventLoop = inEventLoop();//判断当前线程是否是Netty的Reactor线程if (inEventLoop) {addTask(task);} else {startThread();addTask(task);if (isShutdown() && removeTask(task)) reject();}if (!addTaskWakesUp && wakesUpForTask(task)) wakeup(inEventLoop);}private void startThread() {//判断Reactor线程有没有被启动；如果没有被启动，则通过CAS调用doStartThread()方法启动线程if (STATE_UPDATER.get(this) == ST_NOT_STARTED) {if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {doStartThread();}}}private void doStartThread() {assert thread == null;//executor.execute()方法会创建出一个FastThreadLocalThread线程来执行Runnable任务//所以在Runnable的run()方法中，Thread.currentThread()指的是这个FastThreadLocalThread线程executor.execute(new Runnable() {@Overridepublic void run() {//Thread.currentThread()指的是FastThreadLocalThread线程thread = Thread.currentThread();...SingleThreadEventExecutor.this.run();//启动线程...}});}//具体的run()方法由子类比如NioEventLoop来实现protected abstract void run();...
}

SingleThreadEventExecutor的execute()方法的说明如下：

 

一.这个方法也可能会被用户代码使用，如ctx.executor().execute(task)。所以execute()方法里又调用inEventLoop()方法进行了一次外部线程判断，确保执行task任务时不会遇到线程问题。

 

二.如果当前线程不是Netty的Reactor线程，则调用startThread()方法启动一个Reactor线程。在startThread()方法中首先会判断当前NioEventLoop对应的Reactor线程实体有没有被启动。如果没有被启动，则通过设置CAS成功后调用doStartThread()方法启动线程。

 

三.执行doStartThread()方法时，会调用NioEventLoop的内部成员变量executor的execute()方法。executor就是线程执行器ThreadPerTaskExecutor，它的作用是每次执行Runnable任务时都会创建一个线程来执行。也就是executor.execute()方法会通过DefaultThreadFactory的newThread()方法，创建出一个FastThreadLocalThread线程来执行Runnable任务。

 

四.doStartThread()方法的Runnable任务会由一个FastThreadLocalThread线程来执行。在Runnable任务的run()方法里，会保存ThreadPerTaskExecutor创建出来的FastThreadLocalThread对象到SingleThreadEventExecutor的成员变量thread中，然后调用SingleThreadEventExecutor的run()方法。

//Abstract base class for EventExecutorGroup implementations that handles their tasks with multiple threads at the same time.
public abstract class MultithreadEventExecutorGroup extends AbstractEventExecutorGroup {private final EventExecutor[] children;private final EventExecutorChooserFactory.EventExecutorChooser chooser;...    //Create a new instance.protected MultithreadEventExecutorGroup(int nThreads, Executor executor, EventExecutorChooserFactory chooserFactory, Object... args) {if (nThreads <= 0) throw new IllegalArgumentException(String.format("nThreads: %d (expected: > 0)", nThreads));//1.创建ThreadPerTaskExecutor线程执行器if (executor == null) executor = new ThreadPerTaskExecutor(newDefaultThreadFactory());//2.创建NioEventLoopchildren = new EventExecutor[nThreads];for (int i = 0; i < nThreads; i ++) {...//创建每一个NioEventLoop时，都会调用newChild()方法给每一个NioEventLoop配置一些核心参数//传入线程执行器executor去创建NioEventLoopchildren[i] = newChild(executor, args);}//3.创建线程选择器chooser = chooserFactory.newChooser(children);...}protected ThreadFactory newDefaultThreadFactory() {//getClass()是获取该方法所属的对象类型，也就是NioEventLoopGroup类型//因为是通过NioEventLoopGroup的构造方法层层调用到这里的return new DefaultThreadFactory(getClass());}...
}public final class ThreadPerTaskExecutor implements Executor {private final ThreadFactory threadFactory;public ThreadPerTaskExecutor(ThreadFactory threadFactory) {if (threadFactory == null) throw new NullPointerException("threadFactory");this.threadFactory = threadFactory;}@Overridepublic void execute(Runnable command) {//调用DefaultThreadFactory的newThread()方法执行Runnable任务threadFactory.newThread(command).start();}
}//A ThreadFactory implementation with a simple naming rule.
public class DefaultThreadFactory implements ThreadFactory {private static final AtomicInteger poolId = new AtomicInteger();private final AtomicInteger nextId = new AtomicInteger();private final boolean daemon;private final int priority;protected final ThreadGroup threadGroup;...public DefaultThreadFactory(Class<?> poolType) {this(poolType, false, Thread.NORM_PRIORITY);}public DefaultThreadFactory(Class<?> poolType, boolean daemon, int priority) {//toPoolName()方法会把NioEventLoopGroup的首字母变成小写this(toPoolName(poolType), daemon, priority);}public DefaultThreadFactory(String poolName, boolean daemon, int priority) {this(poolName, daemon, priority, System.getSecurityManager() == null ? Thread.currentThread().getThreadGroup() : System.getSecurityManager().getThreadGroup());}public DefaultThreadFactory(String poolName, boolean daemon, int priority, ThreadGroup threadGroup) {...//prefix用来标记线程名字的前缀prefix = poolName + '-' + poolId.incrementAndGet() + '-';this.daemon = daemon;this.priority = priority;this.threadGroup = threadGroup;}@Overridepublic Thread newThread(Runnable r) {Thread t = newThread(new DefaultRunnableDecorator(r), prefix + nextId.incrementAndGet());if (t.isDaemon()) {if (!daemon) t.setDaemon(false);} else {if (daemon) t.setDaemon(true);}if (t.getPriority() != priority) t.setPriority(priority);return t;}protected Thread newThread(Runnable r, String name) {return new FastThreadLocalThread(threadGroup, r, name);}...
}

NioEventLoop是如何与一个线程实体绑定的？NioEventLoop会通过线程执行器ThreadPerTaskExecutor创建一个FastThreadLocalThread，然后再将该FastThreadLocalThread线程保存到其成员变量中，从而实现与一个线程实体进行绑定。

 

(4)NioEventLoop的启动总结

一.在注册服务端Channel的过程中，主线程最终会调用AbstractUnsafe的register()方法。该方法首先会将一个NioEventLoop绑定到这个服务端Channel上，然后把实际注册Selector的逻辑封装成一个Runnable任务，接着调用NioEventLoop的execute()方法来执行这个Runnable任务。

 

二.NioEventLoop的execute()方法其实就是其父类SingleThreadEventExecutor的execute()方法，它会先判断当前调用execute()方法的线程是不是Netty的Reactor线程，如果不是就调用startThread()方法来创建一个Reactor线程。

 

三.startThread()方法会通过线程执行器ThreadPerTaskExecutor的execute()方法来创建一个线程。这个线程是一个FastThreadLocalThread线程，这个线程需要执行如下逻辑：把线程保存到NioEventLoop的成员变量thread中，然后调用NioEventLoop的run()方法来启动NioEventLoop。

 

NioEventLoop的启动流程如下：

bind() -> initAndRegister() -> config().group().register() -> eventloop.execute() //入口startThread() -> doStartThread() //创建线程ThreadPerTaskExecutor.execute() //线程执行器创建FastThreadLocalThread线程thread = Thread.currentThread() //保存FastThreadLocalThread线程到NioEventLoop的成员变量中NioEventLoop.run() //启动NioEventLoop

 

NioEventLoop的启动流程说明如下：

 

首先bind()方法会将具体绑定端口的操作封装成一个Runnable任务，然后调用NioEventLoop的execute()方法，接着Netty会判断调用execute()方法的线程是否是NIO线程，如果发现不是就会调用startThread()方法开始创建线程。

 

创建线程是通过线程执行器ThreadPerTaskExecutor来创建的。线程执行器的作用是每执行一个任务都会创建一个线程，而且创建出来的线程就是NioEventLoop底层的一个FastThreadLocalThread线程。

 

创建完FastThreadLocalThread线程后会执行一个Runnable任务，该Runnable任务首先会将这个线程保存到NioEventLoop对象。保存的目的是为了判断后续对NioEventLoop的相关执行线程是否为本身。如果不是就将封装好的一个任务放入TaskQueue中进行串行执行，实现线程安全。该Runnable任务然后会调用NioEventLoop的run()方法，从而启动NioEventLoop。NioEventLoop的run()方法是驱动Netty运转的核心方法。