聊聊jetty的线程模型

Acceptor

监听

每个acceptor绑定一个ServerSocketChannel通道

1. ServerConnector由其管理容器启动start
2. 打开连接：打开ServerSocketChannel通道，并配置accept通道（_acceptChannel）为阻塞模式，绑定地址端口
3. 构建org.eclipse.jetty.server.AbstractConnector.Acceptor并提交至线程池（org.eclipse.jetty.server.Server#_threadPool，默认大小200=org.eclipse.jetty.util.thread.QueuedThreadPool#QueuedThreadPool()）执行启动，如果通道已打开，阻塞在accept接受客户端请求，等待数据，线程池中异步执行
4. 如果存在多个Acceptors则配置accept通道（_acceptChannel=ServerSocketChannel）为非阻塞模式
5. 选择器管理器提交org.eclipse.jetty.io.ManagedSelector.Acceptor（绑定SocketChannel、attachment为null）至队列并唤醒选择器通知选择器存在更新选择器更新

• 选择器生产者自旋阻塞在java.nio.channels.Selector#select()等待唤醒并处理更新
• 处理选择器更新，回调选择器更新实例的org.eclipse.jetty.io.ManagedSelector.Acceptor#update方法，将SelectableChannel（accept通道（_acceptChannel=ServerSocketChannel））注册至选择器Selector

数据到达服务端

客户端数据到达服务端，阻塞唤醒

1. 服务端接收到数据唤醒accept阻塞
2. 配置SocketChannel通道为非阻塞模式
3. 选择器管理器提交org.eclipse.jetty.io.ManagedSelector.Accept（绑定SocketChannel、attachment为null）至队列并唤醒选择器通知选择器存在更新选择器更新
4. 选择器生产者自旋阻塞在java.nio.channels.Selector#select()等待唤醒并处理更新
5. 处理选择器更新，回调选择器更新实例的org.eclipse.jetty.io.ManagedSelector.Accept#update方法，将SelectableChannel（与客户端建立的数据通道SocketChannel）注册至选择器Selector

jetty线程模型

总结

Acceptor可以理解为NIO的：java.nio.channels.ServerSocketChannel
Selector可以理解为NIO的：java.nio.channels.Selector

线程池类型

Jetty默认线程池为：org.eclipse.jetty.util.thread.QueuedThreadPool，线程池资源监控：org.eclipse.jetty.server.LowResourceMonitor.LowResourceCheck（例如：org.eclipse.jetty.server.LowResourceMonitor.MainThreadPoolLowResourceCheck#isLowOnResources，org.eclipse.jetty.server.LowResourceMonitor.ConnectorsThreadPoolLowResourceCheck#isLowOnResources）

线程池资源监控处理

线程池处于高水位线时间超出最大值：org.eclipse.jetty.server.LowResourceMonitor#_maxLowResourcesTime，默认0，则将所有链接设置为拒绝接收数据：org.eclipse.jetty.server.LowResourceMonitor#setLowResources

线程池低于高水位线时恢复接收数据：org.eclipse.jetty.server.LowResourceMonitor#clearLowResources

监控

public class MyJettyServerCustomizer implements JettyServerCustomizer {

  @Override
  public void customize(Server server) {
    LowResourceMonitor lowResourceMonitor = new LowResourceMonitor(server);
    LowResourceCheck lowResourceCheck = lowResourceMonitor.new MainThreadPoolLowResourceCheck();
    lowResourceMonitor.addLowResourceCheck(lowResourceCheck);
    server.addBean(lowResourceMonitor);
  }
}

配置监控

    @Bean
    public ServletWebServerFactory servletContainer() {
        JettyServletWebServerFactory jetty = new JettyServletWebServerFactory();
        QueuedThreadPool queuedThreadPool = new QueuedThreadPool(4,4);
        jetty.setThreadPool(queuedThreadPool);
        jetty.setPort(Integer.parseInt(serverPort.trim()));
        MyJettyServerCustomizer myJettyServerCustomizer = new MyJettyServerCustomizer();
        jetty.addServerCustomizers(myJettyServerCustomizer);
        return jetty;
    }

启用低资源时拒绝服务

public class MyJettyServerCustomizer implements JettyServerCustomizer {
  @Override
  public void customize(Server server) {
    LowResourceMonitor lowResourceMonitor = new LowResourceMonitor(server);
    // 启用低资源拒绝服务
    lowResourceMonitor.setAcceptingInLowResources(false);
    // 低资源持续时间达到阈值拒绝服务
    lowResourceMonitor.setMaxLowResourcesTime(1);
    LowResourceCheck lowResourceCheck = lowResourceMonitor.new MainThreadPoolLowResourceCheck();
    lowResourceMonitor.addLowResourceCheck(lowResourceCheck);
    server.addBean(lowResourceMonitor);
  }
}

启用后可能在资源不足时拒绝服务，出现报错

客户端错误信息

org.apache.http.NoHttpResponseException: localhost:8000 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:939)
	at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:650)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1301)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1290)
	at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651)
	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570)
	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501)
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268)
	at java.base/java.lang.Thread.run(Thread.java:833)

服务端错误信息

2023-03-05 10:55:49,918 WARN [qtp1493142905-23] o.e.j.u.t.QueuedThreadPool:execute:538 {} QueuedThreadPool[qtp1493142905]@58ff8d79{STARTED,4<=4<=4,i=0,r=1,q=0}[ReservedThreadExecutor@6f79390f{s=0/1,p=1}] rejected org.eclipse.jetty.io.ManagedSelector$Accept@640d906c
2023-03-05 10:55:49,919 WARN [qtp1493142905-23] o.e.j.u.t.QueuedThreadPool:execute:538 {} QueuedThreadPool[qtp1493142905]@58ff8d79{STARTED,4<=4<=4,i=1,r=1,q=0}[ReservedThreadExecutor@6f79390f{s=0/1,p=1}] rejected CEP:SocketChannelEndPoint@19cfb8d{/1.0.0.0:21363<->/[0:0:0:0:0:0:7f00:1]:8000,OPEN,fill=FI,flush=-,to=1/30000}{io=1/0,kio=1,kro=1}->HttpConnection@64db9962[p=HttpParser{s=START,0 of -1},g=HttpGenerator@6c1819f3{s=START}]=>HttpChannelOverHttp@3b35730{s=HttpChannelState@b3529da{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
2023-03-05 10:55:49,919 WARN [qtp1493142905-23] o.e.j.u.t.s.EatWhatYouKill:execute:378 {} 
java.util.concurrent.RejectedExecutionException: CEP:SocketChannelEndPoint@19cfb8d{/1.0.0.0:21363<->/[0:0:0:0:0:0:7f00:1]:8000,OPEN,fill=FI,flush=-,to=1/30000}{io=1/0,kio=1,kro=1}->HttpConnection@64db9962[p=HttpParser{s=START,0 of -1},g=HttpGenerator@6c1819f3{s=START}]=>HttpChannelOverHttp@3b35730{s=HttpChannelState@b3529da{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=0,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
	at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:539) ~[jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.execute(EatWhatYouKill.java:373) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:308) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) [jetty-util-9.4.22.v20191022.jar:9.4.22.v20191022]
	at java.lang.Thread.run(Thread.java:833) [?:?]

线程池

1. 默认线程池：org.eclipse.jetty.util.thread.QueuedThreadPool，min=8,max=200
2. 队列：org.eclipse.jetty.util.BlockingArrayQueue#BlockingArrayQueue(int, int)，capacity=grawBy=Math.max(_minThreads, 8) * 1024
3. 拒绝策略：直接抛异常throw new RejectedExecutionException(job.toString());
4. 如果开启了监控LowResourceMonitor，监控在检测到系统繁忙时会暂停数据处理，不再繁忙时恢复数据处理
5. 注意如果线程池类型为QueuedThreadPool配置会被重写，需要结合jetty配置，重写逻辑设计配置可以参考源码：org.springframework.boot.autoconfigure.web.embedded.JettyWebServerFactoryCustomizer#customize

优点

架构设计的优点很明显，并发扩展能力很强大，可以支持多个Connector的垂直向扩展，由于使用了多路复用IO，每个Selector可以监听多个Channel，进一步提升了框架的并发能力

缺点

线程模型较为单调，对于多个Connector其实可以理解为多个租户，如果对多个租户做到线程池的资源隔离的话个人感觉效果会更好。不过其实也还好，算是鸡蛋里挑骨头了，毕竟多Connector的场景以及多租户的场景可以通过水平方向的扩展+DNS路由来弥补