ES启动报错：Caused by: java.net.SocketException: No buffer space available (maximum connections reached?)

问题描述：

线上服务的es突然挂掉导致网站无法正常访问，看了下是es服务停止了，于是重启后出现了如下报错，对于报错信息的重点关注：Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
看报错信息是连接数太多，耗尽了某种资源。
查看当前所有的连接，执行命令：netstat -ano ，发现大量的TCP连接。尤其是应用的80端口的连接数达到了65535，是应用中调用连接过多导致资源耗尽无法启动es，先关停java应用，然后重启es解决

报错信息：

[2023-02-28T14:52:32,279][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) ~[?:?]
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) ~[?:?]
	at org.elasticsearch.transport.SharedGroupFactory.getGenericGroup(SharedGroupFactory.java:89) ~[?:?]
	at org.elasticsearch.transport.SharedGroupFactory.getTransportGroup(SharedGroupFactory.java:70) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:125) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:84) ~[?:?]
	at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:46) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:233) ~[elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:59) ~[elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.node.Node.start(Node.java:778) ~[elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317) ~[elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402) [elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.9.2.jar:7.9.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.9.2.jar:7.9.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.9.2.jar:7.9.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.9.2.jar:7.9.2]
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	... 24 more
Caused by: java.io.IOException: Unable to establish loopback connection
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:101) ~[?:?]
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:68) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_60]
	at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:170) ~[?:?]
	at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50) ~[?:?]
	at java.nio.channels.Pipe.open(Pipe.java:155) ~[?:1.8.0_60]
	at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127) ~[?:?]
	at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	... 24 more
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
	at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
	at sun.nio.ch.Net.connect(Net.java:454) ~[?:?]
	at sun.nio.ch.Net.connect(Net.java:446) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:?]
	at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[?:1.8.0_60]
	at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:130) ~[?:?]
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:83) ~[?:?]
	at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:68) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_60]
	at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:170) ~[?:?]
	at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50) ~[?:?]
	at java.nio.channels.Pipe.open(Pipe.java:155) ~[?:1.8.0_60]
	at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127) ~[?:?]
	at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) ~[?:?]
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) ~[?:?]
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) ~[?:?]
	... 24 more