服务器打开文件太多,"通过对等方重置连接"和"太多打开文件"异常将尝试扩展Vert.x服务器...

我试图在Mac OS X上测试Java Vert.x网络服务器(基于Netty)的可扩展性.为此,我将连接限制增加到100万:

sudo sysctl -w kern.maxfiles=1000200

sudo sysctl -w kern.maxfilesperproc=1000100

sudo ulimit -n 1000000

然后我创建了localhost别名:

for i in `seq 200 250`; do sudo ifconfig lo0 alias 172.16.123.$i ; done

我现在能够连接到这些IP地址; 但是,在大约1000-2000个已建立的连接之后,我得到"通过对等连接重置"连接:

java.net.SocketException: Connection reset by peer

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)

at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:191)

at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:228)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:447)

它们之后是"太多打开文件"例外:

Caused by: java.net.SocketException: Too many open files

at sun.nio.ch.Net.socket0(Native Method)

at sun.nio.ch.Net.socket(Net.java:393)

at sun.nio.ch.Net.socket(Net.java:386)

at sun.nio.ch.SocketChannelImpl.(SocketChannelImpl.java:104)

at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)

at java.nio.channels.SocketChannel.open(SocketChannel.java:145)

at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:48)

有趣的是,当我列出所有与netstat的连接时,我发现FIN_WAIT_1状态中有很多连接:

tcp4 0 213 172.16.123.230.55618 172.16.123.230.8877 FIN_WAIT_1

tcp4 0 213 172.16.123.229.55624 172.16.123.229.8877 FIN_WAIT_1

tcp4 0 213 172.16.123.228.55617 172.16.123.228.8877 FIN_WAIT_1

tcp4 0 213 172.16.123.227.55616 172.16.123.227.8877 FIN_WAIT_1

tcp4 0 213 172.16.123.226.55612 172.16.123.226.8877 FIN_WAIT_1

其他结果:

$ netstat -a | wc -l:

14282

$ lsof | wc -l

6922

服务器代码(简化):

HttpServer server = vertx.createHttpServer();

server.websocketHandler(ws -> {

connections++;

ws.exceptionHandler(t -> t.printStackTrace());

ws.closeHandler(h -> {

connections--;

});

if (ws.path().equals("/app")) {

ws.dataHandler(data -> {

// ...

});

} else {

ws.reject();

}

}).listen(8877);

客户代码(简化):

Handler connectHandler = new Handler() {

@Override

public void handle(Void e) {

HttpClient client = vertx.createHttpClient().setHost(nextHost()).setPort(8877);

client.connectWebsocket("/app", ws -> {

connections++;

ws.exceptionHandler(t -> t.printStackTrace());

ws.closeHandler(h -> {

connections--;

});

// ...

});

if (connections < 1000000) {

vertx.runOnContext(this);

}

}

};

connectHandler.handle(null);

两者都开始于(在同一台计算机上):

vertx run Class -cp classes:... -instances 8

有没有人知道可能是什么问题？我还没有经历过这样的设置,也许我刚刚犯了一个简单的错误.它无助于增加接受积压.

更多的信息:

更新:

添加后

limit maxfiles 1000100 2000200

到/etc/launchd.conf和

kern.maxfiles=2000400

kern.maxfilesperproc=1000200

到/etc/sysctl.conf并重新启动计算机,它已经变得更好了.但是,在我获得"太多打开文件"之后,即使读取普通文件也是例外,尽管打开的文件不超过10000个:

Caused by: sun.nio.fs.UnixException: Too many open files

at sun.nio.fs.UnixNativeDispatcher.getcwd(Native Method)

at sun.nio.fs.UnixFileSystem.(UnixFileSystem.java:67)

at sun.nio.fs.BsdFileSystem.(BsdFileSystem.java:41)

at sun.nio.fs.MacOSXFileSystem.(MacOSXFileSystem.java:44)

at sun.nio.fs.MacOSXFileSystemProvider.newFileSystem(MacOSXFileSystemProvider.java:45)

at sun.nio.fs.MacOSXFileSystemProvider.newFileSystem(MacOSXFileSystemProvider.java:38)

at sun.nio.fs.UnixFileSystemProvider.(UnixFileSystemProvider.java:56)

at sun.nio.fs.BsdFileSystemProvider.(BsdFileSystemProvider.java:36)

at sun.nio.fs.MacOSXFileSystemProvider.(MacOSXFileSystemProvider.java:40)

更新2:

我也尝试了其他几个选项,但似乎有一个限制,正好是10180个连接,不能超过.除非有人有另一个想法,否则我可能只会切换到linux.

更新3:

添加JVM选项"-XX:-MaxFDLimit"之后,我现在可以打开16331个连接,直到出现以下异常:

java.net.BindException: Can't assign requested address

at sun.nio.ch.Net.connect0(Native Method)

at sun.nio.ch.Net.connect(Net.java:435)

at sun.nio.ch.Net.connect(Net.java:427)

at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:643)

at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:176)

这可能是因为默认的端口范围(16383端口):

net.inet.ip.portrange.first: 49152

net.inet.ip.portrange.last: 65535

但是,由于我使用了50个localhost alises,我原本希望能够打开16383*50 = 819150个连接.或者它是一个已知的限制,你不能使用相同的端口用于不同的别名？

更新4:

我试图用networksetup创建虚拟接口,但结果是一样的:

for i in `seq 200 250`; do sudo networksetup -createnetworkservice lo$i lo0 ; sudo networksetup -setmanual lo$i 172.16.123.$i 255.240.0.0 ; done