- 异常日志
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:326)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
问题分析
网上说No buffer space available (maximum connections reached?)是因为机器端口号用尽导致tcp无法申请新的端口去建立连接。首先想到去查看机器端口号使用情况。
大量的端口处于TIME_WAIT状态,初步估计可能是客户端每次去请求服务端都创建了一次tcp连接,使用完就关闭连接,但是tcp协议中如果客户端主动关闭,为了确保发送的报文不影响下一次连接,主动关闭的一方需要停留在TIME_WAIT状态2*MSL(Max Segment Lifetime)。当并发高的时候,就会产生大量TIME_WAIT,而处于TIME_WAIT状态的端口是不能马上去建立新的连接的,这就导致No buffer space available (maximum connections reached?)这个异常。问题修复
排查代码发现有一处rest请求每次都会创建一个httpclient。相当于每次都要重新建立tcp连接,完成3次握手和4次挥手。
修复后的代码是使用http连接池,服用重复的http连接。这样不必每次都去建立新的tcp连接。不仅可以避免端口被占满,也可以提升io的性能(tcp连接建立和关闭是需要开销的)。httpclient提供PoolingHttpClientConnectionManager类支持http连接池的管理。