线上有个老应用,在流量增长的时候,HttpClient抛出了BindException。部分的StackTrace信息如下:
java.net.BindException: Address already in use (Bind failed) at
java.net.PlainSocketImpl.socketBind(Native Method) ~[?:1.8.0_162] at
java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) ~[?:1.8.0_162] at
java.net.Socket.bind(Socket.java:644) ~[?:1.8.0_162] at
sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source) ~[?:?] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at
java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at
org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) ~[commons-httpclient-3.1.jar:?]`
Ephemeral Port Exhausted
先Google,很多人说是操作系统的临时端口号耗尽了。倒也说得通,线上服务没有连接池,流量一大,HttpClient每创建一个连接就会占用一个临时端口号。
但我还是有疑问。
说疑问之前先简单介绍下临时端口号(Ephemeral Port)。
一个TCP连接由四元组标识:
{source_ip, source_port, destination_ip, destination_port}
对于HttpClient来说,每次都是作为source创建TCP连接,也就是说destination_ip和destination_port是确定的,只需要调用系统调用connect,操作系统会自动分配source_ip和source_port。
这个分配过程不仅HttpClient的使用者不关心,HttpClient的开发者也不用关心。
不过临时端口号对操作系统来说是有限的资源,有个范围限制,同时创建的连接太多,就不够用了。再创建连接,就会报错。
比如下面这条nginx log,就是因为临时端口号耗尽, Nginx无法创建到upstream的连接了:2016/03/18 09:08:37 [crit] 1888#1888: *13 connect() to 10.2.2.77:8081 failed (99: Cannot assign requested address) while connecting to upstream, client: 10.2.2.42, server: , request: "GET / HTTP/1.1", upstream: "http://10.2.2.77:8081/", host: "10.2.2.77"
这个时候我的疑问来了。
如果原因是临时端口号耗尽,HttpClient为什么会抛出BindException呢?作为创建TCP连接的source这一方,只需要系统调用connect,没必要系统调用bind啊。
如果原因是临时端口号耗尽,像上面