一次Commons-HttpClient的BindException排查

线上有个老应用，在流量增长的时候，HttpClient抛出了BindException。部分的StackTrace信息如下：

 java.net.BindException: Address already in use (Bind failed) at
 java.net.PlainSocketImpl.socketBind(Native Method) ~[?:1.8.0_162] at
 java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) ~[?:1.8.0_162] at
 java.net.Socket.bind(Socket.java:644) ~[?:1.8.0_162] at
 sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source) ~[?:?] at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at
 java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at
 org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) ~[commons-httpclient-3.1.jar:?] at
 org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) ~[commons-httpclient-3.1.jar:?]`

Ephemeral Port Exhausted

先Google，很多人说是操作系统的临时端口号耗尽了。倒也说得通，线上服务没有连接池，流量一大，HttpClient每创建一个连接就会占用一个临时端口号。

但我还是有疑问。

说疑问之前先简单介绍下临时端口号（Ephemeral Port）。

一个TCP连接由四元组标识：

   {source_ip, source_port, destination_ip, destination_port}

对于HttpClient来说，每次都是作为source创建TCP连接，也就是说destination_ip和destination_port是确定的，只需要调用系统调用connect，操作系统会自动分配source_ip和source_port。

这个分配过程不仅HttpClient的使用者不关心，HttpClient的开发者也不用关心。

不过临时端口号对操作系统来说是有限的资源，有个范围限制，同时创建的连接太多，就不够用了。再创建连接，就会报错。

比如下面这条nginx log，就是因为临时端口号耗尽， Nginx无法创建到upstream的连接了：

2016/03/18 09:08:37 [crit] 1888#1888: *13 connect() to 10.2.2.77:8081 failed (99: Cannot assign requested address) while connecting to upstream, client: 10.2.2.42, server: , request: "GET / HTTP/1.1", upstream: "http://10.2.2.77:8081/", host: "10.2.2.77"

这个时候我的疑问来了。

如果原因是临时端口号耗尽，HttpClient为什么会抛出BindException呢？作为创建TCP连接的source这一方，只需要系统调用connect，没必要系统调用bind啊。

如果原因是临时端口号耗尽，像上面