远端服务重启导致httpclient连接池卡死的问题

从锁说起吧，内置锁synchroinzed和jucl（java.util.concurrent.locks）lock还是有很大区别的，一个很重要的区别就是使用jstack导出线程dump，使用synchronized命令可以很容易看到锁被哪个线程持有，但是jucl lock缺不行。 上文 中使用httpclient写了一个连接池，使用了一个socket设置，setSoLinger(60)，然后在一次测试中重启了远程的服务，结果导致了系统的卡死，查看本机的连接，发现很多连接都处于close_wait状态。考虑到远程服务重启，所以已有的连接肯定需要全部断开，断开就需要四次挥手，那么close_wait是什么状态呢？

看上图，很明显是远程server要重启，主动关闭连接，发送了FIN，本机收到应该会立刻回应ACK，然后，本机应立刻发送FIN，但是却没有发，停留在CLOSE_WAIT状态。联想到使用的SocketConfig setSoLinger(60)，可能是这个设置导致这个问题。在hang住的时候，使用jstack查看线程dump，如下：

"Thread-2158" #2171 prio=5 os_prio=0 tid=0x00000000193fe800 nid=0xa3c80 waiting on condition [0x000000003e70e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000081eab2a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.apache.http.pool.AbstractConnPool.getTotalStats(AbstractConnPool.java:509)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.formatStats(PoolingHttpClientConnectionManager.java:227)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:265)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at http.connectionPool.TestConnectionPool.lambda$main$1(TestConnectionPool.java:85)
	at http.connectionPool.TestConnectionPool$$Lambda$3/670035812.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:748)

很多线程在等待进入锁，那么查询一下谁持有0x0000000081eab2a8不就行了，哈哈。可是查了整个线程dump文件，却找不到到底是哪个线程持有了这把锁，因为用的是ReentrantLock，和synchronized不一样。 jstack命令可以加参数，-l打印lock详情 -e打印线程详情，试试 jstack -l pid 多了这么一句

Locked ownable synchronizers:

再次搜索，可以使用正则表达式排除大部分记录

^.*(?<!wait for  )<0x0000000081eb5388>.*$
查询不是wait for的行

结果：

"Thread-2623" #2636 prio=5 os_prio=0 tid=0x000000001aa7f000 nid=0xba4c8 runnable [0x000000005daed000]
   java.lang.Thread.State: RUNNABLE
	at java.net.DualStackPlainSocketImpl.close0(Native Method)
	at java.net.DualStackPlainSocketImpl.socketClose0(DualStackPlainSocketImpl.java:167)
	at java.net.AbstractPlainSocketImpl.socketPreClose(AbstractPlainSocketImpl.java:693)
	at java.net.AbstractPlainSocketImpl.close(AbstractPlainSocketImpl.java:530)
	- locked <0x00000000839b0280> (a java.lang.Object)
	at java.net.PlainSocketImpl.close(PlainSocketImpl.java:237)
	at java.net.SocksSocketImpl.close(SocksSocketImpl.java:1075)
	at java.net.Socket.close(Socket.java:1495)
	- locked <0x00000000839b00e0> (a java.lang.Object)
	- locked <0x00000000839b00c0> (a java.net.Socket)
	at sun.security.ssl.BaseSSLSocketImpl.close(BaseSSLSocketImpl.java:624)
	- locked <0x00000000839aff68> (a sun.security.ssl.SSLSocketImpl)
	at sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1585)
	at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1723)
	at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1612)
	at org.apache.http.impl.BHttpConnectionBase.close(BHttpConnectionBase.java:334)
	at org.apache.http.impl.conn.LoggingManagedHttpClientConnection.close(LoggingManagedHttpClientConnection.java:81)
	at org.apache.http.impl.conn.CPoolEntry.closeConnection(CPoolEntry.java:70)
	at org.apache.http.impl.conn.CPoolEntry.close(CPoolEntry.java:96)
	at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:318)
	at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
	at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
	- locked <0x00000000d814f998> (a org.apache.http.pool.AbstractConnPool$2)
	at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at http.connectionPool.TestConnectionPool.lambda$main$1(TestConnectionPool.java:90)
	at http.connectionPool.TestConnectionPool$$Lambda$3/670035812.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x0000000081eb5388> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

找到了一个线程！！，可以看到正在close，并且持有了连接池的锁，这样的话，由于设置了setSoLinger，它会一直等待60s，于是请求连接的线程就堆积起来了。