公司一项目在高峰期报异常,且请求响应时间比预期明显长很多,排查问题发现和httpclient有关,在这里描述一下3.1版本中容易踩到的坑
问题:生产环境异常信息:
2016-12-19 14:43:27,697 ERROR - Http post occur error!url=XXXXXX
org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:497)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
……
为了排除由于网络延时,httpclient请求访问的系统响应时间太长等其他因素的影响,写了一个mock应用实现:(1)接收httpclient发送过来的http请求,(2)等待200ms,(3)返回信息“请求成功”
排查过程:
1.单线程调用httpclient方法,循环执行100次,无异常
2.多线程并发调用httpclient,10个线程,循环执行100次,报异常ConnectionPoolTimeoutException: Timeout waiting for connection,问题复现
3.检查连接时间设置5000ms,远大于200ms(mock程序等待时间),无问题
4.检查连接数设置DEFAULT_MAX_TOTAL_CONNECTIONS = 20;大于10个线程,无问题
5.观察mock应用线程状态,发现只有2个http连接,找到另外一个可疑参数DEFAULT_MAX_HOST_CONNECTIONS = 2;每个主机最大连接数,重新赋值后,解决 Timeout waiting for connection问题。
总结:org.apache.commons.httpclient.ConnectionPoolTimeoutException: Timeout waiting for connection这个问题通常是由这三个参数配置不合理导致的
连接时间、最大连接数、每个主机最大连接数,通常每个主机最大连接数DEFAULT_MAX_HOST_CONNECTIONS容易被我们忽略。
httpclient4.5版本中也有类似相关参数的配置,需要合理设置:
cm = new PoolingHttpClientConnectionManager(registry);
// 将最大连接数增加到200
cm.setMaxTotal(200);
// 将每个路由基础的连接增加到20
cm.setDefaultMaxPerRoute(20);
httpclient4.5连接数设置不合理抛出异常信息:
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
……