记录一次线上排查HttpClient超时问题

线上一个对外服务使用HttpClient调用其它服务时，平均调用1000次，出现10次左右超时异常，占比1%，问了下被调用服务，他们说没限制，看了下线上并发也不大，先看下线上异常：

message:
Timeout waiting for connection from pool
stack:
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:254)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:231)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:173)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute$original$359BcZ03(InternalHttpClient.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute$original$359BcZ03$accessor$kwq4tKpV(InternalHttpClient.java)
at org.apache.http.impl.client.InternalHttpClient$auxiliary$UOMniH0V.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:93)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
....
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

然后再看看代码：

    private HttpClient httpClient;

    private HttpClient getHttpClient() {
        if (null == httpClient) {
            synchronized (this) {
                if (null == httpClient) {
                    RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).setConnectionRequestTimeout(5000).build();
                    httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).build();
                }
            }
        }

        return httpClient;
    }
   public String xxx（xxx，xxx）{
   		String url = "xxxxx";
        try {
            HttpPost httpPost = new HttpPost(url);
            HttpResponse response = getHttpClient().execute(httpPost);
            System.out.println(EntityUtils.toString(response.getEntity(), "utf-8"));
        } catch (IOException e) {
            e.printStackTrace();
        }
   } 

原来是利用DCL获取httpClient单例（同事秀骚操作），看起来并没啥毛病，来看下debug获取的httpClient实例信息：

soga，原来默认获取httpClient的连接池（maxTotal）最大是20个，defalutMaxPerRoute为2（这个参数是针对单个客户端的，比如服务端连接池是20个，defalutMaxPerRoute为2，那么小张客户通过他的客户端访问最大支持2个并发，小李也是。。加起来不能超过20个并发，否则就会超时），看到这里，原来是线上流量偶尔会较大，而最大连接池是20个，有点小，所以我们可以调大连接池的大小来减少超时的情况，改造后代码：将maxTotal设置为200，defalutMaxPerRoute设置为10

private HttpClient httpClient;

    private HttpClient getHttpClient() {
        if (null == httpClient) {
            synchronized (this) {
                if (null == httpClient) {
                    RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).setConnectionRequestTimeout(5000).build();
                    httpClient = HttpClientBuilder.create().setDefaultRequestConfig(requestConfig).setMaxConnTotal(200).setMaxConnPerRoute(10).build();
                }
            }
        }

        return httpClient;
    }