由http调用引起的服务线程数高的问题处理

 

服务启动后线程数持续升高，重启后也是迅速上升，查看dump日志发现如下很多这种错误信息

"Connection evictor" #10756 daemon prio=5 os_prio=0 tid=0x00002ba4c102d000 nid=0x32ca waiting on condition [0x00002ba58ecc0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

经过分析是由于使用httpclient进行http调用导致，代码中使用了如下方式创建httpclient客户端

 HttpClientBuilder builder = HttpClientBuilder.create().evictIdleConnections(5L, TimeUnit.SECONDS);

这种使用方法，每new一个httpclient，就会多创建一个idleconnection线程，IdleConnectionEvictor是为每个httpclient创建一个超时回收器，这个回收器就是一个单独的线程，但是这个IdleConnectionEvictor不会自动回收，所以随着httpclient创建的数据增多，这个IdleConnectionEvictor也会越来越多，最终导致线程数持续上升

List<Closeable> closeablesCopy = closeables != null ? new ArrayList<Closeable>(closeables) : null;
  if (!this.connManagerShared) {
   if (closeablesCopy == null) {
    closeablesCopy = new ArrayList<Closeable>(1);
   }
   final HttpClientConnectionManager cm = connManagerCopy;
 
   if (evictExpiredConnections || evictIdleConnections) {
    final IdleConnectionEvictor connectionEvictor = new IdleConnectionEvictor(cm,
      maxIdleTime > 0 ? maxIdleTime : 10, maxIdleTimeUnit != null ? maxIdleTimeUnit : TimeUnit.SECONDS,
      maxIdleTime, maxIdleTimeUnit);
    closeablesCopy.add(new Closeable() {
 
     @Override
     public void close() throws IOException {
      connectionEvictor.shutdown();
      try {
       connectionEvictor.awaitTermination(1L, TimeUnit.SECONDS);
      } catch (final InterruptedException interrupted) {
       Thread.currentThread().interrupt();
      }
     }
 
    });
    connectionEvictor.start();
   }
   closeablesCopy.add(new Closeable() {
 
    @Override
    public void close() throws IOException {
     cm.shutdown();
    }
 
   });
  }

解决方案有两个：

1.将httpclient变为static，只创建一个idleconnection线程，由它来负责全部httpclient的回收工作

static {
    Registry<ConnectionSocketFactory> registry = RegistryBuilder.create().register("http", PlainConnectionSocketFactory.getSocketFactory()).register("https", SSLConnectionSocketFactory.getSocketFactory()).build();
    PoolingHttpClientConnectionManager gcm = new PoolingHttpClientConnectionManager(registry);
    gcm.setMaxTotal(400);
    gcm.setDefaultMaxPerRoute(200);
    RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).setSocketTimeout(30000).setConnectionRequestTimeout(5000).build();
    HttpClientBuilder httpClientBuilder = HttpClients.custom();
    httpClient = httpClientBuilder.setConnectionManager(gcm).setDefaultRequestConfig(requestConfig).setDefaultSocketConfig(SocketConfig.custom().setSoTimeout(30000).build()).evictExpiredConnections().evictIdleConnections(30L, TimeUnit.SECONDS).build();
    defaultHeaders = new Header[]{new BasicHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"), new BasicHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"), new BasicHeader("Accept-Encoding", "gzip, deflate"), new BasicHeader("Accept-Language", "zh-CN,zh;q=0.9"), new BasicHeader("Connection", "keep-alive"), new BasicHeader("Upgrade-Insecure-Requests", "1")};
} 

2.每次都创建一个新的，但是当自定义的client类被回收时，手动将httpClient涉及的东西都关闭

 if (httpClient != null) {
        try {
            httpClient.close();
        } catch (IOException e) {
            logger.error("browserHttpClient close exception", e);
        }
    }