生产排查org.apache.http.NoHttpResponseException: 127.0.0.1:9000 failed to respond

生产环境，请求方调用我方地址，发生异常NoHttpResponseException，错误详情：

org.apache.http.NoHttpResponseException: 127.0.0.1:9000 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.http.CommonHttpClient.sendPost(CommonHttpClient.java:96)
	at com.http.CommonHttpClient.lambda$main$0(CommonHttpClient.java:121)
	at java.util.stream.ForEachOps$ForEachOp$OfInt.accept(ForEachOps.java:205)
	at java.util.Spliterators$IntArraySpliterator.forEachRemaining(Spliterators.java:1032)
	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

看一下报错的源代码，相关注释提示服务端关闭了连接导致的：

想要解决问题，首先需要复现一下，于是本地启动服务，设置tomcat保活时间1s，

server:
  tomcat:
    keep-alive-timeout: 1000

使用common-httpclient4.5.10连接池，模拟一下请求，这里设置：

connManager.setMaxTotal(5);
connManager.setDefaultMaxPerRoute(5);

因为连接就是一组 ip:port 抽象，所以在模拟代码里获取一下本地的连接端口，在main测试中模拟20个请求，发送请求后休眠一会，推进连接过期，服务端关闭连接：

BasicHttpContext httpContext = new BasicHttpContext();
final long l = System.currentTimeMillis();
try {
    response = httpClient.execute(httpPost, httpContext);
    local = JSONObject.parseObject(JSON.toJSONString(httpContext.getAttribute("http.connection"))).getString("localPort");
    HttpEntity httpEntity = response.getEntity();
    result = EntityUtils.toString(httpEntity, "utf-8");
    System.out.println("normal" + " : " + l + " : " + result + " : " + local);
} catch (Exception e) {
    System.out.println("error " + " : " + l + " : " + local);
    e.printStackTrace();
}
-----------------------------------------------------------------------------------
public static void main(String[] args) {
        try {
            IntStream.range(1, 20).parallel().sorted().forEach(e -> {
                try {
                    sendPost("http://127.0.0.1:9000/receive", "{}");
                    TimeUnit.MILLISECONDS.sleep(995L);
                } catch (Exception exception) {
                    exception.printStackTrace();
                }
            });
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

可以看到本地和服务器共有5个连接，51573，51574，51575，51576，51577，其中有两笔异常数据报错：NoHttpResponseException，其中75，76两个端口连接后续没有被继续使用，怀疑已经被关闭了看一下抓包数据：

这两个端口处确实已经四次挥手关闭连接了，看样子应该是连接已不可用，发起请求时返回的错误。如果连接不可用，连接池会清除掉不可用连接，重新创建新的连接。再看下httpclient默认的保活策略：

问题也复现了，相关原因也基本确定了，解决办法就是保证conn的可用性：
1.关闭keep-alive，设置header Connection close
2.客户端实现一下保活策略，时间小于服务端的keep-alive-timeout，连接提前过期释放；当然应该配置合理的连接池大小，不必过大
3.客户端发生此类错误时，应该启用重试机制