最近发现一个问题,在服务器上通过netstat命令发现有大量的Close_Wait长时间存在,甚至有时候数量接近1000:
查看服务器参数(etc/sysctl.conf):
net.ipv4.tcp_keepalive_time 网管已经修改成1200。
参数值还可以改小,但似乎是治标不治本,出现这种问题,肯定是某个地方的程序本身存在问题。
根据ip及端口信息,不难发现是什么地方除问题了,项目中有涉及到图片上传,于是找到图片上传的代码,结果发现代码非常简单,一行上传权限初始化代码,一行CDN官方提供的一个静态方法,之后就是处理响应结果的代码了。代码少且简单,上传调用代码没什么问题,那么问题可能出在CDN官方提供的jar包了,好在CDN有提供源码,于是查看源码,源码中使用apache 的是httpClient包,调用代码大致如下:
String response = "";
HttpPost httpPost= null;
CloseableHttpResponse ht= null;
String startTime= formatter.format(new Date());//请求时间
String endTime = "-";
String statusCode= "-";
String contentLength= "-";
String contentType= "-";try{
httpPost= newHttpPost(url);
List paramsList = new ArrayList();if (file != null) {
MultipartEntityBuilder mEntityBuilder=MultipartEntityBuilder.create();
BandwithLimiterFileBody fileBody= new BandwithLimiterFileBody(file, null, "application/octet-stream", null, BaseBlockUtil.maxRate, progressNotifier);
mEntityBuilder.addPart("file", fileBody);
mEntityBuilder.addTextBody("desc", file.getName());if (params != null && params.size() > 0) {for(String name : params.keySet()) {
mEntityBuilder.addTextBody(name, params.get(name), ContentType.create("text/plain", Charset.forName("UTF-8")));
}
}
httpPost.setEntity(mEntityBuilder.build());
}else if (params != null && params.size() > 0) {for(String name : params.keySet()) {
paramsList.add(newBasicNameValuePair(name, params.get(name)));
}
HttpEntity he= new UrlEncodedFormEntity(paramsList, "utf-8");
httpPost.setEntity(he);
}if (headMap != null && headMap.size() > 0) {for(String name : headMap.keySet()) {
httpPost.addHeader(name, headMap.get(name));
}
}if(!httpPost.containsHeader("User-Agent"))
httpPost.addHeader("User-Agent", Config.VERSION_NO);
CloseableHttpClient hc=HttpClients.createDefault();
RequestConfig requestConfig= RequestConfig.custom().setSocketTimeout(30000).setConnectTimeout(30000).build();//设置请求和传输超时时间
httpPost.setConfig(requestConfig);
ht=hc.execute(httpPost);
endTime= formatter.format(newDate());
Header[] headerArr=ht.getAllHeaders();for(Header header : headerArr) {
BufferedHeader bh=(BufferedHeader) header;if (bh.getBuffer().toString().contains("Content-Length")) {
contentLength=bh.getValue();
}else if (bh.getBuffer().toString().contains("Content-Type")) {
contentType=bh.getValue();
}
}
HttpEntity het=ht.getEntity();
InputStream is=het.getContent();
BufferedReader br= new BufferedReader(new InputStreamReader(is, "utf8"));
String readLine;while ((readLine = br.readLine()) != null) {
response= response +readLine;
}
is.close();
br.close();int status =ht.getStatusLine().getStatusCode();
statusCode=String.valueOf(status);if (status == 200) {if (!newJsonValidator().validate(response)) {
response=EncodeUtils.urlsafeDecodeString(response);
}
}return newHttpClientResult(status, response);
}catch(Exception e) {
statusCode= "500";
endTime= formatter.format(newDate());throw newHttpClientException(e);
}finally{if (httpPost != null) {
httpPost.releaseConnection();
}if (ht != null) {try{
ht.close();
}catch(IOException ignored) {
}
}
writeHttpLog(startTime, url,"-", (null != params ? params.get("token") : "-"), (null != file ? file.getName() : "-"), "-", "-", endTime, statusCode, contentType, contentLength, response);
}
查看TCP协议端口状态说明 , 如果一直保持在CLOSE_WAIT状态,那么只有一种情况,就是在对方关闭连接之后服务器程序自己没有进一步发出ack信号。因此要解决这个问题大致有以下几种方案:
a、实例化httpClient 时,使用alwaysClose 的SimpleHttpConnectionManager
通常默认情况实例化httpClient 的时候使用的是无参构造的SimpleHttpConnectionManager,因此可替换为带参构造:
new HttpClient(new SimpleHttpConnectionManager(true));
b、在method.releaseConnection() 之后 通过获取HttpConnectionManager,进行关闭(getConnectionManager方法在httpclient 4.3之后已经标记为过期,后期可能会移除该方法):
hc.getConnectionManager().shutdown();
c、在method.releaseConnection() 之后 通过获取HttpConnectionManager 调用closeIdleConnections方法进行关闭,(getConnectionManager方法在httpclient 4.3之后已经标记为过期,后期可能会移除该方法):
hc.getConnectionManager().releaseConnection(conn, validDuration, timeUnit);
d、通过设置header由服务端自动关闭.
method.setHeader("Connection", "close");
HTTP协议中有关于这个属性的定义:
HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. For example:
Connection: close
e、使用MultiThreadedHttpConnectionManager 进行复用,但同时也必须在合适的地方进行关闭处理;
f、请求之后未收到响应信息时,调用method.abort()进行处理
TCP/IP详解卷一