在最近的项目中因为采用了ES的技术,在最近的很长一段时间都会出现ES超时的情况,但是无论如何都没有查出原因。最开始认为是因为我们把ES从腾讯云迁移到了微软云导致的。因为这个现象就是因为迁移的ES后才出现的,同时在客户那边也没有出现同样的现象。
超时的出现让我们再开发的时候很难受,因为它时常好时常坏。我们也尝试了很多方式去排查原因,包括ES语句优化,ES配置修改等,但是都没什么用。
在最近,当项目频繁刷新的时候还会出现服务假死close_wait的现象,针对此情况我们对项目又做了一次排查。同时根据网上资料最终确定 就是因为ES超时导致链接未释放 最终造成服务假死现象。
对此我又对我们的ES连接配置进行了一次检查,发现我们的ES根本没有做链接回收的工作(开始以为它会自己回收,不需要自己动手的)。
最终我们对ES的配置进行了重新修改,终于解决了ES超时和服务假死的问题。
最终代码如下:
import cn.com.do1.do1cloud.financial.audit.model.common.Constant;
import com.google.common.collect.Lists;
import org.apache.http.HttpHost;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.ArrayList;
import java.util.Arrays;
/**
* Created by lijiawei on 2019/08/10
* ElasticSearch Client 配置
*
* @since 1.8
*/
@Configuration
public class ElasticClientConfig {
@Value("${elastic.search.host}")
private String host;
@Value("${elastic.search.port}")
private int port;
@Bean
public ArrayList<HttpHost> httpHost(){
ArrayList<HttpHost> hosts = Lists.newArrayList();
Arrays.stream(host.split(",")).forEach(host -> {
hosts.add(new HttpHost(host, port, "http"));
});
return hosts;
}
@Bean(initMethod="init",destroyMethod="close")
public ESClientSpringFactory getFactory(){
return ESClientSpringFactory.
build(httpHost(), Constant.MAX_CONNECT_NUM,
Constant.MAX_CONNECT_PER_ROUTE);
}
@Bean("elastic")
public RestHighLevelClient getRestHighLevelClient() {
return getFactory().getRhlClient();
}
}
package cn.com.do1.do1cloud.financial.audit.core.config;
/**
* @program: do1cloud-log-audit-financial
* @description: Es工厂类
* @author: lijiawei
* @create: 2020-04-16 12:46
*/
import com.google.common.collect.Lists;
import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import java.io.IOException;
import java.util.ArrayList;
public class ESClientSpringFactory {
public static int CONNECT_TIMEOUT_MILLIS = 1000;
public static int SOCKET_TIMEOUT_MILLIS = 30000;
public static int CONNECTION_REQUEST_TIMEOUT_MILLIS = 500;
public static int MAX_CONN_PER_ROUTE = 10;
public static int MAX_CONN_TOTAL = 30;
private static ArrayList<HttpHost> HTTP_HOST = Lists.newArrayList();
private RestClientBuilder builder;
private RestClient restClient;
private RestHighLevelClient restHighLevelClient;
private static ESClientSpringFactory esClientSpringFactory = new ESClientSpringFactory();
private ESClientSpringFactory(){}
/**
* 设置连接
* @param httpHost 节点集合
* @param maxConnectNum 最大连接数
* @param maxConnectPerRoute 最大重试次数
* @return
*/
public static ESClientSpringFactory build(ArrayList<HttpHost> httpHost,
Integer maxConnectNum, Integer maxConnectPerRoute){
HTTP_HOST = httpHost;
MAX_CONN_TOTAL = maxConnectNum;
MAX_CONN_PER_ROUTE = maxConnectPerRoute;
return esClientSpringFactory;
}
/**
* 设置连接
* @param httpHost 节点集合
* @param connectTimeOut 连接超时时间
* @param socketTimeOut socket 超时时间
* @param connectionRequestTime 连接请求时间
* @param maxConnectNum 最大连接数
* @param maxConnectPerRoute 最大重试次数
* @return
*/
public static ESClientSpringFactory build(ArrayList<HttpHost> httpHost,Integer connectTimeOut, Integer socketTimeOut,
Integer connectionRequestTime,Integer maxConnectNum, Integer maxConnectPerRoute){
HTTP_HOST = httpHost;
CONNECT_TIMEOUT_MILLIS = connectTimeOut;
SOCKET_TIMEOUT_MILLIS = socketTimeOut;
CONNECTION_REQUEST_TIMEOUT_MILLIS = connectionRequestTime;
MAX_CONN_TOTAL = maxConnectNum;
MAX_CONN_PER_ROUTE = maxConnectPerRoute;
return esClientSpringFactory;
}
public void init(){
builder = RestClient.builder(HTTP_HOST.toArray(new HttpHost[0]));
setConnectTimeOutConfig();
setMutiConnectConfig();
restClient = builder.build();
restHighLevelClient = new RestHighLevelClient(builder);
System.out.println("init factory");
}
/**
* 配置连接时间延时
*/
public void setConnectTimeOutConfig(){
builder.setRequestConfigCallback(requestConfigBuilder -> {
requestConfigBuilder.setConnectTimeout(CONNECT_TIMEOUT_MILLIS);
requestConfigBuilder.setSocketTimeout(SOCKET_TIMEOUT_MILLIS);
requestConfigBuilder.setConnectionRequestTimeout(CONNECTION_REQUEST_TIMEOUT_MILLIS);
return requestConfigBuilder;
});
}
/**
* 使用异步httpclient时设置并发连接数
*/
public void setMutiConnectConfig(){
builder.setHttpClientConfigCallback(httpClientBuilder -> {
httpClientBuilder.setMaxConnTotal(MAX_CONN_TOTAL);
httpClientBuilder.setMaxConnPerRoute(MAX_CONN_PER_ROUTE);
return httpClientBuilder;
});
}
public RestClient getClient(){
return restClient;
}
public RestHighLevelClient getRhlClient(){
return restHighLevelClient;
}
/**
* 连接关闭
*/
public void close() {
if (restClient != null) {
try {
restClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
System.out.println("close client");
}
}
真是一个坑了蛮久的问题。。。。希望这个解决方案是有效的