Elastic专栏目录
第一章 SpringBoot集成ElasticSearch(2023-01-28)
文章目录
前言
例如:业务中需要使用es,所以做一些客户端选型,熟悉一下基本的操作,所以记录这篇博客,有关概念理论性的文章还在整理过程中,后续会整理个系列
参考文章
Spring认证中国教育管理中心-Spring Data Elasticsearch教程一
SpringData集成Elasticsearch
Spring Data ElasticSearch简介
SpringBoot整合ES的三种方式(API、REST Client、Data-ES)
(一)springboot集成ES
02-Elasticsearch-集群概念入门
springboot项目使用ES
Elasticsearch【快速入门】
第一节参考
:
干货 | Elasticsearch Java 客户端演进历史和选型指南
第二节参考
:
基于spring-boot使用elasticsearch Java API 客户端
Elasticsearch8.x版本中RestHighLevelClient被弃用,新版本中全新的Java客户端Elasticsearch Java API Client中常用API练习
Elasticsearch RestHighLevelClient 已标记为被弃用 它的替代方案 Elasticsearch Java API Client 的基础教程及迁移方案
第三节参考:
ES客户端(RestHighLevelClient、SpringDataElasticsearch 框架)使用指南
一、Elasticsearch客户端现状
说实话,在es的java客户端这方面es官方就突出两个字:坑爹
RestHighLevelClient在7.15之后的版本已被标记为弃用状态,8.0新版本将不再支持,而将Elasticsearch Java API Client作为新的推荐客户端
es官方文档说明
SpringDataElasticsearch(4.0.0.RELEASE)使用的客户端为RestHighLevelClient
Spring官方对于es默认客户端变动的一些说明:
https://github.com/spring-projects/spring-data-elasticsearch/tree/5.0.x
https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/
选型关注要点:
Elasticsearch 集群的版本。
历史版本的兼容性问题。
未来升级版本、扩展性问题。
所选型的客户端是否更新及时,能适配将来的版本。
如果当前是:7.X 版本且不考虑升级,那就 High Level REST 客户端。
如果当前是:8.X 版本,那就 Elasticsearch Java API 客户端。
如果当前是:5.X、6.X 版本,推荐尽早升级集群版本。
JEST 已不更新和维护,不推荐使用。
BBoss 客户端,根据自己业务需要做选型。
Spring 框架的 Web 项目,可以使用 Spring Data Elasticsearch,但关注它的更新版本,截止:2022-06-17,支持到:7.17.3 版本
spring官方说明在springboot3版本才会支持新客户端,之前还是使用RestHighLevelClient,而springboot3基于spring6和jdk17,这就让人很难受了,自己玩玩用什么jdk都无所谓,但如果业务上就行不通了,所以以下展示如何不使用SpringDataElasticsearch接入最新的Elasticsearch Java API Client
二、SpringBoot集成ElasticSearch(Elasticsearch Java API Client)
1.Java API Client介绍
maven中央仓库中的elasticsearch-java相关版本
从7.15.0版本(beta版本)开始,截至2023-01-28最新版本为8.6.1
官方文档介绍,7.16版本
https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/7.16/introduction.html
重点是强类型
特性
强类型
提供阻塞和异步的客户端
Java API Client使用建造者模式和函数模式
新版本主要变动
兼容性
2.引入库
直接使用7的最后一个版本
<dependency>
<groupId>co.elastic.clients</groupId>
<artifactId>elasticsearch-java</artifactId>
<version>7.17.8</version>
</dependency>
3.1 配置客户端(无密码)
代码如下(示例):
@Configuration
public class JavaApiClientConfiguration {
@Value("${es.hosts.name}")
private String hosts;
/**
* 解析配置的字符串,转为HttpHost对象数组, hosts example: 127.0.0.1:9200,127.0.0.1:9300
*
* @return
*/
private HttpHost[] toHttpHost() {
if (StringUtils.isEmpty(hosts)) {
throw new RuntimeException("invalid elasticsearch configuration");
}
String[] hostArray = hosts.split(",");
HttpHost[] httpHosts = new HttpHost[hostArray.length];
HttpHost httpHost;
for (int i = 0; i < hostArray.length; i++) {
String[] strings = hostArray[i].split(":");
httpHost = new HttpHost(strings[0], Integer.parseInt(strings[1]), "http");
httpHosts[i] = httpHost;
}
return httpHosts;
}
@Bean
public ElasticsearchClient elasticsearchClient() {
HttpHost[] httpHosts = toHttpHost();
RestClient restClient = RestClient.builder(httpHosts).build();
RestClientTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
return new ElasticsearchClient(transport);
}
@Bean
public ElasticsearchAsyncClient elasticsearchAsyncClient() {
HttpHost[] httpHosts = toHttpHost();
RestClient restClient = RestClient.builder(httpHosts).build();
RestClientTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
return new ElasticsearchAsyncClient(transport);
}
}
3.2 配置客户端(有密码)
@Configuration
public class EsClientConfig {
@Value("${spring.elasticsearch.uris}")
private String hosts;
@Value("${spring.elasticsearch.username}")
private String username;
@Value("${spring.elasticsearch.password}")
private String password;
/**
* 解析配置的字符串,转为HttpHost对象数组, hosts example: 127.0.0.1:9200,127.0.0.1:9300
*
* @return
*/
private HttpHost[] toHttpHost() {
if (StringUtils.isEmpty(hosts)) {
throw new RuntimeException("invalid elasticsearch configuration");
}
String[] hostArray = hosts.split(",");
HttpHost[] httpHosts = new HttpHost[hostArray.length];
HttpHost httpHost;
for (int i = 0; i < hostArray.length; i++) {
String[] strings = hostArray[i].split(":");
httpHost = new HttpHost(strings[0], Integer.parseInt(strings[1]), "http");
httpHosts[i] = httpHost;
}
return httpHosts;
}
@Bean
public ElasticsearchClient elasticsearchClient() {
HttpHost[] httpHosts = toHttpHost();
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
RestClient restClient = RestClient.builder(httpHosts).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)).build();
ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
return new ElasticsearchClient(transport);
}
@Bean
public ElasticsearchAsyncClient elasticsearchAsyncClient() {
HttpHost[] httpHosts = toHttpHost();
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
RestClient restClient = RestClient.builder(httpHosts).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)).build();
ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
return new ElasticsearchAsyncClient(transport);
}
}
4. 启动异常
由于使用的es服务器版本为7.6.2,使用的 Java API Client版本为7.17.8(7的最后一个版本),版本不一致,所以出现了一些异常需要处理,如果一致,可直接跳过本节,本节纯属瞎折腾。
4.1兼容性请求头compatible-with=7
解决办法
如何在 Elasticsearch Java API 客户端中禁用兼容性标头?
es服务端是旧版本,而这个请求头是给8版本的es服务端使用,用以兼容7客户端的请求,所以需要覆盖这个请求头
3.1的客户端配置代码修改elasticsearchclient的构造方法
@Bean
public ElasticsearchClient elasticsearchClient() {
HttpHost[] httpHosts = toHttpHost();
RestClient restClient = RestClient
.builder(httpHosts)
.setDefaultHeaders(new Header[]{
new BasicHeader("Content-Type", "application/json; charset=UTF-8")})
.build();
RestClientTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
return new ElasticsearchClient(transport);
}
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/api-conventions.html
4.2缺失响应头X-Elastic-Product(Elasticsearch)
co.elastic.clients.transport.TransportException: [es/search] Missing [X-Elastic-Product] header
新版本的服务端es会响应这个header,但由于是旧版本,所以未响应,而客户端使用的新版本会检查这个响应头,所以会报错,但服务端已经正常执行
客户端代码中会单独检查数据库属于 Elastic 还是分叉产物。更新说明中提到,“如果响应当中没有 X-Elastic-Product HTTP 标头,或者 X-Elastic-Product HTTP 标头的值不是 Elasticsearch,就会引发错误。
之所以加入此请求头的原因是elastic官方和aws的争端,如要详细了解渊源,请查看此篇博客
换协议、改代码,Elastic 要逼开发者二选一?
官方链接文档:
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/release-notes-7.14.0.html
https://github.com/elastic/elasticsearch/issues/73424
https://github.com/elastic/elasticsearch/issues/73434
解决办法
新建配置类
@Component
public class ResponseInterceptor implements RestClientBuilder.HttpClientConfigCallback {
/**
* Allows to customize the {@link CloseableHttpAsyncClient} being created and used by the {@link RestClient}.
* Commonly used to customize the default {@link CredentialsProvider} for authentication
* or the {@link SchemeIOSessionStrategy} for communication through ssl without losing any other useful default
* value that the {@link RestClientBuilder} internally sets, like connection pooling.
*
* @param httpClientBuilder
*/
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
HttpResponseInterceptor httpResponseInterceptor = new HttpResponseInterceptor() {
/**
* Processes a response.
* On the server side, this step is performed before the response is
* sent to the client. On the client side, this step is performed
* on incoming messages before the message body is evaluated.
*
* @param response the response to postprocess
* @param context the context for the request
* @throws HttpException in case of an HTTP protocol violation
* @throws IOException in case of an I/O error
*/
@Override
public void process(HttpResponse response, HttpContext context) throws HttpException, IOException {
response.addHeader("X-Elastic-Product", "Elasticsearch");
}
};
httpClientBuilder.addInterceptorFirst(httpResponseInterceptor);
return httpClientBuilder;
}
}
3.1配置类中注入拦截器
@Autowired
private ResponseInterceptor responseInterceptor;
修改elasticsearchclient的构造方法
@Bean
public ElasticsearchClient elasticsearchClient() {
HttpHost[] httpHosts = toHttpHost();
RestClient restClient = RestClient
.builder(httpHosts)
.setDefaultHeaders(new Header[]{
new BasicHeader("Content-Type", "application/json; charset=UTF-8")})
.setHttpClientConfigCallback(responseInterceptor)
.build();
RestClientTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
return new ElasticsearchClient(transport);
}
5.基本使用
@RestController
public class JavaApiClientService {
@Autowired
private ElasticsearchClient client;
public void createIndex() {
try {
client.indices().create(builder -> {
builder.index("animal")
.settings(s -> s.numberOfShards("3").numberOfReplicas("0"));
return builder;
});
} catch (IOException e) {
e.printStackTrace();
}
}
public void deleteIndex() throws IOException {
client.indices().delete(builder -> {
builder.index("animal");
return builder;
});
}
public void addDocument() throws IOException {
client.index(builder -> {
builder.index("animal").document(new Animal()).id("0");
return builder;
});
}
public void deleteDocument() throws IOException {
client.delete(builder -> {
builder.index("animal").id("0");
return builder;
});
}
}
三、SpringBoot集成SpringDataElasticsearch
虽然旧版本的客户端会逐步废弃,但估计是个较长的过程,当前使用的es服务器主流版本仍旧为7,所以也需要熟悉SpringDataElasticsearch,RestHighLevelClient的使用1. 引入库
2.1 配置客户端
3. 基本使用
总结
例如:以上就是今天要讲的内容,本文仅仅简单介绍了pandas的使用,而pandas提供了大量能使我们快速便捷地处理数据的函数和方法。