【ES一】SpringBoot2.x+ES8.11.1（windows环境）整合

zcccc_

已于 2024-05-06 10:09:12 修改

阅读量1.1k

点赞数 21

文章标签： elasticsearch 搜索引擎 spring boot

于 2024-05-06 10:03:09 首次发布

本文链接：https://blog.csdn.net/zcccc_/article/details/137971573

版权

一、两种整合方式

目前有两种最常用的整合方式，一种是ElasticSearch官方提供的 Java High Level Rest Client，一种是Spring提供的 spring-boot-starter-data-elasticsearch 方式：

1.spring-boot-starter-data-elasticsearch 方式

由 Spring 提供，是 Spring 在 ES 官方接口基础之上的二次封装，使用简单，易于上手；
缺点是更新太慢，SpringBoot 2.2.x 才提供对 es7.x 的支持，版本关联性很大，不易维护；

2. Java High Level Rest Client 方式（推荐）（我使用的）

由ES官方提供，代码语法和DSL语法相似（即Json格式的ES操作语法）；
用法灵活，可以自由使用；
SpringBoot 和 ES 版本的关联性较小；

二、开始搭建

项目基础

1.SpringBoot3.x JDK17 Es8.11.1 ik分词 8.11.1（启动报错暂时放弃……）

2.SpringBoot2.x JDK17 Es8.11.1 ik分词 8.11.1（es_work)

windows环境

地址：C:\java\elasticsearch8111

JDK ：ES_JAVA_HOME (JDK17.0.3)

前提了解

es与jdk版本对应：支持一览表 | Elastic

es与Spring Data Elasticsearch 之间版本对应：Spring Data Elasticsearch - Reference Documentation

步骤

1.安装ES

链接：win10+elasticsearch8.12 安装教程_win10 elasticsearch8.12-CSDN博客

下载地址：Elasticsearch 8.11.1 | Elastic

本地地址：C:\java\elasticsearch8111\elasticsearch-8.11.1

注意：环境变量JDK配置正常是JAVA_HOME,es配置里是ES_JAVA_HOME，启动时会报错，所以在环境变量里添加上ES_JAVA_HOM，如果你有其他方法也可以。

默认地址：localhost:9200

账号密码：账号：elastic 密码：自动生成(Sx6+sb3Ags45pNyrfws=)，也可自定义。（忘记密码怎么办？进入es安装目录的bin文件中执行一下命令会返回最新的密码命令 : elasticsearch-reset-password -u elastic）

2.安装ik分词

链接：本地elasticsearch中文分词器 ik分词器安装及使用_es安装ik分词器-CSDN博客

本地地址：C:\java\elasticsearch8111\elasticsearch-8.11.1\plugins\ik

IK分词器有两种分词模式：ik_max_word（最细粒度拆分）和ik_smart（最粗粒度）模式。

自定义分词bic文件：C:\java\elasticsearch8111\elasticsearch-8.11.1\plugins\ik\config

3.安装客户端Kibana

需要和es版本保持一致

默认地址：http://localhost:5601/ 初次访问一般后边还有code拼接

账号密码：es的账号密码

本地地址：C:\java\elasticsearch8111\kibana-8.11.1-windows-x86_64\kibana-8.11.1

学习链接：ElasticSearch——Kibana Windows下的安装和Dev Tools的使用_elastic dev tools-CSDN博客

使用教程：ES8.8生产实践——数据查询与数据可视化(Kibana)_es可视化-CSDN博客

4.JAVA代码

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.es</groupId>
    <artifactId>es-work</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>es-work</name>

    <packaging>jar</packaging>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.4.4</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>5.9.1</junit.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.20</version>
        </dependency>
        <!-- @ConfigurationProperties(prefix = "elasticsearch") 注解依赖      -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <!-- 若不存在Spring Data ES的某个版本支持你下的ES版本,则使用  -->
        <!-- ES 官方提供的在JAVA环境使用的依赖 -->
        <dependency>
            <groupId>co.elastic.clients</groupId>
            <artifactId>elasticsearch-java</artifactId>
            <version>8.11.1</version>
        </dependency>
       <!-- 和第一个依赖是一起的,为了解决springboot项目的兼容性问题  -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>
        <!-- 原来只有前两依赖，但是启动报错找不到依赖，要注意版本号是否兼容  -->
        <dependency>
            <groupId>jakarta.json</groupId>
            <artifactId>jakarta.json-api</artifactId>
            <version>2.0.1</version>
        </dependency>
        <!-- es的客户端，连接，版本号要与ES一致  -->
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>8.11.1</version>
        </dependency>
        <!-- lombok -->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

</project>

application.yml

server:
  port: 8092

spring:
# elasticsearch配置
elasticsearch:
  # 自定义属性---设置是否开启ES,false表不开窍ES
  open: true
  # es集群名称,如果下载es设置了集群名称,则使用配置的集群名称
  clusterName: es
  hosts: 127.0.0.1:9200
  # es 请求方式
  scheme: http
  # es 连接超时时间
  connectTimeOut: 1000
  # es socket 连接超时时间
  socketTimeOut: 30000
  # es 请求超时时间
  connectionRequestTimeOut: 500
  # es 最大连接数
  maxConnectNum: 100
  # es 每个路由的最大连接数
  maxConnectNumPerRoute: 100
  userName: elastic
  password: Sx6+sb3Ags45pNyrfws=

config

package com.es.eswork.config;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.ElasticsearchTransport;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import lombok.Data;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
@Data
@Configuration
@ConfigurationProperties(prefix = "elasticsearch")
public class ElasticSearchConfig {

    // 是否开启ES
    private Boolean open;

    // es 集群host ip 地址
    private String hosts;

    // es用户名
    private String userName;

    // es密码
    private String password;

    // es 请求方式
    private String scheme;

    // es集群名称
    private String clusterName;

    // es 连接超时时间
    private int connectTimeOut;

    // es socket 连接超时时间
    private int socketTimeOut;

    // es 请求超时时间
    private int connectionRequestTimeOut;

    // es 最大连接数
    private int maxConnectNum;

    // es 每个路由的最大连接数
    private int maxConnectNumPerRoute;

    // es api key
    private String apiKey;


    public RestClientBuilder creatBaseConfBuilder(String scheme){


        // 1. 单节点ES Host获取
        String host = hosts.split(":")[0];
        String port = hosts.split(":")[1];


        // The value of the schemes attribute used bynoSafeRestClient() is http
        // but The value of the schemes attribute used by safeRestClient() is https
        HttpHost httpHost = new HttpHost(host, Integer.parseInt(port),scheme);

        // 1.1 设置用户名和密码(账号密码连接，xpack.security.enabled: true)
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, password));


        // 2. 创建构建器对象
        //RestClientBuilder: ES客户端库的构建器接口,用于构建RestClient实例;允许你配置与Elasticsearch集群的连接,设置请求超时,设置身份验证,配置代理等
        RestClientBuilder builder = RestClient.builder(httpHost);



        // 连接延时配置
        builder.setRequestConfigCallback(requestConfigBuilder -> {
            requestConfigBuilder.setConnectTimeout(connectTimeOut);
            requestConfigBuilder.setSocketTimeout(socketTimeOut);
            requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeOut);
            return requestConfigBuilder;
        });

        // 3. HttpClient 连接数配置
        builder.setHttpClientConfigCallback(httpClientBuilder -> {
            httpClientBuilder.setMaxConnTotal(maxConnectNum);
            httpClientBuilder.setMaxConnPerRoute(maxConnectNumPerRoute);
            httpClientBuilder.disableAuthCaching();  // 1.1.1 设置账号密码
            httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            return httpClientBuilder;
        });



        return builder;
    }

    /**
     * @function: 创建使用http连接来直接连接ES服务器的客户端
     * 如果@Bean没有指定bean的名称,那么这个bean的名称就是方法名
     */
    @Bean
    public ElasticsearchClient elasticsearchClient(){
        RestClientBuilder builder = creatBaseConfBuilder((scheme == "http")?"http":"http");
        //Create the transport with a Jackson mapper
        ElasticsearchTransport transport = new RestClientTransport(builder.build(), new JacksonJsonpMapper());
        //And create the API client
        ElasticsearchClient esClient = new ElasticsearchClient(transport);
        return esClient;
    }
}

test测试

package com.es.eswork;

import com.es.eswork.entity.User;
import com.es.eswork.utils.ElasticsearchHandle;
import lombok.extern.slf4j.Slf4j;
import net.minidev.json.JSONArray;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

import java.util.ArrayList;
import java.util.List;

@SpringBootTest
@Slf4j
public class EsTestByUtil {


    @Autowired
    ElasticsearchHandle client;

   
    @Test
    public void test() {
        //测试logstash
        log.info("log");
        String indexName = "test_index";
        try {
            boolean b = client.hasIndex(indexName);
            System.out.println(b);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Test
    public void search() {
        List<User> search = client.search("test_index", "hello", 5, "name");

        System.out.println(JSONArray.toJSONString(search));
    }

    @Test
    public void searchCategoryIds() {
        List<String> ca = new ArrayList<>();
        ca.add("hello");
        List<User> search = client.searchCategoryIds("test_index", "hello", ca, 5);
        System.out.println(JSONArray.toJSONString(search));
    }

}

utils

package com.es.eswork.utils;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.FieldValue;
import co.elastic.clients.elasticsearch._types.mapping.Property;
import co.elastic.clients.elasticsearch._types.query_dsl.Query;
import co.elastic.clients.elasticsearch._types.query_dsl.TermsQuery;
import co.elastic.clients.elasticsearch._types.query_dsl.TermsQueryField;
import co.elastic.clients.elasticsearch.core.DeleteResponse;
import co.elastic.clients.elasticsearch.core.GetResponse;
import co.elastic.clients.elasticsearch.core.IndexResponse;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.elasticsearch.core.search.Hit;
import co.elastic.clients.elasticsearch.indices.CreateIndexResponse;
import co.elastic.clients.elasticsearch.indices.DeleteIndexResponse;
import co.elastic.clients.transport.endpoints.BooleanResponse;
import com.es.eswork.entity.User;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

@Slf4j
@Component
public class ElasticsearchHandle {

    @Autowired
    private ElasticsearchClient client;

    /**
     * 判断索引是否存在
     *
     * @param indexName
     * @return
     * @throws IOException
     */
    public boolean hasIndex(String indexName) throws IOException {
        BooleanResponse exists = client.indices().exists(d -> d.index(indexName));
        return exists.value();
    }

    /**
     * 删除索引
     *
     * @param indexName
     * @throws IOException
     */
    public boolean deleteIndex(String indexName) throws IOException {
        DeleteIndexResponse response = client.indices().delete(d -> d.index(indexName));
        return true;
    }

    /**
     * 创建索引
     *
     * @param indexName
     * @return
     * @throws IOException
     */
    public boolean createIndex(String indexName) {
        try {
            CreateIndexResponse indexResponse = client.indices().create(c -> c.index(indexName));
        } catch (IOException e) {
            log.error("索引创建失败：{}", e.getMessage());
            // throw new ExploException(HttpCode.INDEX_CREATE_ERROR, "创建索引失败");
        }
        return true;
    }

    /**
     * 创建索引，不允许外部直接调用
     *
     * @param indexName
     * @param mapping
     * @throws IOException
     */
    private boolean createIndex(String indexName, Map<String, Property> mapping) throws IOException {
        CreateIndexResponse createIndexResponse = client.indices().create(c -> {
            c.index(indexName).mappings(mappings -> mappings.properties(mapping));
            return c;
        });
        return createIndexResponse.acknowledged();
    }


    /**
     * 重新创建索引,如果已存在先删除
     *
     * @param indexName
     * @param mapping
     */
    public void reCreateIndex(String indexName, Map<String, Property> mapping) {
        try {
            if (this.hasIndex(indexName)) {
                this.deleteIndex(indexName);
            }
        } catch (IOException e) {
            e.printStackTrace();
            log.error("删除索引失败：{}", e.getMessage());

            // throw new ExploException(HttpCode.INDEX_DELETE_ERROR, "删除索引失败");
        }

        try {
            this.createIndex(indexName, mapping);
        } catch (IOException e) {
            e.printStackTrace();
            log.error("重新创建索引失败：{}", e.getMessage());

            //throw new ExploException(HttpCode.INDEX_CREATE_ERROR, "重新创建索引失败");
        }
    }


    /**
     * 新增数据
     *
     * @param indexName
     * @throws IOException
     */
    public boolean insertDocument(String indexName, Object obj, String id) {
        try {
            IndexResponse indexResponse = client.index(i -> i
                    .index(indexName)
                    .id(id)
                    .document(obj));
            return true;
        } catch (IOException e) {
            log.error("数据插入ES异常：{}", e.getMessage());
            // throw new ExploException(HttpCode.ES_INSERT_ERROR, "ES新增数据失败");
        }
        return false;
    }

    /**
     * 查询数据
     *
     * @param indexName
     * @param id
     * @return
     */
    public GetResponse<User> searchDocument(String indexName, String id) {

        try {
            GetResponse<User> getResponse = client.get(g -> g
                            .index(indexName)
                            .id(id)
                    , User.class
            );
            return getResponse;
        } catch (IOException e) {
            log.error("查询ES异常：{}", e.getMessage());
            // throw new ExploException(HttpCode.ES_SEARCH_ERROR, "查询ES数据失败");
        }
        return null;
    }

    /**
     * 删除数据
     *
     * @param indexName
     * @param id
     * @return
     */
    public boolean deleteDocument(String indexName, String id) {
        try {
            DeleteResponse deleteResponse = client.delete(d -> d
                    .index(indexName)
                    .id(id)
            );
        } catch (IOException e) {
            log.error("删除Es数据异常：{}", e.getMessage());
            //throw new ExploException(HttpCode.ES_DELETE_ERROR, "数据删除失败");
        }
        return true;
    }




    /**
     *
     * 查询满足条件的数据  --User表
     * @param indexName 为索引名称
     * @param query 查询的内容
     * @param top 查询条数
     * @param field 参数
     * @return
     */
    public List<User> search(String indexName, String query, int top,String field) {
        List<User> documentParagraphs = new ArrayList<>();
        try {
            SearchResponse<User> search = client.search(s -> s
                            .index(indexName)
                            .query(q -> q
                                    .match(t -> t
                                            .field(field)
                                            .query(query)
                                    ))
                            .from(0)
                            .size(top)
                            .highlight(h -> h
                                    .fields("name", f -> f
                                            .preTags("<em>")
                                            .postTags("</em>")
                                    )
                            )
//                    .sort(f -> f.field(o -> o.field("docId").order(SortOrder.Desc)))
                    , User.class
            );
            for (Hit<User> hit : search.hits().hits()) {
                //User user = hit.source();
                User user = highLight(hit);
                documentParagraphs.add(user);

            }
        } catch (IOException e) {
            log.error("查询ES异常：{}", e.getMessage());
            // throw new ExploException(HttpCode.ES_SEARCH_ERROR, "查询ES数据失败");
        }
        return documentParagraphs;
    }
    /**
     * 高亮数据提取
     */
    private User highLight(Hit<User> hit) {
        User paragraph = hit.source();
        try {
            Map<String, List<String>> highlight = hit.highlight();
            List<String> list = highlight.get("name");
            String join = StringUtils.join(list, "");
            if (StringUtils.isNotBlank(join)) {
                paragraph.setName(join);
               // paragraph.setAge(hit.score());

            }
        } catch (Exception e) {
            log.error("获取ES高亮数据异常：{}", e.getMessage());
        }
        return paragraph;
    }



//    /**
//     *解析高亮数据
//     */
//    Map<String, List<String>> highlight = hit.highlight();
//    List<String> list = highlight.get("name");
//    String join = StringUtils.join(list, "");
//            if (StringUtils.isNotBlank(join)) {
//        paragraph.setContent(join);
//    }



    public List<User> searchCategoryIds(String indexName, String query, List<String> categoryId,int top) {
        List<User> documentParagraphs = new ArrayList<>();
        List<FieldValue> values = new ArrayList<>();
        for (String id : categoryId) {
            values.add(FieldValue.of(id));
        }
        Query categoryQuery = TermsQuery.of(t -> t.field("name.keyword").terms(new TermsQueryField.Builder()
                .value(values).build()
        ))._toQuery();

        try {
            SearchResponse<User> search = client.search(s -> s
                            .index(indexName)
                            .query(q -> q
                                    .bool(b -> b
                                            .must(categoryQuery
                                            )
                                            .should(sh -> sh
                                                    .match(t -> t
                                                            .field("name")
                                                            .query(query)
                                                    )
                                            )
                                    )
                            )
                            .highlight(h -> h
                                    .fields("name", f -> f
                                            .preTags("<em>")
                                            .postTags("</em>")
                                    )
                            )
                            .from(0)
                            .size(top)
                    , User.class
            );
            for (Hit<User> hit : search.hits().hits()) {
                User pd = hit.source();
                documentParagraphs.add(pd);
            }
        } catch (IOException e) {
            log.error("查询ES异常：{}", e.getMessage());
            // throw new ExploException(HttpCode.ES_SEARCH_ERROR, "查询ES数据失败");
        }
        return documentParagraphs;
    }

    /**
     * 高亮数据提取
//     */
//    private DocumentParagraph highLight(Hit<DocumentParagraph> hit) {
//        DocumentParagraph paragraph = hit.source();
//        try {
//            Map<String, List<String>> highlight = hit.highlight();
//            List<String> list = highlight.get("content");
//            String join = StringUtils.join(list, "");
//            if (StringUtils.isNotBlank(join)) {
//                paragraph.setContent(join);
//                paragraph.setScore(hit.score());
//
//            }
//        } catch (Exception e) {
//            log.error("获取ES高亮数据异常：{}", e.getMessage());
//        }
//        return paragraph;
//    }
//    /**
//     *解析高亮数据
//     */
//    Map<String, List<String>> highlight = hit.highlight();
//    List<String> list = highlight.get("content");
//    String join = StringUtils.join(list, "");
//            if (StringUtils.isNotBlank(join)) {
//        paragraph.setContent(join);
//    }

}

5. 过程中遇见的问题

5.1 项目启动时异常，pom依赖缺少

（原文章中只有前两个依赖，上边代码已添加）

<!-- 若不存在Spring Data ES的某个版本支持你下的ES版本,则使用  -->
<!-- ES 官方提供的在JAVA环境使用的依赖 -->
<dependency>
    <groupId>co.elastic.clients</groupId>
    <artifactId>elasticsearch-java</artifactId>
    <version>8.11.1</version>
</dependency>
<!-- 和第一个依赖是一起的,为了解决springboot项目的兼容性问题  -->
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
</dependency>
<!-- 原来只有前两依赖，但是启动报错找不到依赖，要注意版本号是否兼容  -->
<dependency>
    <groupId>jakarta.json</groupId>
    <artifactId>jakarta.json-api</artifactId>
    <version>2.0.1</version>
</dependency>
<!-- es的客户端，连接，版本号要与ES一致  -->
<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-client</artifactId>
    <version>8.11.1</version>
</dependency>

5.2 密码连接Es时

项目启动后，测试向es新建索引

5.2.1 报错Connet 超时

原因是是因为开启了 ssl 认证。

elasticsearch.yml 配置 xpack.security.http.ssl: enabled: false（true改成false)

5.2.2 报错 missing authentication credentials for REST…

[es/indices.create] failed: [security_exception] missing authentication credentials for REST request [/direct_connect…

因为设置了账号密码链接，但是config类未配置账号密码，原学习文章未包含以下代码，需在ElasticSearchConfig.class 中加上 1.1 和 1.1.1 配置（上边代码已添加）

// 1.1 设置用户名和密码 (账号密码连接，xpack.security.enabled: true)
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, password));


// 3. HttpClient 连接数配置
builder.setHttpClientConfigCallback(httpClientBuilder -> {
    httpClientBuilder.setMaxConnTotal(maxConnectNum);
    httpClientBuilder.setMaxConnPerRoute(maxConnectNumPerRoute);
    httpClientBuilder.disableAuthCaching();  // 1.1.1 设置账号密码
    httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
    return httpClientBuilder;
});