【Elastic Search 学习】

环境搭建

• 版本（各版本可自行搭配选择）
  • Java版本 jdk17
  • es版本 8.11.3
  • logstash 8.11.3
  • kibana 8.11.3
  • spring cloud 2022.0.3、springboot 3.0.2、cloud alibaba 2022.0.0.0-RC2

环境搭建

1. elastic、logstash、kibana下载 link
2. 配置文件修改

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: platform-es
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: 更换为你自己的数据文件路径
#
# Path to log files:
#
path.logs: 更换为你自己的日志文件路径
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 20-12-2023 09:34:29
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: false

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: false
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["node-1"]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

由于是单机测试使用，所以关闭了身份验证功能
在bin目录直接双击 elasticsearch.bat 文件即可启动

3. logstash配置

将logstash-sample.conf文件复制一份改名为logstash.conf
添加如下配置

input {
  stdin { }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "apache-%{+YYYY.MM.dd}"
  }
}

logstash.ym文件中指定es用户名和密码，es用户名和密码会在第一次启动时生成默认用户名为 elastic

 api.auth.basic.username: "elastic"
 api.auth.basic.password: "****"

启动logstash 需要在bin目录执行命令启动，指定logstash配置文件

logstash -f config/logstash.conf

4. kinbana配置

防止乱码

i18n.locale: "zh-CN"

环境搭建启动成功之后开始搭建项目
以下是自己搭建的的学习项目结构 已上传 Github
Github项目地址，欢迎学习交流

5. bootstrap.yml文件配置

server:
  port: 8914
spring:
  application:
    name: platform-es
  profiles:
    active: dev
  cloud:
    nacos:
      config:
        server-addr: 127.0.0.1:8848
        file-extension: yml
        namespace: ****************************
        group: DEFAULT_GROUP
        data-id: ${spring.application.name}.${spring.cloud.nacos.config.file-extension}
        refresh: true
      discovery:
        server-addr: 127.0.0.1:8848
        namespace: ****************************
        group: DEFAULT_GROUP
  elasticsearch:
    cluster-name: ${spring.application.name}
    cluster-nodes: 127.0.0.1:9200
    connection-timeout: 5000
    uris: 127.0.0.1:9200
    port: 9200
    host: 127.0.0.1
    username: elastic
    password: ******************

自己使用的nacos做配置中心及注册中心，多余配置不在一一贴出
6. pom.xml引入

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.learn.platform</groupId>
        <artifactId>platform-service</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <artifactId>platform-es</artifactId>
    <packaging>jar</packaging>

    <name>platform-es</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
        </dependency>



        <dependency>
            <groupId>co.elastic.clients</groupId>
            <artifactId>elasticsearch-java</artifactId>
            <version>8.11.3</version>
        </dependency>

        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.12.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo</groupId>
            <artifactId>dubbo-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

7. 编写config配之类

package com.learn.platform.config;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import co.elastic.clients.transport.ElasticsearchTransport;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.elasticsearch.client.RestClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @ClassName ElasticsearchConfig
 * @Description es 配之类
 * @Author xue
 * @Date 2023/12/21 10:58
 */
@Configuration
public class ElasticsearchConfig {


    @Value("${spring.elasticsearch.uris}")
    private String[] uris;
    @Value("${spring.elasticsearch.connection-timeout}")
    private Integer connectionTimeout;
    @Value("${spring.elasticsearch.port}")
    private Integer port;
    @Value("${spring.elasticsearch.host}")
    private String host;
    @Value("${spring.elasticsearch.username}")
    private String userName;
    @Value("${spring.elasticsearch.password}")
    private String password;


    @Bean
    public ElasticsearchClient elasticsearchClient(){
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(
                AuthScope.ANY, new UsernamePasswordCredentials(userName, password));

        RestClient client = RestClient.builder(new HttpHost(host, port))
                .setHttpClientConfigCallback(httpAsyncClientBuilder ->
                        httpAsyncClientBuilder.setDefaultCredentialsProvider(credentialsProvider))
                .build();
        ElasticsearchTransport transport = new RestClientTransport(client, new JacksonJsonpMapper());
        return new ElasticsearchClient(transport);
    }
}

8. service业务类

package com.learn.platform.service.Impl;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.query_dsl.MatchQuery;
import co.elastic.clients.elasticsearch._types.query_dsl.Query;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.elasticsearch.core.bulk.BulkOperation;
import co.elastic.clients.elasticsearch.core.search.Highlight;
import co.elastic.clients.elasticsearch.core.search.HighlightField;
import co.elastic.clients.elasticsearch.core.search.Hit;
import co.elastic.clients.elasticsearch.indices.CreateIndexRequest;
import co.elastic.clients.elasticsearch.indices.CreateIndexResponse;
import co.elastic.clients.elasticsearch.indices.ExistsRequest;
import co.elastic.clients.elasticsearch.indices.GetIndexResponse;
import co.elastic.clients.transport.endpoints.BooleanResponse;
import com.alibaba.fastjson2.JSONObject;
import com.learn.platform.entity.IndexReq;
import com.learn.platform.entity.es.SearchReq;
import com.learn.platform.entity.es.dto.EsUser;
import com.learn.platform.service.es.PlatformEsService;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.ObjectUtils;
import org.apache.dubbo.config.annotation.DubboService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.*;
import java.util.stream.Collectors;

/**
 * @ClassName PlatformEsServiceImpl
 * @Description es业务类
 * @Author xue
 * @Date 2023/12/21 10:12
 */
@Slf4j
@DubboService
@Service
public class PlatformEsServiceImpl implements PlatformEsService {


    /*  @Autowired
      RestClient restClient;*/
    @Autowired
    ElasticsearchClient elasticsearchClient;

    /**
     * 创建索引
     *
     * @param req 索引名称
     * @return Response 响应结果
     * @throws Exception 异常
     */
    public CreateIndexResponse createIndex(IndexReq req) throws Exception {
        boolean value = elasticsearchClient.indices().exists(ExistsRequest.of(a -> a.index(req.getIndexName()))).value();
        if (value) {
            throw new IllegalArgumentException("当前索引已存在");
        }
        CreateIndexResponse createIndexResponse = elasticsearchClient.indices().create(CreateIndexRequest.of(a -> a.index(req.getIndexName())
                .mappings(m -> m.properties(req.getMappings()))
        ));

        return createIndexResponse;
    }

    /**
     * 批量添加数据
     * @param indexName
     * @param list
     */
    public void addDataForIndex(String indexName,List list) {
        List<BulkOperation> bulkOperations = new ArrayList<>();
        try {
            for (Object o : list) {
                bulkOperations.add(new BulkOperation.Builder().create(d->d.document(o).id(UUID.randomUUID().toString().toUpperCase())).build());
            }
        }catch (Exception e) {
           e.printStackTrace();
        }
    }

    /**
     * 获取索引内容
     *
     * @param indexName 索引名称
     * @throws Exception
     */
    public GetIndexResponse getindex(String indexName) throws Exception {
        GetIndexResponse response = elasticsearchClient.indices().get(req -> req.index(indexName));
        log.info("获取索引结果 response={}", JSONObject.toJSONString(response));
        log.info("结果集{}", response.result());
        return response;
    }

    /**
     * 查询
     *
     * @param req 请求参数
     * @return
     * @throws IOException
     */
    public Object search(SearchReq req) throws IOException {
        MatchQuery matchQuery = new MatchQuery.Builder()
                .field(req.getField())
                .query(req.getQuery())
                .build();
        Query query1 = new Query.Builder().match(matchQuery).build();
        SearchRequest searchRequest = new SearchRequest.Builder()
                .index(req.getIndexName())
                .query(query1)
                .highlight(Highlight.of(a -> a.fields(req.getField(), HighlightField.of(b -> b.matchedFields(Collections.singletonList(req.getQuery()))))))
                .build();
        SearchResponse<EsUser> response = elasticsearchClient.search(searchRequest, EsUser.class);
        log.info("查询结果 result={}", response.hits());
        for (Hit<EsUser> hit : response.hits().hits()) {
            EsUser source = hit.source();
            assert source != null;
            Map<String, List<String>> highlight = hit.highlight();
            if (ObjectUtils.isNotEmpty(highlight)) {
                source.setHighLight(highlight.get(req.getField()).get(0));
            }
        }
        return response.hits().hits().stream().map(Hit::source).collect(Collectors.toList());
    }

    public boolean existsIndex(String indexName) throws IOException {
        BooleanResponse exists = elasticsearchClient.indices().exists(a -> a.index(indexName));
        return exists.value();

    }
}

9. controller层

package com.learn.platform.service.Impl;

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.query_dsl.MatchQuery;
import co.elastic.clients.elasticsearch._types.query_dsl.Query;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import co.elastic.clients.elasticsearch.core.SearchResponse;
import co.elastic.clients.elasticsearch.core.bulk.BulkOperation;
import co.elastic.clients.elasticsearch.core.search.Highlight;
import co.elastic.clients.elasticsearch.core.search.HighlightField;
import co.elastic.clients.elasticsearch.core.search.Hit;
import co.elastic.clients.elasticsearch.indices.CreateIndexRequest;
import co.elastic.clients.elasticsearch.indices.CreateIndexResponse;
import co.elastic.clients.elasticsearch.indices.ExistsRequest;
import co.elastic.clients.elasticsearch.indices.GetIndexResponse;
import co.elastic.clients.transport.endpoints.BooleanResponse;
import com.alibaba.fastjson2.JSONObject;
import com.learn.platform.entity.IndexReq;
import com.learn.platform.entity.es.SearchReq;
import com.learn.platform.entity.es.dto.EsUser;
import com.learn.platform.service.es.PlatformEsService;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.ObjectUtils;
import org.apache.dubbo.config.annotation.DubboService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.*;
import java.util.stream.Collectors;

/**
 * @ClassName PlatformEsServiceImpl
 * @Description es业务类
 * @Author xue
 * @Date 2023/12/21 10:12
 */
@Slf4j
@DubboService
@Service
public class PlatformEsServiceImpl implements PlatformEsService {


    /*  @Autowired
      RestClient restClient;*/
    @Autowired
    ElasticsearchClient elasticsearchClient;

    /**
     * 创建索引
     *
     * @param req 索引名称
     * @return Response 响应结果
     * @throws Exception 异常
     */
    public CreateIndexResponse createIndex(IndexReq req) throws Exception {
        boolean value = elasticsearchClient.indices().exists(ExistsRequest.of(a -> a.index(req.getIndexName()))).value();
        if (value) {
            throw new IllegalArgumentException("当前索引已存在");
        }
        CreateIndexResponse createIndexResponse = elasticsearchClient.indices().create(CreateIndexRequest.of(a -> a.index(req.getIndexName())
                .mappings(m -> m.properties(req.getMappings()))
        ));

        return createIndexResponse;
    }

    /**
     * 批量添加数据
     * @param indexName
     * @param list
     */
    public void addDataForIndex(String indexName,List list) {
        List<BulkOperation> bulkOperations = new ArrayList<>();
        try {
            for (Object o : list) {
                bulkOperations.add(new BulkOperation.Builder().create(d->d.document(o).id(UUID.randomUUID().toString().toUpperCase())).build());
            }
        }catch (Exception e) {
           e.printStackTrace();
        }
    }

    /**
     * 获取索引内容
     *
     * @param indexName 索引名称
     * @throws Exception
     */
    public GetIndexResponse getindex(String indexName) throws Exception {
        GetIndexResponse response = elasticsearchClient.indices().get(req -> req.index(indexName));
        log.info("获取索引结果 response={}", JSONObject.toJSONString(response));
        log.info("结果集{}", response.result());
        return response;
    }

    /**
     * 查询
     *
     * @param req 请求参数
     * @return
     * @throws IOException
     */
    public Object search(SearchReq req) throws IOException {
        MatchQuery matchQuery = new MatchQuery.Builder()
                .field(req.getField())
                .query(req.getQuery())
                .build();
        Query query1 = new Query.Builder().match(matchQuery).build();
        SearchRequest searchRequest = new SearchRequest.Builder()
                .index(req.getIndexName())
                .query(query1)
                .highlight(Highlight.of(a -> a.fields(req.getField(), HighlightField.of(b -> b.matchedFields(Collections.singletonList(req.getQuery()))))))
                .build();
        SearchResponse<EsUser> response = elasticsearchClient.search(searchRequest, EsUser.class);
        log.info("查询结果 result={}", response.hits());
        for (Hit<EsUser> hit : response.hits().hits()) {
            EsUser source = hit.source();
            assert source != null;
            Map<String, List<String>> highlight = hit.highlight();
            if (ObjectUtils.isNotEmpty(highlight)) {
                source.setHighLight(highlight.get(req.getField()).get(0));
            }
        }
        return response.hits().hits().stream().map(Hit::source).collect(Collectors.toList());
    }

    public boolean existsIndex(String indexName) throws IOException {
        BooleanResponse exists = elasticsearchClient.indices().exists(a -> a.index(indexName));
        return exists.value();

    }
}