Elasticsearch单机版整合Java从无到有

最新推荐文章于 2024-05-29 13:52:57 发布

小天啊啊

最新推荐文章于 2024-05-29 13:52:57 发布

阅读量655

点赞数 2

文章标签： elasticsearch java 大数据

本文链接：https://blog.csdn.net/weixin_44900881/article/details/125765637

版权

一、安装ElasticSearch和Kibana（本人是基于Docker方式安装）

注意：ElasticSearch和Kibana版本保持一致，ElasticSearch的版本和spirngboot版本对应，和JDK版本对应，不然后续会有很多问题，版本对应自己上网查。本次安装是ElasticSearch版本是7.14.0

安装ElasticSearch

1.获取镜像
docker pull elasticsearch:7.14.0
2.运行es
docker run -d -p 9200:9200 -p 9300:9300 -e “discovery.type=single-node” elasticsearch:7.14.0
3.访问ES
http://127.0.0.1:9200/

安装kibana

1.获取镜像
docker pull kibana:7.14.0
2.运行kibana
docker run -d --name kibana -p 5601:5601 kibana:7.14.0
3.基于数据卷加载配置文件方式运行
a.从容器复制kibana配置文件出来
b.修改配置文件为对应ES服务器地址
c.通过数据卷加载配置文件方式启动
`docker run -d -v /root/kibana.yml:/usr/share/kibana/config/kibana.yml --name kibana -p 5601:5601 kibana:7.14.0
4.访问kibana
http://127.0.0.1:5601

基于docker-compose方式启动ElasticSearch和kibana

安装docker-compose
curl -L https://get.daocloud.io/docker/compose/releases/download/v2.3.4/docker-compose-uname -s-uname -m > /usr/local/bin/docker-compose
赋予权限
chmod +x /usr/local/bin/docker-compose
输入命令有版本号证明成功
docker-compose version
编写docker-compose.yml

version: "3.8"
networks:
  es:
services:
  elasticsearch:
    image: elasticsearch:7.14.0
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - "es"
    environment:
      - "discovery.type=single-node" //单节点启动
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" 
    volumes:
      - ./elasticsearch/data:/usr/share/elasticsearch/data //指定es数据存放位置
      - ./ik-7.14.0:/usr/share/elasticsearch/plugins/ik-7.14.0 //指定ik分词器

  kibana:
    image: kibana:7.14.0
    ports:
      - "5601:5601"
    networks:
      - "es"
    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml //指定kibana的配置文件

编写kibana.yml

server.host: "0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "http://elasticsearch:9200" ] //指定es位置
monitoring.ui.container.elasticsearch.enabled: true

docker相关命令

查看docker 镜像
docker images
查看docker 运行镜像
docker ps
查看日志
docker logs -f 进程号
进入运行镜像的物理地址
docker exec -it 进程号 bash

docker-compose相关命令

docker-compose up -d 启动
docker-compose down 停止

安装ik分词器

开源分词器 Ik 的github:https://github.com/medcl/elasticsearch-analysis-ik 自己去下载版本和es保持一致

二、JAVA整合ElasticSearch

引入pom.xml

        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
        </dependency>

编写ElasticSearchConfig 项目启动就连接ElasticSearch

import org.apache.http.HttpHost;
import org.elasticsearch.client.*;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ElasticSearchConfig {

     //esIP地址
    @Value("${elasticsearch.host}")
    private String host;
    //es 端口号
    @Value("${elasticsearch.port}")
    private Integer port;
    @Bean
    public RestHighLevelClient client() {
        RestClientBuilder builder=RestClient.builder(new HttpHost(host, port, "http"));
        RestHighLevelClient client = new RestHighLevelClient(builder);
        return client;
    }
}

三、ElasticSearch相关工具类

mport com.alibaba.fastjson.JSON;
import org.apache.http.HttpHost;
import org.apache.poi.ss.formula.functions.T;
import org.elasticsearch.action.admin.indices.create.CreateIndexRequest;
import org.elasticsearch.action.admin.indices.create.CreateIndexResponse;
import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
import org.elasticsearch.action.admin.indices.get.GetIndexRequest;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.delete.DeleteRequest;
import org.elasticsearch.action.delete.DeleteResponse;
import org.elasticsearch.action.get.GetRequest;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.action.support.master.AcknowledgedResponse;
import org.elasticsearch.action.update.UpdateRequest;
import org.elasticsearch.action.update.UpdateResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;

import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.xcontent.XContentType;

import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.elasticsearch.search.suggest.Suggest;
import org.elasticsearch.search.suggest.SuggestBuilder;
import org.elasticsearch.search.suggest.SuggestBuilders;
import org.elasticsearch.search.suggest.completion.CompletionSuggestion;
import org.elasticsearch.search.suggest.completion.CompletionSuggestionBuilder;
import org.elasticsearch.search.suggest.phrase.PhraseSuggestionBuilder;
import org.elasticsearch.search.suggest.term.TermSuggestionBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.stereotype.Component;
import org.springframework.web.bind.annotation.RestController;

import javax.annotation.PostConstruct;
import javax.annotation.Resource;
import java.io.IOException;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.*;

/**
 * @author zjp
 * @create 2022-06-27 10:16
 */
@Component
public  class EsUtil {

    private static RestHighLevelClient client;

    @Autowired
    public EsUtil(RestHighLevelClient client) {
        this.client = client;
    }

    /**
     * 默认类型(整个type类型传闻在8.0版本后可能会废弃,但是目前7.13版本下先保留)
     */
    public static final String DEFAULT_TYPE = "_doc";

    /**
     * set方法前缀
     */
    public static final String SET_METHOD_PREFIX = "set";

    /**
     * 返回状态-CREATED
     */
    public static final String RESPONSE_STATUS_CREATED = "CREATED";

    /**
     * 返回状态-OK
     */
    public static final String RESPONSE_STATUS_OK = "OK";

    /**
     * 返回状态-NOT_FOUND
     */
    public static final String RESPONSE_STATUS_NOT_FOUND = "NOT_FOUND";

    /**
     * 需要过滤的文档数据
     */
    public static final String[] IGNORE_KEY = {"@timestamp", "@version", "type"};

    /**
     * 超时时间 1s
     */
    public static final TimeValue TIME_VALUE_SECONDS = TimeValue.timeValueSeconds(1);

    /**
     * 批量新增
     */
    public static final String PATCH_OP_TYPE_INSERT = "insert";

    /**
     * 批量删除
     */
    public static final String PATCH_OP_TYPE_DELETE = "delete";

    /**
     * 批量更新
     */
    public static final String PATCH_OP_TYPE_UPDATE = "update";

//==========================================数据操作(工具)(不参与调用es)=================================================

    /**
     * 方法描述: 剔除指定文档数据,减少不必要的循环
     *
     * @param map 文档数据
     * @return: void
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:39 上午
     */
    private static void ignoreSource(Map<String, Object> map) {
        for (String key : IGNORE_KEY) {
            map.remove(key);
        }
    }


    /**
     * 方法描述: 将文档数据转化为指定对象
     *
     * @param sourceAsMap 文档数据
     * @param clazz       转换目标Class对象
     * @return 对象
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:38 上午
     */
    private static <T> T dealObject(Map<String, Object> sourceAsMap, Class<T> clazz) {
        try {
            ignoreSource(sourceAsMap);
            Iterator<String> keyIterator = sourceAsMap.keySet().iterator();
            T t = clazz.newInstance();
            while (keyIterator.hasNext()) {
                String key = keyIterator.next();
                String replaceKey = key.replaceFirst(key.substring(0, 1), key.substring(0, 1).toUpperCase());
                Method method = null;
                try {
                    method = clazz.getMethod(SET_METHOD_PREFIX + replaceKey, sourceAsMap.get(key).getClass());
                } catch (NoSuchMethodException e) {
                    continue;
                }
                method.invoke(t, sourceAsMap.get(key));
            }
            return t;
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }


//==========================================索引操作=================================================

    /**
     * 方法描述: 创建索引,若索引不存在且创建成功,返回true,若同名索引已存在,返回false
     *
     * @param: [index] 索引名称
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 11:01 上午
     */
    public static boolean insertIndex(String index) {
        //创建索引请求
        CreateIndexRequest request = new CreateIndexRequest(index);
        //执行创建请求IndicesClient,请求后获得响应
        try {
            CreateIndexResponse response = client.indices().create(request, RequestOptions.DEFAULT);
            return response != null;
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }


    /**
     * 方法描述: 判断索引是否存在,若存在返回true,若不存在或出现问题返回false
     *
     * @param: [index] 索引名称
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 11:09 上午
     */
    public static boolean isExitsIndex(String index) {
        GetIndexRequest request = new GetIndexRequest();
        request.indices(index);
        try {
            return client.indices().exists(request, RequestOptions.DEFAULT);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }


    /*
     * 方法描述: 删除索引,删除成功返回true,删除失败返回false
     * @param: [index] 索引名称
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 11:23 上午
     */
    public static boolean deleteIndex(String index) {
        DeleteIndexRequest request = new DeleteIndexRequest(index);
        try {
            AcknowledgedResponse response = client.indices().delete(request, RequestOptions.DEFAULT);
            return response.isAcknowledged();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }


//==========================================文档操作(新增,删除,修改)=================================================

    /**
     * 方法描述: 新增/修改文档信息
     *
     * @param index 索引
     * @param id    文档id
     * @param data  数据
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:34 上午
     */
    public static boolean insertOrUpdateDocument(String index, String id, Object data) {
        try {
            IndexRequest request = new IndexRequest(index,DEFAULT_TYPE);
            request.timeout(TIME_VALUE_SECONDS);
            if (id != null && id.length() > 0) {
                request.id(id);
            }
            request.source(JSON.toJSONString(data), XContentType.JSON);
            IndexResponse response = client.index(request, RequestOptions.DEFAULT);
            String status = response.status().toString();
            if (RESPONSE_STATUS_CREATED.equals(status) || RESPONSE_STATUS_OK.equals(status)) {
                return true;
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }

    /**
     * 方法描述: 更新文档信息
     *
     * @param index 索引
     * @param id    文档id
     * @param data  数据
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:34 上午
     */
    public static boolean updateDocument(String index, String id, Object data) {
        try {
            UpdateRequest request = new UpdateRequest(index,DEFAULT_TYPE,id);
            request.doc(JSON.toJSONString(data), XContentType.JSON);
            UpdateResponse response = client.update(request, RequestOptions.DEFAULT);
            String status = response.status().toString();
            if (RESPONSE_STATUS_OK.equals(status)) {
                return true;
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }

    /**
     * 方法描述:删除文档信息
     *
     * @param index 索引
     * @param id    文档id
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:33 上午
     */
    public static boolean deleteDocument(String index, String id) {
        try {
            DeleteRequest request = new DeleteRequest(index, DEFAULT_TYPE,id);
            DeleteResponse response = client.delete(request, RequestOptions.DEFAULT);
            String status = response.status().toString();
            if (RESPONSE_STATUS_OK.equals(status)) {
                return true;
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }


    /**
     * 方法描述: 小数据量批量新增
     *
     * @param index    索引
     * @param dataList 数据集 新增修改需要传递
     * @param timeout  超时时间 单位为秒
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:31 上午
     */
    public static boolean simplePatchInsert(String index, List<Object> dataList, long timeout) {
        try {
            BulkRequest bulkRequest = new BulkRequest();
            bulkRequest.timeout(TimeValue.timeValueSeconds(timeout));
            if (dataList != null && dataList.size() > 0) {
                for (Object obj : dataList) {
                    bulkRequest.add(
                            new IndexRequest(index)
                                    .source(JSON.toJSONString(obj), XContentType.JSON)
                    );
                }
                BulkResponse response = client.bulk(bulkRequest, RequestOptions.DEFAULT);
                if (!response.hasFailures()) {
                    return true;
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }


//==========================================文档操作(查询)=================================================

    /**
     * 方法描述: 判断文档是否存在
     *
     * @param index 索引
     * @param id    文档id
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:36 上午
     */
    public static boolean isExistsDocument(String index, String id) {
        return isExistsDocument(index, DEFAULT_TYPE, id);
    }


    /**
     * 方法描述: 判断文档是否存在
     *
     * @param index 索引
     * @param type  类型
     * @param id    文档id
     * @return: boolean
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:36 上午
     */
    public static boolean isExistsDocument(String index, String type, String id) {
        GetRequest request = new GetRequest(index, type, id);
        try {
            GetResponse response = client.get(request, RequestOptions.DEFAULT);
            return response.isExists();
        } catch (Exception e) {
            e.printStackTrace();
        }
        return false;
    }


    /**
     * 方法描述: 根据id查询文档
     *
     * @param index 索引
     * @param id    文档id
     * @param clazz 转换目标Class对象
     * @return 对象
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:36 上午
     */
    public static <T> T selectDocumentById(String index, String id, Class<T> clazz) {
        return selectDocumentById(index, DEFAULT_TYPE, id, clazz);
    }


    /**
     * 方法描述: 根据id查询文档
     *
     * @param index 索引
     * @param type  类型
     * @param id    文档id
     * @param clazz 转换目标Class对象
     * @return 对象
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:35 上午
     */
    public static <T> T selectDocumentById(String index, String type, String id, Class<T> clazz) {
        try {
            type = type == null || type.equals("") ? DEFAULT_TYPE : type;
            GetRequest request = new GetRequest(index, type, id);
            GetResponse response = client.get(request, RequestOptions.DEFAULT);
            if (response.isExists()) {
                Map<String, Object> sourceAsMap = response.getSourceAsMap();
                return dealObject(sourceAsMap, clazz);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }


    /**
     * 方法描述:（筛选条件）获取数据集合
     *
     * @param index         索引
     * @param sourceBuilder 请求条件
     * @param clazz         转换目标Class对象
     * @return: java.util.List<T>
     * @author: zjp
     * @date: 2021/6/29
     * @time: 10:35 上午
     */
    public static <T> List<T> selectDocumentList(String index, SearchSourceBuilder sourceBuilder, Class<T> clazz) {
        try {
            SearchRequest request = new SearchRequest(index);
            if (sourceBuilder != null) {
                // 返回实际命中数
                sourceBuilder.trackTotalHits(true);
                request.source(sourceBuilder);
            }
            SearchResponse response = client.search(request, RequestOptions.DEFAULT);
            if (response.getHits() != null) {
                List<T> list = new ArrayList<>();
                SearchHit[] hits = response.getHits().getHits();
                for (SearchHit documentFields : hits) {
                    Map<String, Object> sourceAsMap = documentFields.getSourceAsMap();
                    list.add(dealObject(sourceAsMap, clazz));
                }
                return list;
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }


    /**
     * 方法描述: 筛选查询,返回使用了<span style='color:red'></span>处理好的数据.
     *
     * @param: index 索引名称
     * @param: sourceBuilder sourceBuilder对象
     * @param: clazz 需要返回的对象类型.class
     * @param: fieldNames 需要表现的高亮匹配字段
     * @return: java.util.List<T>
     * @author: zjp
     * @date: 2021/6/29
     * @time: 6:39 下午
     */
    public static <T> List<T> selectDocumentListHighLight(String index, SearchSourceBuilder sourceBuilder, Class<T> clazz, String[] fieldNames) {
        try {
            SearchRequest request = new SearchRequest(index);
            if (sourceBuilder != null) {
                // 返回实际命中数
                sourceBuilder.trackTotalHits(true);
                //高亮
                HighlightBuilder highlightBuilder = new HighlightBuilder();
                if(null!=fieldNames && fieldNames.length>0){
                    for (String highLight : fieldNames) {
                        highlightBuilder.field(highLight);
                    }
                }
                highlightBuilder.requireFieldMatch(false);//多个高亮关闭
                highlightBuilder.preTags("<span style='color:red'>");
                highlightBuilder.postTags("</span>");
                sourceBuilder.highlighter(highlightBuilder);
                request.source(sourceBuilder);
            }
            SearchResponse response = client.search(request, RequestOptions.DEFAULT);
            if (response.getHits() != null) {
                List<T> list = new ArrayList<>();
                for (SearchHit documentFields : response.getHits().getHits()) {
                    Map<String, HighlightField> highlightFields = documentFields.getHighlightFields();
                    Map<String, Object> sourceAsMap = documentFields.getSourceAsMap();
                    if(null!=fieldNames && fieldNames.length>0){
                        for (String highLight : fieldNames) {
                            HighlightField title = highlightFields.get(highLight);
                            if (title != null) {
                                Text[] fragments = title.fragments();
                                String n_title = "";
                                for (Text fragment : fragments) {
                                    n_title += fragment;
                                }
                                sourceAsMap.put(highLight, n_title);//高亮替换原来的内容
                            }
                        }
                    }
                    list.add(dealObject(sourceAsMap, clazz));
                }
                return list;
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }



    /**
     * @author ZJP
     * @Description  搜索提示自动不全
     * @date 2022/6/29 13:59
     * @param index 索引名称
     * @param prefix 搜索关键字
     * @param fieldName 搜索存储信息的字段
     * @return
     */
    public static List<String> getSuggestions(String index,String prefix,String fieldName) {
        try {
            // 1.准备Request
            SearchRequest request = new SearchRequest(index);
            // 2.准备DSL
            request.source().suggest(new SuggestBuilder().addSuggestion(
                    "suggestions",
                    SuggestBuilders.completionSuggestion(fieldName)
                            .prefix(prefix)
                            .skipDuplicates(true)
                            .size(10)
            ));
            // 3.发起请求
            SearchResponse response = client.search(request, RequestOptions.DEFAULT);
            // 4.解析结果
            Suggest suggest = response.getSuggest();
            // 4.1.根据补全查询名称，获取补全结果
            CompletionSuggestion suggestions = suggest.getSuggestion("suggestions");
            // 4.2.获取options
            List<CompletionSuggestion.Entry.Option> options = suggestions.getOptions();
            // 4.3.遍历
            List<String> list = new ArrayList<>(options.size());
            for (CompletionSuggestion.Entry.Option option : options) {
                String text = option.getText().toString();
                list.add(text);
            }
            return list;
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    /**
     * @author ZJP
     * @Description 获取字段名称
     * @date 2022/6/29 14:08
     * @param
     * @return
     */
    public static List<Field> getFields(Object obj) {
        Field[] fields = obj.getClass().getDeclaredFields();
        List<Field> fieldList = new ArrayList<>();
        fieldList.addAll(Arrays.asList(fields));
        return fieldList;
    }
}