ElasticSearch结合springboot常规用法总结

Wu_Teng飞

已于 2023-01-09 16:57:46 修改

阅读量4.5k

点赞数 11

分类专栏： java springboot es 文章标签： elasticsearch spring boot java

于 2022-08-19 11:00:38 首次发布

本文链接：https://blog.csdn.net/yaqiluobei136/article/details/126420569

版权

java 同时被 3 个专栏收录

3 篇文章 0 订阅

订阅专栏

springboot

3 篇文章 0 订阅

订阅专栏

1 篇文章 0 订阅

订阅专栏

ES结合springboot的常规用法总结：
引jar、配置工具类，客户端等忽略。

建议先熟悉利用kibana查询ES的语句，那样写起来更容易，也方便问题排查。

一、es查询必须的几个对象：

1. 普通查询的条件容器：BoolQueryBuilder 多条件查询容器

BoolQueryBuilder builder=QueryBuilders.boolQuery();    普通的QueryBuilders查询的boolQuery()方法，变成多条件查询
        比如时间条件：boolQueryBuilder.filter(QueryBuilders.rangeQuery(EsConstant.ES_INSERT_TIME).gte(twoHoursInfo.get(0)).lte(twoHoursInfo.get(1)));

2. 分组查询的条件容器：AggregationBuilder

 AggregationBuilder field = AggregationBuilders.terms(AGG_NAME).field(Constant.ES_TITLE);

3. 普通的分组查询：

注意如果要分组的字段是text类型，需要转成key类型才能分组，直接字段.key就行。在分组后直接.subAggregation(。。。。。) 可以添加子分组。

查询条件总容器：SearchSourceBuilder

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        比如：searchSourceBuilder.query(boolQueryBuilder).aggregation(termsAggregationBuilder);

4. 创建查询容器：SearchRequest

  方式一：SearchRequest searchRequest = new SearchRequest(INDEX_NAME).source(searchSourceBuilder);
    方式二：SearchRequest searchRequest = new SearchRequest().indices(INDEX_NAME).source(searchSourceBuilder);

5. 放入提前配置好的客户端：

SearchResponse是返回值的容器

SearchResponse search = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);

二、解析容器分为两种：

普通条件结果解析：
        方式一:
        TrendingResponse trendingResponse = new TrendingResponse();  //要装载结果的实体类
        SearchHit[] hits = searchResponse.getHits().getHits();
        int length = hits.length;
        if(length>0){
            Map<String, Object> sourceAsMap = hits[0].getSourceAsMap();
            trendingResponse = modelMapper.map(sourceAsMap, TrendingResponse.class);
        }
        方式二：
        SearchHit[] hits = searchResponse.getHits().getHits();
        if(hits.length>0){
            Arrays.asList(hits).stream().forEach(hit -> {
                TrendingResponse trendingResponse = new TrendingResponse();  //要装载结果的实体类
                Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                trendingResponse = modelMapper.map(sourceAsMap, TrendingResponse.class);
                trendingResponseList.add(trendingResponse);
             });
        }

分组条件结果解析：

 方式一：
        ParsedStringTerms aggregation = searchResponse1.getAggregations().get(AGG_NAME);
        List<? extends Terms.Bucket> buckets = aggregation.getBuckets();
        for(Terms.Bucket bucket:buckets){
            TrendingResponse trendingResponse = new TrendingResponse();  //要装载结果的实体类
            trendingResponse.setPlatform(bucket.getKeyAsString());       //封装的解析方法
            trendingResponseList.add(trendingResponse);
        }    
        
        方式二：
        ParsedTerms terms = response.getAggregations().get("termsAgg")
        return  terms.getBuckets().stream().map(it -> new Target(it.getKeyAsString(),it.getDocCount(),
        dateHistogramAgg(it.getAggregations(), "yyyy-MM-dd HH:mm"))).collect(Collectors.toList());  注意：多利用类的有参构造：直接转换成需要的类对象

这是查询数据内容的方式，每次只能查出10000条，这是es默认的。如果想查出更多，可以采用滚动查询，这个以后再总结

三、日常用法：

1. 只是查询总数的写法：

 logger.info("getAllCount 统计总数");
        long value = 0;
        List<Long> twoHoursInfo = new ArrayList<>();
        twoHoursInfo.add(System.currentTimeMillis()-86400000);
        twoHoursInfo.add(System.currentTimeMillis());
        String[] indexs = new String[]{EsConstant.ES_INDEX_BASEDATA,EsConstant.ES_INDEX_BOARD,EsConstant.ES_INDEX_APPS,EsConstant.ES_INDEX_ACCOUNT};
        CountRequest countRequest = new CountRequest(indexs,QueryBuilders.rangeQuery(EsConstant.ES_INSERT_TIME).gte(twoHoursInfo.get(0)).lte(twoHoursInfo.get(1)));
        try {
            CountResponse count = restHighLevelClient.count(countRequest, RequestOptions.DEFAULT);
            value = count.getCount();
        } catch (IOException e) {
            logger.error("插入基础数据getMsgTypeData 报错:{}",e);
        }
        return value;

2.利用es的分词进行，匹配实现模糊查询：

 boolQueryBuilder.filter(QueryBuilders.wildcardQuery(EsConstant.ES_PRODUCT_NAME,"*"+productName+"*")); 
    注意：ElasticSearchTemplate加中文筛选后无结果的解决办法：title.keyword

3. es实现简单的分页查询：(不推荐)

 new SearchSourceBuilder().query(boolQueryBuilder).from(pageNum*pageSize).size(pageSize);

4. 常规查询工具类

@Component
public class BaseSearchSourceBuilder extends BaseAggregationBuilder {

    @Resource
    private DetectionConfigRepository detectionConfigRepository;

    public SearchSourceBuilder searchSourceBuilder(HeartbeatDetectionListVO condition){
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        Long startTime = null;
        Long endTime =  null;
        if(Objects.nonNull(condition.getStartTime())){
            startTime = condition.getStartTime().getTime();
            endTime = condition.getEndTime().getTime();
        }

        if (Objects.nonNull(startTime) && Objects.nonNull(endTime)){
                                        
boolQueryBuilder.filter(QueryBuilders.rangeQuery(EsConstant.ES_DETECTION_TIME_STMP).gte(startTime).lte(endTime));
        }
        String keyword = condition.getStatus();
        if (StringUtils.isNotBlank(keyword)){
            long state = 0;
            if(keyword.equals("ONLINE")){
                state=1;
            }else {
                state=2;
            }
            //TODO 与或非
            TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery(EsConstant.ES_RESULT_STATUS, state);
            boolQueryBuilder.filter(termQueryBuilder);
        }
        String productCode = condition.getProductCode();
        if (StringUtils.isNotBlank(productCode)){
            boolQueryBuilder.filter(QueryBuilders.termQuery(EsConstant.ES_PRODUCT_CODE,productCode));
        }
        String resultCode = condition.getResultCode();
        if (StringUtils.isNotBlank(resultCode)){
            boolQueryBuilder.filter(QueryBuilders.termQuery(EsConstant.ES_RESULT_CODE,resultCode));
        }
        List<String> messageList = condition.getMessageList();
        if(CollectionUtils.isNotEmpty(messageList)){
            boolQueryBuilder.filter(QueryBuilders.termsQuery(EsConstant.ES_RESULT_CODE,messageList));
        }
        String productName = condition.getProductName();
        if (StringUtils.isNotBlank(productName)){
            boolQueryBuilder.filter(QueryBuilders.wildcardQuery(EsConstant.ES_PRODUCT_NAME,"*"+productName+"*"));
        }
        String message = condition.getMessage();
//        if (StringUtils.isNotBlank(message)){
//            boolQueryBuilder.filter(QueryBuilders.queryStringQuery(message).defaultField(EsConstant.ES_RESULT_REASON));
//        }
        if (StringUtils.isNotBlank(message)){
            boolQueryBuilder.filter(QueryBuilders.termQuery(EsConstant.ES_RESULT_CODE,message));
        }
        Integer pageSize = condition.getSize();
        if (Objects.isNull(pageSize)){
            pageSize = 0;
        }
        Integer pageNum = condition.getNumber();
        if (Objects.isNull(pageNum)){
            pageNum = 0;
        }
        return new SearchSourceBuilder().query(boolQueryBuilder).from(pageNum*pageSize).size(pageSize);
    }
}

5. 聚合查询工具类：

public class BaseAggregationBuilder {

    /**
     * TopHitsAggregation sort <>pt asc</>
     * @param topHitsName
     * @return
     */
    public TopHitsAggregationBuilder topHitsAggregationBuilder(String topHitsName, Integer size){
        return AggregationBuilders.topHits(topHitsName).sort(EsConstant.ES_DETECTION_TIME, SortOrder.DESC).size(size);
    }
    /**
     * TermsAggregation
     * @param termsAggName
     * @param fieldName
     * @param size
     * @return
     */
    public TermsAggregationBuilder termsAggregationBuilder(String termsAggName, String fieldName, Integer size){
        return AggregationBuilders.terms(termsAggName).field(fieldName).size(size).executionHint("map");
    }

    /**
     * AggregationBuilders
     * @param cardinalityAggName
     * @param fieldName
     * @param
     * @return
     */
    public CardinalityAggregationBuilder cardinalityAggregationBuilder(String cardinalityAggName, String fieldName){
        return AggregationBuilders.cardinality(cardinalityAggName).field(fieldName);
    }

    /**
     * DateHistogramAggregation
     * @param dateHistogramFieldName
     * @return
     */
    public DateHistogramAggregationBuilder dateHistogramAggregationBuilder(String dateHistogramName, String dateHistogramFieldName){
        return AggregationBuilders.dateHistogram(dateHistogramName).fixedInterval(new DateHistogramInterval("1d"))
                .field(dateHistogramFieldName);
    }

    /**
     * DateHistogramAggregation
     * @param dateHistogramFieldName
     * @return
     */
//    public DateHistogramAggregationBuilder dateHistogramAggregationFormatBuilder(String dateHistogramName, String dateHistogramFieldName,
//                                                                           DateHistogramInterval interval,Long startTime,Long endTime){
//        return AggregationBuilders.dateHistogram(dateHistogramName).fixedInterval(interval)
//                .field(dateHistogramFieldName).format("yyyy-MM-dd HH:mm").minDocCount(0).extendedBounds(new LongBounds(startTime,endTime)).timeZone(ZoneId.of("Asia/Shanghai"));
//    }

    /**
     * SumAggregation
     * @param sumAggName
     * @param fieldName
     * @return
     */
    public SumAggregationBuilder sumAggregationBuilder(String sumAggName,String fieldName){
        return AggregationBuilders.sum(sumAggName).field(fieldName);
    }

    /**
     * AvgAggregation
     * @param avgAggName
     * @param fieldName
     * @return
     */
    public AvgAggregationBuilder avgAggregationBuilder(String avgAggName, String fieldName){
        return AggregationBuilders.avg(avgAggName).field(fieldName);
    }

    /**
     * MaxAggregation
     * 最大值
     */
    public MaxAggregationBuilder maxAggregationBuilder(String maxAggName, String fieldName){
        return AggregationBuilders.max(maxAggName).field(fieldName);
    }

    /**
     * MinAggregation
     * 最小值
     */
    public MinAggregationBuilder minAggregationBuilder(String maxAggName, String fieldName){
        return AggregationBuilders.min(maxAggName).field(fieldName);
    }



}

6. 多查询并列查询：

当需要多条件查询的时候，尽量一条语句全部查出来，这样性能好很多，类似sql语句的

where (条件一 and 条件二 and 条件三) or (条件四 and 条件五 and 条件六) or 。。。

public  List<Map<String, Object>> getImageInfoEs(List<SearchImageInfo> listSearchImageInfo){
        String[] indexs = new String[]{EsConstant.BASE_DATA,EsConstant.INDEX_APPS,EsConstant.INDEX_ACCOUNT,EsConstant.INDEX_ACCOUNT};
        SearchSourceBuilder searchSourceBuilder = searchSourceBuilder(listSearchImageInfo);
        SearchResponse search = esService.search(indexs, searchSourceBuilder);
        SearchHit[] hits = search.getHits().getHits();
        log.info("================从es查询的结果:hits.length : ==============={}",hits.length);
        Map<String, SearchImageInfo> searchImageInfoMap = new HashMap<>();
        listSearchImageInfo.stream().forEach(t->{
            String key = t.getMedia_id()+""+t.getMsg_id_s()+""+t.getMsg_type();
            searchImageInfoMap.put(key,t);
        });
        List<Map<String, Object>> listMap = new ArrayList<>();
        if(hits.length>0){
            Arrays.asList(hits).stream().forEach(hit -> {
                Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                String keylog = sourceAsMap.get(EsConstant.MEDIA_ID)+"------"+sourceAsMap.get(EsConstant.MSG_ID_S)+"-------"+sourceAsMap.get(EsConstant.MSG_TYPE);
                log.info("-------------------------从es查询的结果--------------------{}",keylog);
                sourceAsMap.put("_id",hit.getId());
                sourceAsMap.put("index",hit.getIndex());
                ......业务代码处理
                listMap.add(sourceAsMap);
            });
        }
        return listMap;
    }

    public SearchSourceBuilder searchSourceBuilder(List<SearchImageInfo> listSearchImageInfo){
        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
        listSearchImageInfo.stream().forEach(t->{
            //通过should进行多条件拼接
                boolQueryBuilder.should(QueryBuilders.boolQuery().must(QueryBuilders.termQuery(EsConstant.MEDIA_ID,t.getMedia_id())).must(QueryBuilders.termQuery(EsConstant.MSG_ID_S,t.getMsg_id_s())).must(QueryBuilders.termQuery(EsConstant.MSG_TYPE,t.getMsg_type())));
        });
        return new SearchSourceBuilder().query(boolQueryBuilder).from(0).size(1000);
    }

7. 只根据数据的es id查询，效率高

根据数据的id直接查询：

方式一：get方式不能用索引别名，但是效率高

		public GetResponse getOne(String index, String docId) {
        GetRequest getRequest = new GetRequest(index,docId);
        getRequest.preference(prefrence());
        GetResponse getResponse = null;
        try {
            getResponse = client.get(getRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            e.printStackTrace();
            log.error("getOne fail,cause by {}", e.getCause());
        }
        return getResponse;
    }
	
方式二：这种方式可以，但是套的层比get方式多，效率略不如。

	//id可以是一个字符串类型的数组
	public SearchResponse getOneById(String index, String docId) {
        SearchRequest searchRequest = new SearchRequest(index);//指定索引
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.idsQuery().addIds(docId));
        searchRequest.source(searchSourceBuilder);
        SearchResponse search = null;
        try {
            search = client.search(searchRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            log.error("getOne fail,cause by {}", e.getCause());
        }
        return search;
    }