SpringBoot整合ES——ElasticSearch&&多种复杂查询api的使用

ElasticSearch

阅读本文前请注意，本文仅仅展示springboot整合es中大部分场景api的操作，对其概念并没有过多的阐述，想获得更完整的文档，请查阅官方文档 https://www.elastic.co/cn/elasticsearch/

在介绍使用之前，先对比一下es与mysql关键字术语描述的比较吧

1、配置环境

1、导入依赖

es镜像： elasticsearch:7.14.0
kibana镜像： kibana:7.14.0
ik分词器：ik-7.14.0
该es版本与springboot 2.3.10.RELEASE对应
注意：es与spring需要有严格的版本对应包括与分词器的对应

<!--elasticsearch-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
            <version> 2.3.10.RELEASE</version>
        </dependency>

2、配置文件

elasticsearch:
  host: 192.168.137.157:9200

3、配置客户端

@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {

  @Value("${elasticsearch.host}")
  private String host;

  @Override
  @Bean
  public RestHighLevelClient elasticsearchClient() {
   final ClientConfiguration clientConfiguration = ClientConfiguration
            .builder().connectedTo(host).build();
    return RestClients.create(clientConfiguration).rest();
  }
}

2、Rest-索引库

1、创建索引库

1、定义常量字符串保存创建索引库操作

public class HotelConstants {
  public static final String MAPPING_TEMPLATE =" {\n" +
          "    \"properties\": {\n" +
          "      \"id\":{\n" +
          "        \"type\":\"keyword\"\n" +
          "      },\n" +
          "      \"name\":{\n" +
          "        \"type\": \"text\",\n" +
          "        \"analyzer\": \"ik_max_word\",\n" +
          "        \"copy_to\": \"all\"\n" +
          "      },\n" +
          "      \"address\":{\n" +
          "        \"type\": \"keyword\",\n" +
          "        \"index\": false\n" +
          "      },\n" +
          "      \"price\":{\n" +
          "        \"type\": \"integer\"\n" +
          "      },\n" +
          "      \"score\":{\n" +
          "        \"type\": \"integer\"\n" +
          "      },\n" +
          "      \"brand\":{\n" +
          "        \"type\": \"keyword\"\n" +
          "      },\n" +
          "      \"city\":{\n" +
          "        \"type\": \"keyword\"\n" +
          "      },\n" +
          "      \"starName\":{\n" +
          "        \"type\": \"keyword\"\n" +
          "      },\n" +
          "      \"business\":{\n" +
          "        \"type\": \"keyword\",\n" +
          "        \"copy_to\": \"all\"\n" +
          "      },\n" +
          "      \"location\":{\n" +
          "        \"type\":\"geo_point\"\n" +
          "      },\n" +
          "      \"pic\":{\n" +
          "        \"type\": \"keyword\",\n" +
          "        \"index\": false\n" +
          "      },\n" +
          "      \"all\":{\n" +
          "        \"type\": \"text\",\n" +
          "        \"analyzer\": \"ik_max_word\"        \n" +
          "      }\n" +
          "    }\n" +
          "  }";

2、restclient方式创建索引库

/**
 * 创建索引
 * @throws IOException
 */
@Test
void createIndex() throws IOException {
  CreateIndexRequest request = new CreateIndexRequest("hotel"); // 索引库名
  request.mapping(MAPPING_TEMPLATE, XContentType.JSON); // 常量静态导入

  // 发起请求
  CreateIndexResponse createIndexResponse = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
}

2、判断索引库是否存在

/**
   * 索引是否存在
   */
  @Test
  void existsIndex() throws IOException {
    GetIndexRequest indexRequest = new GetIndexRequest("hotel");
    boolean exists = restHighLevelClient.indices().exists(indexRequest, RequestOptions.DEFAULT);
    System.out.println(exists?"已存在":"不存在");
  }

3、删除索引库

/**
 * 删除索引
 */
@Test
void deleteIndex() throws IOException {
  DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest("hotel");
  AcknowledgedResponse delete = restHighLevelClient.indices().delete(deleteIndexRequest, RequestOptions.DEFAULT);
  System.out.println(delete.toString());
}

4、总结

可以发现创建删除都是采用rest风格的去定义的java API，创建请求是都需要传入索引名，注意索引库是不支持修改操作的，如果有需要修改，直接删除重新创建即可

3、Rest-Document

1、新增单条文档

/**
   * 新增单条数据
   * @throws IOException
   */
  @Test
  void testAddOneDocument() throws IOException {
    Hotel hotel = iHotelService.getById(45845L); // 数据库中查询的数据
    HotelDoc hotelDoc = new HotelDoc(hotel);  // 数据实体转换

    // 1、准备Request对象
    // 索引库中对id的要求必须是字符串
    IndexRequest request = new IndexRequest("hotel").id(hotelDoc.getId().toString());
    // 2、准备Json文档
    // 将实体类转换为JSON格式的数据
    request.source(JSONUtil.toJsonStr(hotelDoc), XContentType.JSON);
    // 3、发送请求
    client.index(request, RequestOptions.DEFAULT);
  }

java api 对应DSL图

2、查询单条文档

/**
 * 查询单条数据
 */
@Test
void testQueryOneDocuemnt() throws IOException {
  GetRequest getRequest = new GetRequest("hotel", "45845");
  GetResponse documentFields = client.get(getRequest, RequestOptions.DEFAULT);
  String jsonString = documentFields.getSourceAsString();
  // 将json格式数据转换为实体
  System.out.println(JSONUtil.toBean(jsonString, HotelDoc.class));

}

3、修改文档

 /**
   * 修改文档的方式有两种
   * 方式一：全量更新，写入一样的id，就会删除旧文档，添加新文档
   * 方式二：局部更新，只更新部分字段
   */
  @Test
  void testUpdateDocumentById() throws IOException {
   /* Hotel hotel = iHotelService.getById(45845L);
    hotel.setScore(46);
    HotelDoc hotelDoc = new HotelDoc(hotel);
    UpdateRequest updateRequest = new UpdateRequest("hotel", hotel.getId().toString());
    updateRequest.doc(JSONUtil.toJsonStr(hotelDoc),XContentType.JSON);*/
    UpdateRequest updateRequest = new UpdateRequest("hotel", "45845");
    updateRequest.doc("score", "48");  // 直接按照kv的格式
    UpdateResponse update = client.update(updateRequest, RequestOptions.DEFAULT);

  }

4、删除文档

/**
   * 删除文档
   */
@Test
void testDeleteDocument() throws IOException {
    DeleteRequest deleteRequest = new DeleteRequest("hotel", "45845");
    System.out.println(client.delete(deleteRequest, RequestOptions.DEFAULT).getVersion());
}

5、批量导入文档

/**
 * 批量导入文档
 */
@SneakyThrows
@Test
void testBatchAddDoucument(){
    // 查询所有记录并转换成HotelDoc
    List<HotelDoc> hotelDocs = iHotelService.list().stream().map(hotel -> new HotelDoc(hotel)).collect(Collectors.toList());

    BulkRequest bulkRequest = new BulkRequest("hotel");
    for (HotelDoc hotelDoc : hotelDocs) {
        bulkRequest.add(new IndexRequest("hotel")
                        .id(hotelDoc.getId().toString())
                        .source(JSONUtil.toJsonStr(hotelDoc),XContentType.JSON));
    }
    client.bulk(bulkRequest,RequestOptions.DEFAULT);

}

除了批量新增，同时可以批量修改删除

4、复杂查询

1、query下的查询

1、match_all 全查询

DSL:

java rest:

@Test
  @SneakyThrows
  void testMatchAll(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source().query(QueryBuilders.matchAllQuery());
    SearchHit[] hits = client.search(searchRequest, RequestOptions.DEFAULT).getHits().getHits();  // 第二个hits才能获取到数据
    List<HotelDoc> hotelDocs = Arrays.stream(hits).map(hit -> {
        // 使用流提取其中的json转换成HotelDoc实体后添加进集合中
      return JSONUtil.toBean(hit.getSourceAsString().toString(), HotelDoc.class);
    }).collect(Collectors.toList());
    hotelDocs.forEach(hotelDoc -> System.out.println(hotelDoc.getName()));
  }

2、multi_match 多字段查询

@Test
@SneakyThrows
void testMultipMatch(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source().query(QueryBuilders.multiMatchQuery("如家","brand","name"));
    handleResponse(searchRequest);
}

3、term 精准查询

@SneakyThrows
  @Test
  void testTerm(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source().query(QueryBuilders.termQuery("city","北京"));
    handleResponse(searchRequest);
  }

4、range 范围查询

@SneakyThrows
  @Test
  void testTerm(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source()
            .query(QueryBuilders.rangeQuery("price").lte(400));
    handleResponse(searchRequest);
  }

5、bool 复合查询

• must：必须匹配每个子查询，类似“与”
• should：选择性匹配子查询，类似“或”
• must_not：必须不匹配，不参与算分，类似“非”
• filter：必须匹配，不参与算分

@Test
@SneakyThrows
void testBooleanQuery(){
  SearchRequest searchRequest = new SearchRequest("hotel");
  searchRequest.source().query(QueryBuilders.boolQuery()
          .must(QueryBuilders.termQuery("brand","如家"))
          .mustNot(QueryBuilders.rangeQuery("price").gte(500))
          .filter(QueryBuilders.geoDistanceQuery("location").point(31,121 ).distance(50, DistanceUnit.KILOMETERS))
  );
  handleResponse(searchRequest);
}

6、地理坐标查询

@Test
@SneakyThrows
void testGeoDistance(){
  SearchRequest searchRequest = new SearchRequest("hotel");
  searchRequest.source().query(QueryBuilders.geoDistanceQuery("location")
          .distance(15,DistanceUnit.KILOMETERS).point(31.21,121.5));
  SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

  List<HotelDoc> hotelDocs = Arrays.stream(searchResponse.getHits().getHits()).map(hit -> {
    return JSONUtil.toBean(hit.getSourceAsString().toString(), HotelDoc.class);
  }).collect(Collectors.toList());
  hotelDocs.forEach(hotelDoc -> {
    System.out.println(hotelDoc.getName());
  });
}

7、算法函数查询

@Test
@SneakyThrows
void testFunctionScore(){
  SearchRequest searchRequest = new SearchRequest("hotel");
  searchRequest.source().query(QueryBuilders.functionScoreQuery(
          QueryBuilders.matchQuery("all","上海"),
          new FunctionScoreQueryBuilder.FilterFunctionBuilder[]{
                  new FunctionScoreQueryBuilder.FilterFunctionBuilder(
                          QueryBuilders.rangeQuery("price").gte(700),
                          ScoreFunctionBuilders.weightFactorFunction(10)
                  )
          }
  ));
  SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);

  List<HotelDoc> hotelDocs = Arrays.stream(searchResponse.getHits().getHits()).map(hit -> {
    return JSONUtil.toBean(hit.getSourceAsString().toString(), HotelDoc.class);
  }).collect(Collectors.toList());
  hotelDocs.forEach(hotelDoc -> {
    System.out.print("name:"+hotelDoc.getName());
    System.out.println("  age:"+hotelDoc.getPrice());
  });
}

2、对查询结果的操作，与query同级

1、排序&分页

@Test
  @SneakyThrows
  void testSortPageQuery(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source()
            .query(QueryBuilders.termQuery("brand","如家"))
            .from(10).size(1)
            .sort("price", SortOrder.DESC).sort("score",SortOrder.ASC);
    handleResponse(searchRequest);

  }

2、高亮

注意：如果查询的字段不是高亮的字段，必须显示修改require_field_match的值为false

高亮的结果与查询的文档结果默认是分离的，并不在一起。因此解析高亮的代码需要额外的做处理

 @Test
@SneakyThrows
void testHighlighter(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source()
        .query(QueryBuilders.termQuery("name","如家"))
        .highlighter(new HighlightBuilder().field("name").requireFieldMatch(false)
                     .preTags("<b>").postTags("<b>")
                     // 以最后一个标签为主
                     .field("brand").preTags("<em>").postTags("</em>"));
    handleResponse(searchRequest);
}


private void handleResponse(SearchRequest searchRequest) throws IOException {
    SearchHits searchHits = client.search(searchRequest, RequestOptions.DEFAULT).getHits();
    System.out.printf("总共获取的条数为：%s", searchHits.getTotalHits().value+"\n");
    
    SearchHit[] hits = searchHits.getHits();
    List<HotelDoc> hotelDocs = Arrays.stream(hits).map(hit -> {
        String highlightName = null;
        String highlightBrand = null;
        if (!CollectionUtils.isEmpty(hit.getHighlightFields())){
            // 获取高亮字段
            highlightName = hit.getHighlightFields().get("name").getFragments()[0].string();
            highlightBrand = hit.getHighlightFields().get("brand").getFragments()[0].string();
        }
        HotelDoc hotelDoc = JSONUtil.toBean(hit.getSourceAsString().toString(), HotelDoc.class);
        if (highlightName!=null){
            // 将查询出来的高亮字段覆盖到原查询的实体中的字段
            hotelDoc.setName(highlightName);
        }
        if (highlightBrand!=null){
            hotelDoc.setBrand(highlightBrand);
        }
        return hotelDoc;})
        .collect(Collectors.toList());
    // hotelDocs.forEach(hotelDoc -> System.out.println("酒店名："+hotelDoc.getName()+" 酒店品牌："+hotelDoc.getBrand()));
    hotelDocs.forEach(System.out::println);
}

3、数据聚合

聚合常见的有三类：

• **桶（Bucket）**聚合：用来对文档做分组一定不分词
  • TermAggregation：按照文档字段值分组，例如按照品牌值分组、按照国家分组
  • Date Histogram：按照日期阶梯分组，例如一周为一组，或者一月为一组
• **度量（Metric）**聚合：用以计算一些值，比如：最大值、最小值、平均值等
  • Avg：求平均值
  • Max：求最大值
  • Min：求最小值
  • Stats：同时求max、min、avg、sum等
• **管道（pipeline）**聚合：其它聚合的结果为基础做聚合

这里只展示前面两种

1、Bucket聚合&排序&范围

品牌聚合，并且排序，限定聚合范围

@Test
  @SneakyThrows
  void testAggregationsBrand(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source()
            .query(QueryBuilders.rangeQuery("price").gte(100).lte(500))
            .aggregation(AggregationBuilders.terms("brand_agg") // 聚合字段名
            .field("brand").order(BucketOrder.key(false)).size(10)
            );
    SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
      // 注意转换的类型为Terms
    Terms brand_agg = (Terms) searchResponse.getAggregations().get("brand_agg");
    for (Terms.Bucket bucket : brand_agg.getBuckets()) {
      System.out.println(bucket.getKeyAsString());
    }
  }

2、Metric聚合 min、max、avg

前面的Bucket聚合对酒店按照品牌分组，形成了一个个桶。现在我们需要对桶内的酒店做运算，获取每个品牌的用户评分的min、max、avg等值。

这就要用到Metric聚合了，例如stat聚合：就可以获取min、max、avg等结果

  @Test
  @SneakyThrows
  void testAggregationBrand_score(){
    SearchRequest searchRequest = new SearchRequest("hotel");
    searchRequest.source().aggregation(AggregationBuilders.terms("brand_agg")
            .field("brand").size(20)
             // 在对品牌做聚合的前提下做分数的聚合
            .subAggregation(AggregationBuilders.stats("score_stats").field("score")));
    SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
    Terms terms = (Terms) searchResponse.getAggregations().get("brand_agg");
    terms.getBuckets().forEach(bucket ->{
      System.out.println("品牌："+bucket.getKeyAsString());
      ParsedStats stats = (ParsedStats) bucket.getAggregations().get("score_stats");
      System.out.printf("最大值为：%s ", stats.getMaxAsString());
      System.out.printf("最小值为：%s ", stats.getMinAsString());
      System.out.printf("平均值为：%s ", stats.getAvgAsString());
      System.out.printf("和为：%s ", stats.getSumAsString());
      System.out.printf("总数为：%s\n", stats.getCount());
    } );
  }

好啦，本文关于springboot整合es的主要api的操作就介绍到这里啦，如果你对本文的内容有疑问或者其他方面的见解，欢迎到评论区留言