ElasticSearch相关

一、梗概

1. Elasticsearch 是一个分布式的 RESTful 风格的搜索和数据分析引擎；

2. 参考链接：
   官网
   整体讲解
   ES查询实例1
   ES查询实例2

3. 个别示例：

   解除返回1w条的设置

   PUT use-portrait-2020-05/_settings
   {
   "index.max_result_window":200000
   }
   

   根据id修改

   POST /use-portrait-2020-05/portrait/s30002870/_update
   {
   "doc" : {
       "OPERATION_01_001" : 0
   	}
   }
   

   复合简单查询

   GET /use-portrait-2020-05/portrait/_search
   {
    "query": {
   	"bool": {
    	 "must": [
      	 {
        	 "range": {
         	  "PIPELINE_01_002": {
            	 "gte": "0.7",
            	 "lte": "0.8"
           	}
        	 }
      	 }
     	],
    	 "must_not": [
     	  {
        	 "range": {
          	 "PIPELINE_01_002": {
             	"lte": "0.75"
          	 }
        	 }
      	 }
   	  ]
    }
    },
   "shoule":[],
     "from": 0,
    "size": 100
   }
   

   复合聚合查询

   GET /use-portrait-2020-05/portrait/_search
   {
    "size": 0,
    "aggs": {
   	"gender_1_follower": {
    	 "filter": {
       	"range": {
        	 "PIPELINE_01_001": {
          	 "gt": "0"
        	 }
       	}
    	 },
     	"aggs": {
      	 "grade_ranges": {
        	 "histogram": {
          	 "field": "PIPELINE_01_002",
          	 "interval": 0.1
         	}
       	}
    	 }
    	}
   	}
   }
   

二、结合java灵活查询

1. 动态简单复合查询

       public SearchResponse getEsData(List<QueryConditionVo> list) {
       SearchResponse response = null;
       try {
           // 初始化查询器并制定es索引
           SearchRequest request = new SearchRequest().indices(index).types(type);
           // 初始化条件构造器
           SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
           // AggregationBuilders.count()
           BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
           // 根据数据 定义查询
           if (list != null && !list.isEmpty()) {
               list.forEach(item -> setBoolQuery(boolQuery, item));
               // 默认数据只返回10条，直接设置20w
               sourceBuilder.size(200000);
               // 不返回_source体，提高查询性能（是包含id的）
               sourceBuilder.fetchSource(false);
               sourceBuilder.query(boolQuery);
               request.source(sourceBuilder);
               long start = System.currentTimeMillis();
               response = restHighLevelClient.search(request);
               log.info("-------ES查询用时--------：" + (System.currentTimeMillis() - start) + "ms");
           }
       } catch (Exception e) {
           log.error("ES 查询数量异常：{}", e.getMessage());
       }
       return response;
   }
   
   private void setBoolQuery(BoolQueryBuilder boolQuery, QueryConditionVo item) {
       if (Constant.ES_TYPE_MUST.equals(item.getBool())) {
           boolQuery.must(getQueryBuilder(item));
       }
       if (Constant.ES_TYPE_MUST_NOT.equals(item.getBool())) {
           boolQuery.mustNot(getQueryBuilder(item));
       }
       if (Constant.ES_TYPE_SHOULD.equals(item.getBool())) {
           boolQuery.should(getQueryBuilder(item));
       }
   }
   
   private QueryBuilder getQueryBuilder(QueryConditionVo item) {
       if (Constant.QUERY_TYPE_RANGE.equals(item.getQueryType())) {
           return QueryBuilders.rangeQuery(item.getTagCode())
               .gte("".equals(item.getRangeMin()) ? null : item.getRangeMin())
               .lte("".equals(item.getRangeMax()) ? null : item.getRangeMax());
       }
       if (Constant.QUERY_TYPE_MATCH_PHRASE.equals(item.getQueryType())) {
           return QueryBuilders.matchPhraseQuery(item.getTagCode(),
               item.getKeyword() == null ? "" : item.getKeyword());
       }
       if (Constant.QUERY_TYPE_TERM.equals(item.getQueryType())) {
           return QueryBuilders.termQuery(item.getTagCode() + ".keyword",
               item.getKeyword() == null ? "" : item.getKeyword());
       }
       return null;
   }
   

2. 聚合分析（以最大值最小值为界分成20份，主要用于分布图分析）

   public Result dataAnalysis(String tagCode, Integer tagType) {
       SearchResponse response = null;
       try {
           // 初始化查询器并制定es索引
           SearchRequest request = new SearchRequest().indices(index).types(type);
           // 初始化条件构造器
           SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
           // 过滤条件（暂时写死流水线大于等于一次）
           QueryBuilder query = QueryBuilders.rangeQuery("PIPELINE_01_001").gte(1);
           AggregationBuilder aggregation = AggregationBuilders.filter("pipelinefilter",query);
           if (tagType == 2) {
               aggregation.subAggregation(AggregationBuilders.histogram("analysis")
                   .field(tagCode)
                   .interval(getInterval(tagCode) == null ? 1L : getInterval(tagCode))
                   .order(BucketOrder.key(true)));
           } else {
               aggregation.subAggregation(AggregationBuilders.terms("analysis")
                   .field(tagCode + ".keyword")
                   .size(10)
                   .order(BucketOrder.count(true)));
           }
           // 默认数据只返回10条，直接设置0
           sourceBuilder.size(0);
           sourceBuilder.aggregation(aggregation);
           // 不返回_source体，提高查询性能（是包含id的）
           request.source(sourceBuilder);
           long start = System.currentTimeMillis();
           response = restHighLevelClient.search(request);
           log.info("-------ES 聚合分析用时--------：" + (System.currentTimeMillis() - start) + "ms");
       } catch (Exception e) {
           log.error("ES 聚合分析异常：{}", e.getMessage());
       }
       return ResultUtil.success(response.getAggregations().get("pipelinefilter"));
   }
   
   private Double getInterval(String tagCode) throws IOException {
       SearchRequest maxRequest = new SearchRequest().indices(index)
           .types(type)
           .source((new SearchSourceBuilder()).size(0).aggregation(AggregationBuilders.max("analysis").field(tagCode)));
       SearchResponse maxResponse = restHighLevelClient.search(maxRequest);
       double maxCount = ((Max) maxResponse.getAggregations().get("analysis")).getValue();
       Double interval = null;
       if (maxCount > 0) {
           BigDecimal intervalTemp = new BigDecimal(maxCount).divide(new BigDecimal(20), 2, RoundingMode.HALF_UP);
           // 切割后取整（仅针对间隔大于1的）
           if (intervalTemp.compareTo(new BigDecimal(1)) > 0) {
               interval = intervalTemp.setScale(0, BigDecimal.ROUND_UP).doubleValue();
           } else {
               interval = intervalTemp.doubleValue();
           }
       }
       return interval;
   }
   

三、遇到的问题

1. 返回1w条限制（已解决，见上）
2. 描述：将terms聚合的结果直接返给前端处理，如果bucket的key为字符串时，在mvc层jackson进行json序列化处理会报类型转换错误。
   分析：terms聚合的结果中有keyAsNumber字段，是将桶key转为number类型，但是当key为字符串类型的数据时，是无法转为numebr类型的
   解决方案：
   参考：https://blog.csdn.net/qq_28851503/article/details/98314215

   对ParsedStringTerms.ParsedBucket类自定义序列化如下：
   
   public class ParsedStringTermsBucketSerializer extends StdSerializer<ParsedStringTerms.ParsedBucket> {
   public ParsedStringTermsBucketSerializer(Class<ParsedStringTerms.ParsedBucket> t) {
       super(t);
   }
   @Override
   public void serialize(ParsedStringTerms.ParsedBucket value, JsonGenerator gen, SerializerProvider provider) throws IOException {
       gen.writeStartObject();
       gen.writeObjectField("aggregations", value.getAggregations());
       gen.writeObjectField("key", value.getKey());
       gen.writeStringField("keyAsString", value.getKeyAsString());
       gen.writeNumberField("docCount", value.getDocCount());
       gen.writeEndObject();
   }
   }
   
   将自定义的序列化方式设置到jackson的mapper中：
   
   @Configuration
   public class ObjectMapperConfigure {
   @Bean
   public ObjectMapper objectMapper() {
   
       ObjectMapper objectMapper = new ObjectMapper();
       objectMapper.registerModule(simpleModule());
       return objectMapper;
   }
   
   private SimpleModule simpleModule() {
       ParsedStringTermsBucketSerializer serializer = new ParsedStringTermsBucketSerializer(ParsedStringTerms.ParsedBucket.class);
       SimpleModule module = new SimpleModule();
       objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
       module.addSerializer(serializer);
       return module;
   	}
   }
   

   说明：在原博主的基础上多增加了如下一行，还是很有必要的，作用自行百度objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);