[ES笔记]持续更新中

工作中使用es的报错问题记录及常用语法记录，仍在整理中，由于es每次版本更新api变化都会很大，所以这里的整理对于很多api都不适用，这里使用的es版本为6.8.4，使用的spring-data-elasticsearch为3.2.3.RELEASE，elasticsearch-rest-client为6.4.3

单条件查询

		//创建查询构建器
 NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
 //分页
 queryBuilder.withPageable(PageRequest.of(Integer.parseInt(param.getPageNoStr())-1, Integer.parseInt(param.getPageSizeStr())));
 if (!StringUtils.isEmpty(param.getUserName())){
     queryBuilder.withQuery(QueryBuilders.matchQuery("username",param.getUserName()));
 }
 if (!StringUtils.isEmpty(param.getUserCode())){
     queryBuilder.withQuery(QueryBuilders.matchQuery("usercode",param.getUserCode()));
 }
 if (!StringUtils.isEmpty(param.getDeptId())){
     queryBuilder.withQuery(QueryBuilders.termQuery("deptcode",param.getDeptId()));
 }
 if (!StringUtils.isEmpty(param.getUsertype())){
     queryBuilder.withQuery(QueryBuilders.termQuery("usertype",param.getUsertype()));
 }
 if (!StringUtils.isEmpty(param.getAccesscategory())){
     queryBuilder.withQuery(QueryBuilders.termQuery("access_category",param.getAccesscategory()));
 }
 if (!StringUtils.isEmpty(param.getLoginbeginDate())&&!StringUtils.isEmpty(param.getLoginendDate())){
     queryBuilder.withQuery(QueryBuilders.rangeQuery("logintime").from(param.getLoginbeginDate())
             .to(param.getLoginendDate()));
 }else if (!StringUtils.isEmpty(param.getLoginbeginDate())){
     queryBuilder.withQuery(QueryBuilders.rangeQuery("logintime").from(param.getLoginbeginDate()));
 }else if (!StringUtils.isEmpty(param.getLoginendDate())){
     queryBuilder.withQuery(QueryBuilders.rangeQuery("logintime").to(param.getLoginendDate()));
 }
 AggregatedPage<SysUserloginlogDto> result = template.queryForPage(queryBuilder.build(), SysUserloginlogDto.class);

批量存入实体类不保留空字段：

Map param = new HashMap();
param.put("startrow", pageNo*pageSize);
param.put("endrow", (pageNo+1)*pageSize);
final List<SysUserloginlog> SysUserloginloglist = sysUserLoginLogDao.listSysUserLoginLog(param);

if (!CollectionUtils.isEmpty(SysUserloginloglist)) {
    BulkRequest bulkRequest = new BulkRequest();
    for (SysUserloginlog sysUserloginlog : SysUserloginloglist) {
        // sysUserloginlog.getId() 是数据库中的主键，更改成ES的_id
        bulkRequest.add(new IndexRequest(SYS_USER_LOGIN_LOG_ES_INDEX).id(sysUserloginlog.getId().toString()).source(JSONObject.toJSONString(sysUserloginlog), XContentType.JSON).timeout(TimeValue.timeValueSeconds(500)));
    }
    try {
        final BulkResponse bulk = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        //System.out.println(countTotal.addAndGet(bulk.getItems().length));
    } catch (Exception e) {
        e.printStackTrace();
    }
}

批量存入实体类保留空字段：

Map param = new HashMap();
param.put("startrow", pageNo*pageSize);
param.put("endrow", (pageNo+1)*pageSize);
final List<SysUserloginlog> SysUserloginloglist = sysUserLoginLogDao.listSysUserLoginLog(param);

if (!CollectionUtils.isEmpty(SysUserloginloglist)) {
    BulkRequest bulkRequest = new BulkRequest();
    for (SysUserloginlog sysUserloginlog : SysUserloginloglist) {
        // sysUserloginlog.getId() 是数据库中的主键，更改成ES的_id
        SysUserloginlogDto sysUserloginlogDto = new SysUserloginlogDto();
        BeanUtils.copyProperties(sysUserloginlog, sysUserloginlogDto);
        String dataString  = JSON.toJSONString(sysUserloginlogDto, SerializerFeature.WriteMapNullValue);
        JSONObject data = JSON.parseObject(dataString);
        bulkRequest.add(new IndexRequest(SYS_USER_LOGIN_LOG_ES_INDEX)
                .type(SYS_USER_LOGIN_LOG_ES_TYPE)
                .id(sysUserloginlog.getId().toString())
                .source(data)
                .timeout(TimeValue.timeValueSeconds(500)));
    }
    try {
        final BulkResponse bulk = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        //System.out.println(countTotal.addAndGet(bulk.getItems().length));
    } catch (Exception e) {
        e.printStackTrace();
    }
}

分词查询(先对name进行分词，拿分词去分别匹配对应的结果集，加结果集叠加返回)：

QueryBuilders.matchQuery(String name,String Value);

精确查询：

QueryBuilders.termQuery(String name, String value);

模糊查询：

QueryBuilders.wildcardQuery(String name, String query);

范围查询：

QueryBuilders.rangeQuery(String name).from(开始).to(结束);

聚合查询：

//根据usercode进行分组统计，统计出的列别名叫sum
//TermsAggregationBuilder termsBuilder = AggregationBuilders.terms("sum").field("usercode");
/*String index="sysuserloginlog";
String type="sysuserloginlog";
SearchResponse searchResponse = client.prepareSearch(index)
        .setTypes(type)
        .setScroll(TimeValue.timeValueSeconds(8))
        .setQuery(QueryBuilders.matchAllQuery())
        //.addAggregation(termsBuilder)
        .execute().actionGet();*/
//得到这个分组的数据集合
//Terms terms = searchResponse.getAggregations().get("sum");
//List<SysUserloginlogDto> lists = new ArrayList<>();
/*for(int i=0;i<terms.getBuckets().size();i++){
    //statistics
    String id =terms.getBuckets().get(i).getKey().toString();//id
    Long sum =terms.getBuckets().get(i).getDocCount();//数量
    System.out.println("=="+terms.getBuckets().get(i).getDocCount()+"------"+terms.getBuckets().get(i).getKey());
}*/

/*List<SysUserloginlog> list = new ArrayList<>();
SearchHit[] result = searchResponse.getHits().getHits();
for (SearchHit hit: result) {
    SysUserloginlog sysUserloginlog = JSON.parseObject(JSON.toJSONString(hit.getSourceAsMap()), SysUserloginlog.class);
    list.add(sysUserloginlog);
}*/

根据id查询，没有routing的情况：

GetRequest request = new GetRequest("com_performancenotice_stk_basicinfo_com_industry_cnt_industry", "performancenotice");
ActionFuture<GetResponse> getResponseActionFuture = client.get(request);
GetResponse getResponse = getResponseActionFuture.actionGet();
Map<String, Object> sourceAsMap = getResponse.getSourceAsMap();   //将获取到的文档转成Map

根据id查询，有routing的情况，注意，有routing必须指定routing，否则获取不到：

GetRequest request = new GetRequest("com_performancenotice_stk_basicinfo_com_industry_cnt_industry", "performancenotice", "101638200468063").routing("2293951126");
ActionFuture<GetResponse> getResponseActionFuture = client.get(request);
GetResponse getResponse = getResponseActionFuture.actionGet();
Map<String, Object> sourceAsMap = getResponse.getSourceAsMap();

关于ElasticSearch的聚类时出现fielddata=true问题

• 完整报错：

Set fielddata=true on [performancenotice.PUBDATE] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead

• 代码：

.addSort("performancenotice.PUBDATE", SortOrder.DESC)//排序

• 原因：

在ElasticSearch中默认fielddata默认是false的，因为开启Text的fielddata后对内存的占用很高

• 解决方案：关于ElasticSearch的聚类时出现fielddata=true问题

QueryShardException:

• 完整报错：

QueryShardException[Failed to parse query ['query': {'has_parent' : { 'parent_type' : 'performancenotice', 'query' : { 'match' : { 'stk_basicinfo.SECUABBR' : 'ST' } } } }]]; nested: ParseException[Cannot parse ''query': {'has_parent' : { 'parent_type' : 'performancenotice', 'query' : { 'match' : { 'stk_basicinfo.SECUABBR' : 'ST' } } } }': Encountered " <RANGE_GOOP> ": "" at line 1, column 23

• 代码：

QueryStringQueryBuilder queryBuilder = QueryBuilders.queryStringQuery("'query': {'has_parent' : { 'parent_type' : " +
                "'performancenotice', 'query' : { 'match' : { 'stk_basicinfo.SECUABBR' : 'ST' } } } }");

• 原因：

查询字符串中包含有转义字符无法解析导致报错

• 解决方案：ES搜索特殊字符异常