elasticsearch聚合查询

warrah

已于 2023-05-12 11:32:30 修改

阅读量435

点赞数

分类专栏：岁月云——大数据杂烩文章标签： elasticsearch 大数据

于 2023-02-08 19:02:12 首次发布

本文链接：https://blog.csdn.net/warrah/article/details/128818785

版权

本文探讨了Elasticsearch的聚合查询，包括聚合后再过滤超过100万的数据、正则查询、分区聚合、多字段聚合、升级Spring Boot后的调整以及针对text字段聚合的挑战。在实践中遇到的性能问题，如查询速度慢，通过调整查询策略和理解Fielddata限制找到了解决方案。同时，文章提到了text类型字段不支持聚合查询的解决方法，即重建索引。

摘要由CSDN通过智能技术生成

9 聚合后再过滤
查询汇总后多条件过滤超过100万的数据

POST zzp_invoice/_search
{
	"size": 0,
	"query":{
		"range":{
			"SSYF":{
				"gte":"202101",
				"lte":"202112"
			}
		}
	},
	"aggs": {
		"ssyf_group": {
			"terms":{
				"field":"XHDWMC.raw"
			},
			"aggs":{
				"sum_aggs":{
					"sum":{
						"field":"JSHJ"
					}
				},
				"ssyf_group_filter":{
					"bucket_selector":{
						"buckets_path":{"ssyfGroup":"sum_aggs"},
						"script":"params.ssyfGroup>=1000000"
					}
				}
			}
		}
	}	
}

8 正则查询
正则取反，还是使用must_not这个就不用使用?!等

GET zzp_invoice/_search
{
    "query":{
        "bool":{
            "must_not":[
                {
                    "regexp":{  
                        "XHDWMC":"(酒店|餐馆|餐饮|饭店)"
                    }
                }
            ]
        }
    }
}

7 分区聚合

GET zzp_invoice/_search
{
	"size":0,
    "aggs":{
        "xhdwmc_term":{
			"terms":{
				"field":"XHDWMC.raw"
			},
			"aggs":{
				"ssyf_range":{
					"range":{
						"field":"SSYF",
						"ranges":[
							{"from":202101,"to":202103},
							{"from":202101,"to":202106},
							{"from":202101,"to":202112},
							{"from":202101,"to":202212}
						],
						"aggs":{
							"sum_ssyf":{
								"sum":{
									"field":"HJJE"
								}
							}
						}
					}
				}
			}
        }
    }
}

6 multi-terms
多字段聚合查询，es7.17之后的版本支持了
Error while creating Elasticsearch API Client(8.0.1) object using Spring Boot

Caused by: java.lang.ClassNotFoundException: jakarta.json.spi.JsonProvider
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 73 common frames omitted

解决方案是使用2.0.1版本的jakarta.json-api

 <dependency>
            <groupId>jakarta.json</groupId>
            <artifactId>jakarta.json-api</artifactId>
            <version>2.0.1</version>
        </dependency>
        <dependency>
            <groupId>com.tt</groupId>
            <artifactId>bsfx-iv</artifactId>
            <version>${bsfx.version}</version>
            <exclusions>
                <exclusion>
                    <artifactId>jakarta.json-api</artifactId>
                    <groupId>jakarta.json</groupId>
                </exclusion>
            </exclusions>
        </dependency>

因为spring data elasticsearch的更新速度没有elasticsearch版本的迭代快，故而采用es自身的库，才支持multi_terms

import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.elasticsearch._types.aggregations.MultiTermsAggregate;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import co.elastic.clients.elasticsearch.core.SearchResponse;

@Autowired
    private ElasticsearchClient elasticsearchClient;

SearchRequest searchRequest = new SearchRequest.Builder()
                .size(0)
                .index("user_info")
                .aggregations("aggs",agg->agg.multiTerms(multiTerms->
                        multiTerms.terms(term->term.field("WindowsVersion"))
                            .terms(term->term.field("WindowsBits"))
                )).build();
        SearchResponse<UserInfoDto> response = elasticsearchClient.search(searchRequest,UserInfoDto.class);
        MultiTermsAggregate multiTermsAggregate = response.aggregations().get("aggs").multiTerms();
        List<WindowsDto> windowsDtos = new ArrayList<>();
        multiTermsAggregate.buckets().array().stream().forEach(bucket->{
   
            WindowsDto windowsDto = new WindowsDto();
            windowsDto.setCzxt(bucket.key().get(0).stringValue());
            windowsDto.setBits((int)(bucket.key().get(1).longValue()));
            windowsDto.setCn(windowsDto.getBits()==32?-bucket.docCount():bucket.docCount());
            windowsDtos.add(windowsDto);
        }