在使用ElasticSearch中经常会遇到统计、查询需求,实现类似
sql
分组计算、条件查询的语法,ES在这些方面都支持的较不错,用起来也比较方便。笔者就自己开发中用到的Java API举例如下。
分组聚合查询
实现类似select avg(executionTime) from t WHERE executionDate between 'A' AND 'B' group by executionDate ORDER BY executionDate ASC
的查询后聚合。ES分组计算的逻辑是先将executionDate
分组,然后在每个分组内在进行聚合,在API中是两个聚合aggregation
包含的逻辑关系。
-
HTTP请求
POST请求体{
“from”:0,
“size”:2,
“query”: {
“range”: {
“executionDate”: {
“gte”:“2019-01-20”,
“lte”:“2019-01-25”
}
}
},
“aggregations”:{
“executionDateGroup”:{
“terms”:{
“field”:“executionDate”
},
“aggregations”:{
“executionTimeAvg”:{
“avg”:{
“field”:“executionTime”
}
}
}
}}
}
POST返回结果
{
"took": 1510,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 52,
"max_score": 1,
"hits": [
//....
]
},
"aggregations": {
"executionDateGroup": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 2,
"buckets": [
{
"key": 1548425669000,
"key_as_string": "2019-01-25 14:14:29",
"doc_count": 6,
"executionTimeAvg": {
"value": 72.33333333333333
}
},
{
"key": 1548425670000,
"key_as_string": "2019-01-25 14:14:30",
"doc_count": 6,
"executionTimeAvg": {
"value": 67.66666666666667
}
},
//.....
]
}
}
}
JAVA代码API其实与上面HTTP请求一样封装和获取数据,整体思路变化不大。
代码示例
//1.先对executionDate分组,取名executionDateGroup,即实现group by executionDate,如果不设置size,则聚合返回默认10组数据
TermsAggregationBuilder groupTerms = AggregationBuilders.terms("executionDateGroup").field("executionDate").size(Integer.MAX_VALUE);
//设置排序 true为正序、flase为倒序,实现ORDER BY executionDate ASC
groupTerms.order(BucketOrder.key(true));
//2.聚合avg(executionTime),取名executionTimeAvg
AvgAggregationBuilder timeAvg = AggregationBuilders.avg("executionTimeAvg").field("executionTime");
//3.两个aggregation父子关系
groupTerms.subAggregation(timeAvg);
//查询条件
BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
RangeQueryBuilder rangeDateQuery = QueryBuilders.rangeQuery("executionDate").gte(DateFormatUtils.format(fromDate, "yyyy-MM-dd HH:mm:ss"))
.lte(DateFormatUtils.format(toDate, "yyyy-MM-dd HH:mm:ss"));
queryBuilder.must(rangeDateQuery);
//如果方法名不为空,则添加查询条件
if (StringUtils.isNotBlank(condition.getMethodName())) {
queryBuilder.must(new TermQueryBuilder("methodName", condition.getMethodName()));
}
SearchResponse searchRes = esClient.prepareSearch(CommonConstants.EHR_EXCUTION_AOP_LOG_INDEX)
.setTypes(CommonConstants.EHR_EXCUTION_AOP_LOG_TYPE)
.setQuery(queryBuilder)
.addAggregation(groupTerms) //设置聚合
.addSort("executionDate", SortOrder.ASC)
.get();
// 获取结果,提取结果的顺序与aggregation组合关系一致,先提取外层executionDateGroup,再提取多个分组的子层executionTimeAvg
Aggregation executionDateGroup = searchRes.getAggregations().get("executionDateGroup");
Terms timeAvgTerms = null;
if (executionDateGroup instanceof Terms) {
//外层aggregation
timeAvgTerms = (Terms) executionDateGroup;
List<? extends Terms.Bucket> buckets = timeAvgTerms.getBuckets();
for (Terms.Bucket elem : buckets) {
//子级aggregation
InternalAvg executionTimeAvg = (InternalAvg) elem.getAggregations().get("executionTimeAvg");
ExecutionInfoDto executionInfoDto = new ExecutionInfoDto();
executionInfoDto.setExecutionDate(elem.getKeyAsString());
executionInfoDto.setExecutionTime((int)executionTimeAvg.getValue());
executionInfoDtoList.add(executionInfoDto);
}
}
OR条件查询
ES条件查询中有MUST/MUSTNOT/SHOULD逻辑,其中MUST/MUSTNOT与sql
中的AND/NOT理解和用法基本一致,但SHOULD则与sql
中的OR有些不一样,在ES中如果要表示OR查询,则需要配合MUST一起使用,即MUST(SHOULD A, SHOULD B)
,表示A OR B
HTTP POST请求体
{
"query": {
"bool": {
"must": {
//or条件组装
"bool" : {
"should": [
{ "match": { "about": "music" }},
{ "match": { "about": "climb" }} ]
}
},
"must": {
"match": { "first_nale": "John" }
},
"must_not": {
"match": {"last_name": "Smith" }
}
}
}
}
JAVA代码示例
BoolQueryBuilder pinyinQuery = QueryBuilders.boolQuery();
if (!keyword.contains("-")) {
pinyinQuery.should(QueryBuilders.matchQuery("userName.pinyin", pinyin));
}
pinyinQuery.should(QueryBuilders.termQuery("userId", keyword));
BoolQueryBuilder queryEs = QueryBuilders.boolQuery()
.must(pinyinQuery)
.must(new TermQueryBuilder("officeStatus", "1"));
// 构造highlight
HighlightBuilder hiBuilder= new HighlightBuilder();
hiBuilder.preTags("<h2>").postTags("</h2>").field("userName.pinyin");
SearchResponse scrollRes =
client.prepareSearch(ES_INDEX_NAME)
.setTypes(ES_INDEX_USER_TYPE)
.setQuery(queryEs)
.highlighter(hiBuilder)
.setScroll("10s")
.setSize(1000)
.get();