ElasticSearch Scroll滚动查询（查询数据超过10000）

ElasticSearch Scroll滚动查询（查询数据超过10000）

ElasticSearch官方默认查询数据是10000条，查询第10001时会报错：Result window is too large, from + size must be less than or equal to...

以下是From/Size查询方法的官方文档：

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-from-size.html

今天我们重点介绍一下：Scroll滚动查询

以下是Scroll官方文档：

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-scroll.html

代码：

@Autowired
private RestHighLevelClient restHighLevelClient;

public void scrollDemo() {

//构造查询条件
SearchRequest searchRequest = new SearchRequest("索引库");
SearchSourceBuilder builder = new SearchSourceBuilder();
//设置查询超时时间
Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
builder.query(QueryBuilders.rangeQuery("字段").gte("开始时间").lte("结束时间"));
//设置最多一次能够取出10000笔数据，从第10001笔数据开始，将开启滚动查询 PS:滚动查询也属于这一次查询，只不过因为一次查不完，分多次查
builder.size(10000);
searchRequest.source(builder);
//将滚动放入
searchRequest.scroll(scroll);
SearchResponse searchResponse = null;
try {
searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("查询索引库失败", e.getMessage(), e);
}
SearchHits hits = searchResponse.getHits();
//记录要滚动的ID
String scrollId = searchResponse.getScrollId();

//TODO 对结果集的处理

//滚动查询部分，将从第10001笔数据开始取
SearchHit[] hitsScroll = hits.getHits();
while (hitsScroll != null && hitsScroll.length > 0 ) {
//构造滚动查询条件
SearchScrollRequest searchScrollRequest = new SearchScrollRequest(scrollId);
searchScrollRequest.scroll(scroll);
try {
//响应必须是上面的响应对象，需要对上一层进行覆盖
searchResponse = restHighLevelClient.scroll(searchScrollRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("滚动查询失败",e.getMessage(),e);
}
scrollId = searchResponse.getScrollId();
hits = searchResponse.getHits();
hitsScroll = hits.getHits();

//TODO 同上面完全一致的结果集处理
}

//清除滚动，否则影响下次查询
ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
ClearScrollResponse clearScrollResponse = null;
try {
clearScrollResponse = restHighLevelClient.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("滚动查询清除失败",e.getMessage(),e);
}
//清除滚动是否成功
boolean succeeded = clearScrollResponse.isSucceeded();
}

当然，除此之外还有一种查询方法：Search After（它与Scroll非常相似，但是其参数是无状态的，始终针对最新版本的搜索器进行解析。如此在后序遍历的过程中根据索引更新删除而改变；导致查询混乱）

详情可见官方文档：

https://www.elastic.co/guide/en/elasticsearch/reference/6.4/search-request-search-after.html