Java实现es的scroll滚动查询

    public void selectData() throws IOException {
        int i=1,size=2;
        SearchRequest searchRequest = new SearchRequest("index");
        Scroll scroll = new Scroll(TimeValue.timeValueMinutes(5L));
        searchRequest.scroll(scroll);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        MatchAllQueryBuilder matchAllQueryBuilder = QueryBuilders.matchAllQuery();
        searchSourceBuilder.query(matchAllQueryBuilder);
        searchSourceBuilder.size(size);

        searchRequest.source(searchSourceBuilder);
        SearchResponse response = client.search(searchRequest, RequestOptions.DEFAULT);
        String scrollId = response.getScrollId();
        SearchHit[] searchHits = response.getHits().getHits();

        System.out.println(response.getHits().getTotalHits());
        for (SearchHit searchHit : searchHits) {
            System.out.println(searchHit.getSourceAsString());
        }
        while (searchHits != null && searchHits.length > 0) {
            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
            scrollRequest.scroll(scroll);
            response = client.scroll(scrollRequest, RequestOptions.DEFAULT);
            scrollId = response.getScrollId();
            searchHits = response.getHits().getHits();

            for (SearchHit searchHit : searchHits) {
                i++;
                System.out.println(searchHit.getSourceAsString());
            }
            if (i > 10) {
                break;
            }
        }
    }

scroll滚动查询,es默认是存储500条scroll_id,如果超过500条继续使用滚动查询时,会报错,下面代码是对screll_id进行删除操作。

public static boolean clearScrollIds(RestHighLevelClient client,String... scrollIds){
    List<String> sIds = new ArrayList<>();
    for (String scrollId : scrollIds) {
        sIds.add(scrollId);
    }
    ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
    //添加单个id
    clearScrollRequest.addScrollId("滚动id");
    //添加多个id
    clearScrollRequest.setScrollIds(sIds);
    try {
        client.clearScroll(clearScrollRequest,RequestOptions.DEFAULT);
        return true;
    } catch (IOException e) {
        return false;
    }

}

 

  • 6
    点赞
  • 11
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
使用 ElasticsearchScroll API 可以实现高效分页查询大量数据。下面是一个基于 Java 的 demo: ```java import org.apache.http.HttpHost; import org.elasticsearch.action.search.ClearScrollRequest; import org.elasticsearch.action.search.ClearScrollResponse; import org.elasticsearch.action.search.SearchRequest; import org.elasticsearch.action.search.SearchResponse; import org.elasticsearch.action.search.SearchScrollRequest; import org.elasticsearch.client.RestClient; import org.elasticsearch.client.RestHighLevelClient; import org.elasticsearch.common.unit.TimeValue; import org.elasticsearch.index.query.QueryBuilders; import org.elasticsearch.search.SearchHit; import org.elasticsearch.search.builder.SearchSourceBuilder; import java.io.IOException; public class ElasticsearchDemo { public static void main(String[] args) throws IOException { RestHighLevelClient client = new RestHighLevelClient( RestClient.builder( new HttpHost("localhost", 9200, "http"))); SearchSourceBuilder sourceBuilder = new SearchSourceBuilder(); sourceBuilder.query(QueryBuilders.matchAllQuery()); sourceBuilder.size(10000); sourceBuilder.timeout(new TimeValue(60, TimeValue.Unit.SECONDS)); SearchRequest searchRequest = new SearchRequest(); searchRequest.indices("your_index_name"); searchRequest.scroll(new TimeValue(60000)); searchRequest.source(sourceBuilder); SearchResponse searchResponse = client.search(searchRequest); String scrollId = searchResponse.getScrollId(); SearchHit[] hits = searchResponse.getHits().getHits(); while (hits != null && hits.length > 0) { // 处理查询结果 for (SearchHit hit : hits) { // 处理查询结果 } SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId); scrollRequest.scroll(new TimeValue(60000)); searchResponse = client.scroll(scrollRequest); scrollId = searchResponse.getScrollId(); hits = searchResponse.getHits().getHits(); } ClearScrollRequest clearScrollRequest = new ClearScrollRequest(); clearScrollRequest.addScrollId(scrollId); ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest); client.close(); } } ``` 上述代码中,我们通过 ElasticsearchJava 客户端 RestHighLevelClient 发起查询请求,并设置查询条件为 `matchAllQuery()`,查询结果的数量为 10000,查询超时时间为 60 秒。然后使用 `searchRequest.scroll(new TimeValue(60000))` 设置查询结果的滚动时间为 60 秒,获取第一次查询结果 `searchResponse`。 接下来,我们使用 `searchResponse.getScrollId()` 获取查询结果的 scrollId,并使用 `searchResponse.getHits().getHits()` 获取查询结果的 hits 数组,处理查询结果。然后使用 `SearchScrollRequest` 发起下一次查询,获取查询结果 `searchResponse` 和新的 scrollId,重复以上操作,直到查询结果为空。 最后,我们使用 `ClearScrollRequest` 清除 scrollId,释放资源。 使用 Scroll API 可以高效地查询大量数据,但需要注意的是,查询结果会占用 Elasticsearch 的内存资源,需要在使用完毕后及时清除 scrollId。
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值