Elasticsearch的scroll用法

在某些场景中为了能取得所有的数据，只能使用 scroll 的方式代替

Java代码实现scoll

JAVA scrolls API链接 : Using scrolls in Java.

     public List<String> scroll(long lastTime,long nowTime,List<String> list){
      	//设定滚动时间间隔,60秒,不是处理查询结果的所有文档的所需时间
        //游标查询的过期时间会在每次做查询的时候刷新，所以这个时间只需要足够处理当前批的结果就可以了
        final Scroll scroll = new Scroll(TimeValue.timeValueSeconds(60));
        SearchRequest searchRequest = new SearchRequest(INDEX);
        searchRequest.scroll(scroll);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        QueryBuilder queryBuilder = QueryBuilders
                .boolQuery()
                //查询条件在 两个日期范围
                .must(QueryBuilders.rangeQuery("intercept_time").gte(lastTime).lte(nowTime));
        //每个批次实际返回的数量
        searchSourceBuilder..query(queryBuilder );
        searchSourceBuilder.size(10000);
        searchRequest.source(searchSourceBuilder);
        SearchResponse searchResponse = null;
        try {
            searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            logger.error("获取数据错误1 ->", e);
        }
        assert searchResponse != null;
        String scrollId;
        do {
            for (SearchHit hit : searchResponse.getHits().getHits()) {
                Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                //获取需要数据
                list.add(String.valueOf(sourceAsMap.get("xxx"));
            }
            //每次循环完后取得scrollId,用于记录下次将从这个游标开始取数
            scrollId = searchResponse.getScrollId();
            SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
            scrollRequest.scroll(scroll);
            try {
            	//进行下次查询
                searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
            } catch (IOException e) {
                logger.error("获取数据错误2 ->", e);
            }
        } while (searchResponse.getHits().getHits().length != 0);
        //清除滚屏
        ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
        //也可以选择setScrollIds()将多个scrollId一起使用
        clearScrollRequest.addScrollId(scrollId);
        ClearScrollResponse clearScrollResponse = null;
        try {
            clearScrollResponse = client.clearScroll(clearScrollRequest,RequestOptions.DEFAULT);
        } catch (IOException e) {
            logger.warn("清除滚屏错误 ->", e);
        }
        boolean succeeded = false;
        if (clearScrollResponse!=null){
            succeeded = clearScrollResponse.isSucceeded();
        }

所有数据获取完毕之后，需要手动清理掉 scroll_id 。虽然es 会有自动清理机制，但是 srcoll_id 的存在会耗费大量的资源来保存一份当前查询结果集映像，并且会占用文件描述符。所以用完之后要及时清理。

kibana中scoll用法

Search APIs » Request Body Search »: Scroll .

第一次查询, scroll 该 参数告诉Elasticsearch它应该保持“搜索上下文”存活多长时间

	POST /ip_lib/doc/_search?scroll=1m
	{
	  "size": 10000 
	}

返回结果

	{
  "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoCgAAAAABUiHVFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIinRZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSIdgWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiHWFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIh2RZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSIqAWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiHXFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIinhZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSIp8WV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiKhFldyTUFnUHhHUkRDb1RnNDA0b1hvNWc=",
  "took": 121,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 62357,
    "max_score": 1,
    "hits": [
      {
        ...
      },
      ...
      {
        ...
      }
    ]
  }

把上面检索出得scroll_id 传入下面查询条件 执行搜索

POST /_search/scroll
{
  "scroll": "1m",
  "scroll_id": "DnF1ZXJ5VGhlbkZldGNoCgAAAAABUiHVFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIinRZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSIdgWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiHWFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIh2RZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSIqAWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiHXFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIinhZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSIp8WV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiKhFldyTUFnUHhHUkRDb1RnNDA0b1hvNWc="
}

这里需要注意:
GET或者POST可以使用，URL不应包含index 或type名称 - 这在原始search请求中指定。现在通过scroll_id不要再传
返回结果还是如上

如果再次检索时间超过上次检索设置的scroll参数的时间则返回为数据,如下

{
  "error": {
    "root_cause": [
      {
        "type": "search_context_missing_exception",
        "reason": "No search context found for id [22159830]"
      },
      ...
      {
        "type": "search_context_missing_exception",
        "reason": "No search context found for id [22160029]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": -1,
        "index": null,
        "reason": {
          "type": "search_context_missing_exception",
          "reason": "No search context found for id [22159830]"
        }
      },
     ...     
      {
        "shard": -1,
        "index": null,
        "reason": {
          "type": "search_context_missing_exception",
          "reason": "No search context found for id [22160029]"
        }
      }
    ],
    "caused_by": {
      "type": "search_context_missing_exception",
      "reason": "No search context found for id [22160029]"
    }
  },
  "status": 404
}

6万多条数据,当我按正常时间内反复执行完第二段检索语句,到第七次时因为数据已经全部查询完毕返回空列表

{
  "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoCgAAAAABUiNPFldyTUFnUHhHUkRDb1RnNDA0b1hvNWcAAAAAAVIihhZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSI04WV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiNNFldyTUFnUHhHUkRDb1RnNDA0b1hvNWcAAAAAAVIjURZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSIocWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiNQFldyTUFnUHhHUkRDb1RnNDA0b1hvNWcAAAAAAVIigxZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSIoQWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiKFFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHc=",
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 20,
    "successful": 20,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 62357,
    "max_score": 1,
    "hits": []
  }
}

scroll超过超时时，将自动删除搜索上下文。但是，保持滚动打开会产生成本
单个删除

DELETE / _search / scroll
{
     "scroll_id "： "DnF1ZXJ5VGhlbkZldGNoCgAAAAABUiP8FktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIkwxZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSJAAWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiP9FktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIj_hZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSJMYWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiP_FktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIkxRZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSJMQWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiTCFldyTUFnUHhHUkRDb1RnNDA0b1hvNWc="
}

多个滚动ID可以作为数组传递：

DELETE / _search / scroll
{
    "scroll_id": ["DnF1ZXJ5VGhlbkZldGNoCgAAAAABUiP8FktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIkwxZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSJAAWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiP9FktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIj_hZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSJMYWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiP_FktHNXlHQkNSUWlXcDBVZ1p3dmVjdHcAAAAAAVIkxRZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSJMQWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiTCFldyTUFnUHhHUkRDb1RnNDA0b1hvNWc=","DnF1ZXJ5VGhlbkZldGNoCgAAAAABUiTHFldyTUFnUHhHUkRDb1RnNDA0b1hvNWcAAAAAAVIkBBZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSJMsWV3JNQWdQeEdSRENvVGc0MDRvWG81ZwAAAAABUiTKFldyTUFnUHhHUkRDb1RnNDA0b1hvNWcAAAAAAVIkyBZXck1BZ1B4R1JEQ29UZzQwNG9YbzVnAAAAAAFSJAEWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiTJFldyTUFnUHhHUkRDb1RnNDA0b1hvNWcAAAAAAVIkBRZLRzV5R0JDUlFpV3AwVWdad3ZlY3R3AAAAAAFSJAIWS0c1eUdCQ1JRaVdwMFVnWnd2ZWN0dwAAAAABUiQDFktHNXlHQkNSUWlXcDBVZ1p3dmVjdHc="]
}

可以使用以下_all参数清除所有搜索上下文：

DELETE / _search / scroll / _all