Elasticsearch Python查询超过10000笔数据解决方法

最新推荐文章于 2025-03-01 19:52:43 发布

dfy20020530

最新推荐文章于 2025-03-01 19:52:43 发布

阅读量1.7k

点赞数 3

分类专栏：基础文章标签： elasticsearch

本文链接：https://blog.csdn.net/dfy20020530/article/details/115073111

版权

本文介绍了在Elasticsearch中使用Python查询超过10000条数据时遇到的问题及解决方案。当尝试通过size进行分页查询时，ES限制了最多只能获取10000条数据。为解决此问题，文章提出了两种方法：1) 修改index设置，但可能影响ES性能；2) 利用search_after参数进行高效分页。通过示例代码展示了如何使用search_after实现分页查询。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Elasticsearch Python查询超过10000笔数据解决方法

起因

最近在做数据收集以及分析,目前收集的数据使用的是ES目前已经超过10W笔,当我想要将所以有数据从ES抓下来做分析的时候遇到了问题我使用form size 来做分页一开始查询第0至10000笔数据都是正常的但是当我想查询10000 至20000 笔数据就报错了查询代码如下

GET index/_search
{
"from ":10000,
"size" : 10000,
"query":{
"match_all":{}
}
}

报错如下

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [20000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "new_channel",
        "node" : "dLHMyyNfQVuY-RSE1tPguQ",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [20000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Result window is too large, from + size must be less than or equal to: [10000] but was [20000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.",
      "caused_by" : {
        "type" : "illegal_argument_exception",

最低0.47元/天解锁文章