python elasticsearch_dsl search_after翻页

会飞的尼古拉斯

已于 2022-09-24 20:22:58 修改

阅读量728

点赞数

分类专栏： elasticsearch Python 文章标签： elasticsearch 搜索引擎大数据

于 2022-08-18 14:08:34 首次发布

本文链接：https://blog.csdn.net/weixin_43632687/article/details/126404629

版权

Python 同时被 2 个专栏收录

127 篇文章 6 订阅

订阅专栏

elasticsearch

38 篇文章 1 订阅

订阅专栏

想使用search_after 这个方法，搜索的数据必须是有唯一的键，用来排序，这里使用_id。

如下每次查询带上sort参数，然后将上次查询的最后一个数据的sort信息，添加到下次查询的search_after参数里

GET books/_search
{
  
    "size": 10,
    "query": {
        "match_all" : {
           
        }
    },
    "sort": [
        {"_id": "asc"},
        {"_score": "desc"},
        {"title.keyword":"desc"}
    ]
}



GET books/_search
{
  
    "size": 10,
    "query": {
        "match_all" : {
           
        }
    },
    "search_after": [ "-55upIIB7UkVIfnN6-JF",
          1.0,
          "Learn Git in a Month of Lunches"],
    "sort": [
        {"_id": "asc"},
        {"_score": "desc"},
        {"title.keyword":"desc"}
    ]
}

elasticsearch_dsl 包

import time
from elasticsearch_dsl import connections
from elasticsearch_dsl import Search,Q
conn = connections.create_connection(hosts=['192.168.214.131'],port=9200, http_auth="elastic:ellischen")
search = Search(using= conn,index='books')
querys={}
querys['must'] = [Q('regexp', **{'logical_address.keyword': {"value": "{}(.)*".format('2')}})]
search.query = Q({"match_all":{}})

res = search.count()
hits_source=[]
start=time.time()
search_data = search.extra(size=1000).sort('-_id').execute()
while len(search_data['hits']['hits']):
    hits_source.extend(search_data['hits']['hits'])
    search_data = search.extra(size=1000).extra(search_after=search_data['hits']['hits'][-1]['sort']).sort('-_id').execute()
print(time.time()-start)
print(len(hits_source))