【ES】--Elasticsearch的深度分页/内存超限等问题

一、ES存在哪些问题

对于ES搜索，往往会有一系列的问题。
(1)、默认ES翻页到10000条之后，会报错：ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Result window is too large, from + size must be less than or equal to: [10000] but was [10001].This limit can be set by changing the [index.max_result_window] index level setting.

(2)、查询的内容太大，超过内存限制，报错：entity content is too long [138159637] for the configured buffer limit [104857600]。如果记录数据很大，即使很小的翻页查询，sort操作也会触发上面报错。
----解决: 设置ES查询内容限制HeapBufferedResponseConsumerFactory。或者，只返回必要的字段，以便查询的内存不是很大。

(3)、如何提高ES搜索的内容？ES搜索是有分词功能，为了返回高精度的内容，要设置最低分数。

二、以上问题的解决方案

public void multiTableQuery(String searchContent,String knowledgeClassificationNo,String startTe,String endTe){

        BoolQueryBuilder boolBuilder = new BoolQueryBuilder();
        boolBuilder.should(getKnowledgeBoolBuilder(searchContent,knowledgeClassificationNo,startTe,endTe)); //查询索引1的搜索条件
        boolBuilder.should(getFileBoolBuilder(searchContent,knowledgeClassificationNo,startTe,endTe));//查询索引2的搜索条件

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        String[] includeFields = new String[] {};
        // 排除内部字段(更新时间,新建时间等)
        String[] excludeFields = new String[] {
                "content_type.create_user_sid", "content_type.create_name"};
        searchSourceBuilder.fetchSource(includeFields, excludeFields);
        searchSourceBuilder.query(boolBuilder);  //设置查询条件
        boolBuilder.should().forEach(k->{
            BoolQueryBuilder te = (BoolQueryBuilder)k;
            log.info("boolBuilder:{}", JSON.toJSONString(te.must()));
        });

        /**
         * trackTotalHits为true，只会返回10000条，在往后查询，会报错。
         * 这里为了防止深度分页导致报错，就选择trackTotalHits为false
         */
        //searchSourceBuilder.trackTotalHits(true);
        //设置高亮显示
        ElasticQueryBuilder.setHighlight(searchSourceBuilder);
        //设置分页 【这里限制查询到10000】
        int page =1,pageSize = 10000;
        int from = Math.max(0, (page - 1) * pageSize);
        pageSize = Math.min(1000, pageSize);
        pageSize = Math.max(1, pageSize);
        searchSourceBuilder.from(from);
        searchSourceBuilder.size(pageSize);
        /**
         * 设置最小查询分数
         * ES有分词，为了提高返回的结果，设置最低分数下限
         */
        float minScore = 50; //这个按照50%匹配
        searchSourceBuilder.minScore(minScore);

        SearchRequest searchRequest = new SearchRequest();
        searchRequest.source(searchSourceBuilder);
        // es搜索
        try {
            //SearchResponse searchResponse = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            /**
             * 单次查询，查询太大，容易超过限制，
             * 如报错 entity content is too long [138159637] for the configured buffer limit [104857600]
             */
            RequestOptions.Builder queryBuilder = RequestOptions.DEFAULT.toBuilder();
            queryBuilder.setHttpAsyncResponseConsumerFactory(
                    new HttpAsyncResponseConsumerFactory
                            //修改为300MB
                            .HeapBufferedResponseConsumerFactory(300 * 1024 * 1024));
            SearchResponse searchResponse = restHighLevelClient.search(searchRequest, queryBuilder.build());

            SearchHits hits = searchResponse.getHits();
            System.out.println("符合条件的总文档数量：" + hits.getTotalHits().value);
            hits.forEach(p -> {
                System.out.println("文档原生信息：" + p.getSourceAsString());
                System.out.println("高亮信息：" + p.getHighlightFields());
//            //获取高亮字段
//            Map<String, HighlightField> highlightFields = p.getHighlightFields();
            });
            for (SearchHit searchHit : hits.getHits()) {
                Map<String, Object> map = searchHit.getSourceAsMap();
                System.out.println("multiQuery：" + JSON.toJSONString(map));
            }

        }catch (Exception e){

        }

    }