ES查询问题entity content is too long [142501157] for the configured buffer limit [104857600]

背景问题:生产上要导出存在es的对话流水数据Excel,数据量在3万条左右,在Java去es查询获取数据时,就会报错entity content is too long [142501157] for the configured buffer limit [104857600]
原因: 数据量内容大,es回传到Java时有瓶颈.

解决:自定义返回值大小的es请求
 

import lombok.extern.slf4j.Slf4j;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.apache.http.nio.protocol.HttpAsyncResponseConsumer;
import org.elasticsearch.client.*;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

@Configuration
@Slf4j
public class ElasticSearchConfig {

    @Value("${spring.elasticsearch.rest.username}")
    private String userName;

    @Value("${spring.elasticsearch.rest.password}")
    private String password;

    @Value("${spring.elasticsearch.rest.uris}")
    private String uris;

    //全局通用设置项,单实例singleton,构建授权请求头,异步等信息
    public static final RequestOptions COMMON_OPTIONS;
    static {
        RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
//        builder.addHeader("Authorization","Bearer"+TOKEN);
//        builder.setHttpAsyncResponseConsumerFactory(
//                new HttpAsyncResponseConsumerFactory.HeapBufferedResponseConsumerFactory(30*1024*1024*1024));
        COMMON_OPTIONS = builder.build();
    }
    @Bean
    public RestHighLevelClient restHighLevelClient() {
        //设置es查询buffer大小
        RequestOptions requestOptions = RequestOptions.DEFAULT;
        Class<? extends RequestOptions> reqClass = requestOptions.getClass();
        Field reqField = null;
        try {
            reqField = reqClass.getDeclaredField("httpAsyncResponseConsumerFactory");
        reqField.setAccessible(true);
        //去除final
        Field modifiersField = Field.class.getDeclaredField("modifiers");
        modifiersField.setAccessible(true);
        modifiersField.setInt(reqField, reqField.getModifiers() & ~Modifier.FINAL);

        //设置默认的工厂
        reqField.set(requestOptions, new HttpAsyncResponseConsumerFactory() {
            @Override
            public HttpAsyncResponseConsumer<HttpResponse> createHttpAsyncResponseConsumer() {
                //1000MB
                return new HeapBufferedAsyncResponseConsumer(10 * 100 * 1024 * 1024);
            }
        });
    }catch (Exception e) {
            log.error("设置es buck出错");
            e.printStackTrace();
        }
        log.info("esClient启动---------->0301");
        RestHighLevelClient esClient = null;
        String[] hosts = this.uris.split(",");
        HttpHost[] httpHosts = new HttpHost[hosts.length];
        for(int i=0;i<hosts.length;i++) {
            String host = hosts[i].split(":")[0];
            int port = Integer.parseInt(hosts[i].split(":")[1]);
            httpHosts[i] = new HttpHost(host, port, "http");
        }
        final BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, password));
        esClient =  new RestHighLevelClient(RestClient.builder(httpHosts)
                .setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                    public HttpAsyncClientBuilder customizeHttpClient(
                            HttpAsyncClientBuilder httpClientBuilder) {
                        httpClientBuilder.disableAuthCaching();
                        return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                    }
                })
        );
        return esClient;
    }
}

这是一个Java Spring Boot应用程序中的Elasticsearch客户端配置文件，它提供了创建和初始化Elasticsearch RestHighLevelClient的方法。

该文件首先引入了一些必要的包，然后使用注释@Slf4j来启用日志记录（通过Lombok实现）。在@Configuration注释下，定义了一个名为ElasticSearchConfig的类，该类负责配置RestHighLevelClient以连接到Elasticsearch集群。

在此类中，使用@Value注释将application.properties文件中的属性值映射到Java变量。这些属性包括用户名、密码和ES实例的URI。

接着，定义了一个公共的RequestOptions实例COMMON_OPTIONS，以进行全局通用设置项、构建授权请求头、异步等信息。

在restHighLevelClient()方法中，首先对httpAsyncResponseConsumerFactory字段进行修改，使其不再为final。然后，将HeapBufferedAsyncResponseConsumer作为默认的响应consumer工厂，并设置缓冲区大小为10 * 100 * 1024 * 1024字节。随后，将所有的ES实例URI解析成主机和端口，并使用BasicCredentialsProvider设置HTTP身份验证凭据。最后，构建RestHighLevelClient并返回。

总之，这个文件定义了一个用于连接Elasticsearch的RestHighLevelClient，并设置了一些自定义参数。