在生产环境批量同步数据的时候,我写了一个查询请求,然后直接报错:entity content is too long [142501157] for the configured buffer limit [104857600]
具体内容如下:
Caused by: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600]
at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76)
at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137)
at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.lang.Thread.run(Thread.java:748)
Suppressed: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600]
... 16 more
Suppressed: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600]
... 16 more
在HttpAsyncResponseConsumerFactory
类中:
在HeapBufferedAsyncResponseConsumer
类中有一个判断:返回的内容长度大于限定的100MB就会抛出ContentTooLongException
异常。
我这里用的依赖是:
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>6.6.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>6.6.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>6.6.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>6.6.2</version>
</dependency>
而我们对es进行查询操作,或者其他增删改操作是,都会默认使用如下的RequestOptions.DEFAULT
进入RequestOptions.DEFAULT
看它的代码:这里用了HeapBufferedResponseConsumerFactory.DEFAULT
而这个HeapBufferedResponseConsumerFactory.DEFAULT
,就是我们之前看到的,已经被限制了100MB返回值的HeapBufferedResponseConsumerFactory。
所以有两种方式可以解决这个bug。
方式一:
不使用默认的RequestOptions.DEFAULT
,而通过使用自定义RequestOptions
的方式(ES官方api已经给我们开放出来了):
RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
builder.setHttpAsyncResponseConsumerFactory(
new HttpAsyncResponseConsumerFactory
//修改为500MB
.HeapBufferedResponseConsumerFactory(500 * 1024 * 1024));
RequestOptions requestOptions=
builder.build();
方式二:
如果不方便修改怎么办,例如我们使用的是Spring Data ElasticSearch或者是其他的一些框架,而它默认使用的也是RequestOptions.DEFAULT
这时候我们可以通过java反射进行修改:
//设置es查询buffer大小
RequestOptions requestOptions = RequestOptions.DEFAULT;
Class<? extends RequestOptions> reqClass = requestOptions.getClass();
Field reqField = reqClass.getDeclaredField("httpAsyncResponseConsumerFactory");
reqField.setAccessible(true);
//去除final
Field modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(reqField, reqField.getModifiers() & ~Modifier.FINAL);
//设置默认的工厂
reqField.set(requestOptions, new HttpAsyncResponseConsumerFactory() {
@Override
public HttpAsyncResponseConsumer<HttpResponse> createHttpAsyncResponseConsumer() {
//500MB
return new HeapBufferedAsyncResponseConsumer(5 * 100 * 1024 * 1024);
}
});
如果你使用的是springmvc,那么可以把这段代码加入到拦截器中,这样可以拦截需要自定义返回值大小的es请求