gateway 网关运行一段时间后会堆外内存溢出
gateway 网关工程依赖版本如下:
<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Hoxton.SR3</spring-cloud.version>
<spring-cloud-alibaba.version>2.2.3.RELEASE</spring-cloud-alibaba.version>
</properties>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.6.RELEASE</version>
<relativePath />
</parent>
网关分配2G内存, 当进程运行到2.9G时候开始所有请求报错
reactor.netty.ReactorNetty$InternalNettyException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2046820359, max: 2058354688)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2046820359, max: 2058354688)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:775)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:730)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
at io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:174)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:136)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:128)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:378)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:780)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
解决问题:
1、启动加入 -Dio.netty.leakDetectionLevel=advanced 来看详细报错的代码。
2、检查代码,确认业务逻辑上是否有未关闭的堆外内存。发现只有一处使用到并且已经关闭。
public abstract class BaseFilter {
/**
* 返回消息
*
* @param exchange
* @param values
* @return
*/
protected Mono<Void> responseR(ServerWebExchange exchange, R r) {
ServerHttpResponse response = exchange.getResponse();
response.setStatusCode(HttpStatus.OK);
response.getHeaders().setContentType(MediaType.APPLICATION_JSON);
DataBuffer dataBuffer = response.bufferFactory().allocateBuffer()
.write(JSON.toJSONString(r).getBytes(StandardCharsets.UTF_8));;
try {
return response.writeWith(Mono.just(dataBuffer));
} finally {
DataBufferUtils.release(dataBuffer);
}
}
/**
* 从Flux<DataBuffer>中获取字符串的方法
* @return 请求体
*/
protected String resolveBodyFromRequest(ServerHttpRequest serverHttpRequest) {
// 获取请求体
Flux<DataBuffer> body = serverHttpRequest.getBody();
AtomicReference<String> bodyRef = new AtomicReference<>();
body.subscribe(buffer -> {
try {
bodyRef.set(StandardCharsets.UTF_8.decode(buffer.asByteBuffer()).toString());
} finally {
DataBufferUtils.release(buffer);
}
});
return bodyRef.get();
}
}
3、怀疑是网关中redis使用的Lettuce泄露, 更换了redis连接池为Jedis还是一样不行。
4、打开Hoxton.SR3 发现依赖的gateway版本为2.2.2.RELEASE。网上有人说2.2.5以下的是有bug。直接更换Hoxton.SR11 的版本,依赖gateway为2.2.8.RELEASE。 打包上线。解决。
<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Hoxton.SR11</spring-cloud.version>
<spring-cloud-alibaba.version>2.2.3.RELEASE</spring-cloud-alibaba.version>
</properties>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.6.RELEASE</version>
<relativePath />
</parent>