Spring Cloud Gateway 内存溢出解决过程

记 Spring Cloud Gateway 内存溢出查询过程

环境配置：

org.springframework.boot ： 2.1.4.RELEASE

org.springframework.cloud ：Greenwich.SR1

事故记录：

由于网关存在 RequestBody 丢失的情况，顾采用了网上的通用解决方案，使用如下方式解决：

@Bean
public RouteLocator tpauditRoutes(RouteLocatorBuilder builder) {
   
    return builder.routes().route("gateway-post", r -> r.order(1)
    	  .method(HttpMethod.POST)
    	  .and()
    	  .readBody(String.class, requestBody -> {
   return true;}) # 重点在这
    	  .and()
    	  .path("/gateway/**")
    	  .filters(f -> {
   f.stripPrefix(1);return f;})
    	  .uri("lb://APP-API")).build();
}

测试环境，Spring Cloud Gateway 网关功能编写完成。开始进行测试环境压测。

正常采用梯度压测方式，最高用户峰值设置为400并发。经历两轮时长10分钟左右压测，没有异常情况出现。

中午吃饭时间，设置了1个小时的时间进行测试。回来的时候系统报出如下异常。

2019-08-12 15:06:07,296 1092208 [reactor-http-server-epoll-12] WARN  io.netty.channel.AbstractChannelHandlerContext.warn:146 - An exception '{}' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 503316487, max: 504889344)
 at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)
 at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594)
 at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764)
 at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740)
 at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
 at io.netty.buffer.PoolArena.allocate(PoolArena.java:214)
 at io.netty.buffer.PoolArena.allocate(PoolArena.java:146)
 at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324)
 at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
 at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
 at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137)
 at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
 at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:72)
 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:793)
 at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:382)
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:315)
 at io.

当时一脸懵逼，马上开始监控 Jvm 堆栈，减少jvm的内存空间，提升并发数以后，重启项目重新压测，项目启动参数如下：

java -jar -Xmx1024M /opt/deploy/gateway-appapi/cloud-employ-gateway-0.0.5-SNAPSHOT.jar
↓↓↓↓修改为↓↓↓↓
java -jar -Xmx512M /opt/deploy/gateway-appapi/cloud-employ-gateway-0.0.5-SNAPSHOT.jar

缩减了一半内存启动，等待问题复现。等待3分钟问题再次复现，但是同时Jvm却的进行了Full GC。

      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT   
 275456.0 100103.0  484864.0   50280.2  67672.0 64001.3 9088.0 8463.2    501   11.945   3      0.262 
 275968.0 25072.3   484864.0   47329.3  67672.0 63959.4 9088.0 8448.8    502   11.970   4      0.429 

没错，在出现问题的时候，系统出现了Full Gc，但是OU并没有达到触发的原因。结合日志中的 direct memory，想到了Jvm 中的堆外内存。使用 -XX:MaxDirectMemorySize 可以进行设置 Jvm 堆外内存大小，当 Direct ByteBuffer 分配的堆外内存到达指定大小后，即触发Full GC。该值是有上限的，默认是64M，最大为 sun.misc.VM.maxDirectMemory()。结合所有情况，表明堆外内存使用存在内存溢出的情况。

报错内容为Netty框架，新增以下配置，开启Netty错误日志打印：

-Dio.netty.leakDetection.targetRecords=40 #设置Records 上限
-Dio.netty.leakDetection.level=advanced   #设置日志级别

项目启动，没任何问题，开启压测后服务报出如下异常：

2019-08-13 14:59:01,656 18047 [reactor-http-nio-7] ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak:317 - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	org.springframework.core.io.buffer.NettyDataBuffer.release(NettyDataBuffer.java:301)
	org.springframework.core.io.buffer.DataBufferUtils.release(DataBufferUtils.java:420)
	org.springframework.core.codec.StringDecoder.decodeDataBuffer(StringDecoder.java:208)
	org.springframework.core.codec.StringDecoder.decodeDataBuffer(StringDecoder.java:59)
	org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:68)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107)
	reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103)
	reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
	reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:331)
	reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1505)
	reactor.core.publisher.MonoCollectList$MonoBufferAllSubscriber.onComplete(MonoCollectList.java:123)
	reactor.core.publisher.FluxJust$WeakScalarS