Netty Memory Leak Error

1. Problem

1.1 Exception

java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.jcraft.jzlib.InfBlocks.<init>(InfBlocks.java:114)
        at com.jcraft.jzlib.Inflate.inflateInit(Inflate.java:181)
        at com.jcraft.jzlib.Inflater.init(Inflater.java:128)
        at com.jcraft.jzlib.Inflater.init(Inflater.java:114)
        at com.jcraft.jzlib.Inflater.init(Inflater.java:98)
        at io.netty.handler.codec.compression.JZlibDecoder.<init>(JZlibDecoder.java:50)
        at io.netty.handler.codec.compression.ZlibCodecFactory.newZlibDecoder(ZlibCodecFactory.java:111)
        at com.**.ClientHandler.messageReceived(ClientHandler.java:96)
        at com.**.ClientHandler.messageReceived(ClientHandler.java:1)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)


2. Analysis

2.1 Dump Analysis

pool chunk

2. 2 One Client Monitoring

memory leak caused by thread leak? 

noteworthy:

(1) heap size getting smaller

(2) threads number getting larger

(3) some threads in event loop group are in WAITING status.



monitoring-threads-waiting


2.3 Two clients monitoring

Noteworthy: after the second client started, threads number quickly rose up, so the root cause is clear, there are race conditions which leads to dead lock and WAITING threads.

two clients

two clients overnight:

two clients overnight 

Reference:

[1] Netty版本升级血泪史之线程篇. http://www.infoq.com/cn/articles/netty-version-upgrade-history-thread-part/

阅读更多
个人分类: Networking J2EE
上一篇Netty Thread Leak?
下一篇Always Full GC before Dump
想对作者说点什么? 我来说一句

netty codec部分剖析

netty codec

xyls12345 xyls12345

2014-06-18 18:30:16

阅读数:2221

没有更多推荐了,返回首页

关闭
关闭