1. Problem
1.1 Exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.jcraft.jzlib.InfBlocks.<init>(InfBlocks.java:114)
at com.jcraft.jzlib.Inflate.inflateInit(Inflate.java:181)
at com.jcraft.jzlib.Inflater.init(Inflater.java:128)
at com.jcraft.jzlib.Inflater.init(Inflater.java:114)
at com.jcraft.jzlib.Inflater.init(Inflater.java:98)
at io.netty.handler.codec.compression.JZlibDecoder.<init>(JZlibDecoder.java:50)
at io.netty.handler.codec.compression.ZlibCodecFactory.newZlibDecoder(ZlibCodecFactory.java:111)
at com.**.ClientHandler.messageReceived(ClientHandler.java:96)
at com.**.ClientHandler.messageReceived(ClientHandler.java:1)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)
at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
2. Analysis
2.1 Dump Analysis
2. 2 One Client Monitoring
memory leak caused by thread leak?
noteworthy:
(1) heap size getting smaller
(2) threads number getting larger
(3) some threads in event loop group are in WAITING status.
2.3 Two clients monitoring
Noteworthy: after the second client started, threads number quickly rose up, so the root cause is clear, there are race conditions which leads to dead lock and WAITING threads.
two clients overnight:
Reference:
[1] Netty版本升级血泪史之线程篇. http://www.infoq.com/cn/articles/netty-version-upgrade-history-thread-part/