雪崩复盘

1. 表现: Thread pool is EXHAUSTED!

   com.alibaba.dubbo.rpc.RpcException: Failed to invoke remote method: getSmoothMoveLocations, provider: dubbo:///&&pid=35910&revision=1.0.20&side=consumer&timeout=1500&timestamp=1490338639044&version=1.0.0,

cause: com.alibaba.dubbo.remoting.ExecutionException:

class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process received event .
com.alibaba.dubbo.remoting.ExecutionException:
class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process received event .
    at com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler.received(AllChannelHandler.java:58)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeartbeatHandler.received(HeartbeatHandler.java:90)
    at com.alibaba.dubbo.remoting.transport.MultiMessageHandler.received(MultiMessageHandler.java:25)
    at com.alibaba.dubbo.remoting.transport.AbstractPeer.received(AbstractPeer.java:123)
    at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.messageReceived(NettyHandler.java:91)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
    at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:148)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
    at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.RejectedExecutionException: Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-  Pool Size: 300 (active: 266, core: 300, max: 300, largest: 300), Task: 3285448 (completed: 3285184), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://!
    at com.alibaba.dubbo.common.threadpool.support.AbortPolicyWithReport.rejectedExecution(AbortPolicyWithReport.java:53)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
    at com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler.received(AllChannelHandler.java:56)

2. 深层次分析,发现有大量的线程 wait 在 future.get()处.
     1.分析方法,进一步分析原因,发现 task 执行时间很快[前提是要有统计信息,不一定有],但是总的 future 很慢.[前提是要有统计信息,不一定有]

     2. 查看对应的代码和 task 底层调用的外部耗时,并没有太大的变化. 但是看到队列用了无限队列.[最好要有线程满的日志,不一定有]