1、top 命令 找到 storm 占用CPU过高的worker进程
2、top -H -p 25155,查询该进程下的线程资源
3、使用jstack 25155 > 0513.jdump ,输出 该JVM进程下的线程堆栈信息。
4、sz 命令下载 0513.jdump 到本地分析
5、 找CPU占用最大的几个线程比如 25185 ,线程ID都转16进制 15148转成6261。
0513.jdump中找到6261(线程15148)为:
"disruptor-flush-trigger" #19 daemon prio=5 os_prio=0 tid=0x00007f6ad5fa5800 nid=0x6261 in Object.wait() [0x00007f6aba192000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000d0608448> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
0513.jdump中找到6398(线程25496)为:
"disruptor-flush-task-pool" #288 daemon prio=5 os_prio=0 tid=0x00007f6a70067800 nid=0x6398 runnable [0x00007f6a319d0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:260)
at org.apache.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:496)
at org.apache.storm.utils.DisruptorQueue.access$900(DisruptorQueue.java:60)
at org.apache.storm.utils.DisruptorQueue$ThreadLocalBatcher.flush(DisruptorQueue.java:259)
at org.apache.storm.utils.DisruptorQueue$Flusher.run(DisruptorQueue.java:282)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
可以看到消耗CPU的都是Disruptor 线程。
Strom 使用Disruptor编程模型 实现生产者-消费者模型。
Disruptor Queue为消费者提供了若干种消息等待策略
- BlockingWaitStrategy:阻塞等待,CPU占用小,但是会切换线程,延迟较高
- BusySpinWaitStrategy:自旋等待,CPU占用高,但是无需切换线程,延迟低
- YieldingWaitStrategy:先自旋等待,然后使用Thread.yield()唤醒其他线程,CPU占用和延迟比较均衡
- SleepingWaitStrategy:先自旋,然后Thread.yield(),最后调用LockSupport.parkNanos(1L),CPU占用和延迟比较均衡
Storm的默认等待策略为BlockingWaitStrategy。BlockingWaitStrategy内部使用信号量来阻塞Consumer,当await超时后,Consumer线程会被自动唤醒,继续循环查询可用消息。这里的实现有个BUG,在processorNotifyCondition.await超时时应该循环查询,但是代码中实际上跳出了循环,直接返回的当前的cursor。Storm设置等待超时为10ms。推测在没有消息或者消息量较少时,Executor在消费队列时会被阻塞,由于超时时间很短,工作线程会频繁超时,再加上BlockingWaitStrategy的BUG,consumeBatchWhenAvailable会被频繁调用,导致CPU占用飙高。
参考文献:
https://www.cnblogs.com/catkins/p/5353528.html
https://www.cnblogs.com/haiq/p/4112689.html
http://ifeve.com/the-disruptor-lock-free-publishing/