ok,言归正传。上图:
Java的线程状态一共有NEW,RUNNABLE,BLOCKED,WAITING,TIMED_WAITING,TERMINATED 6种状态。这里重点关注一下BLOCKED和TIMED_WAITING状态。
BLOCKED状态:线程进入此状态的前提一般有两个:waiting for monitor(intrinsic or external) entry 或者 reenter 同步代码块。讲到这我们先了解一下Java线程模型中的两个队列。如图所示:
每个 Monitor在某个时刻,只能被一个线程拥有,该线程就是 “Active Thread”,而其它线程都是 “Waiting Thread”,分别在两个队列 “Entry Set”和 “Wait Set”里面等候。在 “Entry Set”中等待的线程状态是 “Waiting for monitor entry”,而在 “Wait Set”中等待的线程状态是 “in Object.wait()”。如果你不恰当的使用了ReentrantLock或者ReentrantReadWriteLock类,就有可能陷入BLOCKED状态,这个也是我们调优中经常会遇到的情况,解决方案也很简单,找到等待上锁的地址,分析是否发生了Thread starvation。
至于TIME_WAITING状态,官方文档也讲解的比较好,即你在调用下面方法时,线程会进入该状态。
Thread.sleep
Object.wait with timeout
Thread.join with timeout
LockSupport.parkNanos
LockSupport.parkUntil
这里重点关注一下LockSupport,该类是用来创建锁和其他同步类的基本线程阻塞原语,是一个针对Thread.suspend和Thread.resume()的优化,也是针对忙等,防止过度自旋的一种优化(关于这一点,感兴趣的同学可以参阅一下文献5)。
ok,在简单介绍完几个重点的线程状态后,我们通过几个具体的case来了解下Thread stack:
Case 1:NIO 中的Acceptor
"qtp589745448-36 Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10 tid=0x00007f02f8eea800 nid=0x18ee runnable [0x00007f02e70b3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked <0x00000000ec8ffde8> (a java.lang.Object)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
瞅瞅源代码中是怎么实现的,如下:
public void accept(int acceptorID) throws IOException
100 {
101 ServerSocketChannel server;
102 synchronized(this)
103 {
104 server = _acceptChannel;
105 }
106
107 if (server!=null && server.isOpen() && _manager.isStarted())
108 {
109 SocketChannel channel = server.accept();
110 channel.configureBlocking(false);
111 Socket socket = channel.socket();
112 configure(socket);
113 _manager.register(channel);
114 }
115 }
关于Thread stack,这里强调一点:nid,native lwp id,即本地轻量级进程(即线程)ID。
Case 2: NIO中的Selector
"qtp589745448-35 Selector0" prio=10 tid=0x00007f02f8ee9800 nid=0x18ed runnable [0x00007f02e71b4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ec9006f0> (a sun.nio.ch.Util$2)
- locked <0x00000000ec9006e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ec9004c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
代码片段如下:
// If we should wait with a select
566 if (wait>0)
567 {
568 long before=now;
569 selector.select(wait);
570 now = System.currentTimeMillis();
571 _timeout.setNow(now);
572
573 // If we are monitoring for busy selector
574 // and this select did not wait more than 1ms
575 if (__MONITOR_PERIOD>0 && now-before <=1)
576 {
577 // count this as a busy select and if there have been too many this monitor cycle
578 if (++_busySelects>__MAX_SELECTS)
579 {
580 // Start injecting pauses
581 _pausing=true;
582
583 // if this is the first pause
584 if (!_paused)
585 {
586 // Log and dump some status
587 _paused=true;
588 LOG.warn("Selector {} is too busy, pausing!",this);
589 }
590 }
591 }
592 }
Case 3: ActveMQ中针对MQTT协议的Handler
"ActiveMQ Transport Server Thread Handler: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600" daemon prio=10 tid=0x00007f02f8ba6000 nid=0x18dc waiting on condition [0x00007f02ec824000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000faad0458> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:373)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
代码片段:
@Override
protected void doStart() throws Exception {
if (useQueueForAccept) {
Runnable run = new Runnable() {
@Override
public void run() {
try {
while (!isStopped() && !isStopping()) {
Socket sock = socketQueue.poll(1, TimeUnit.SECONDS);
if (sock != null) {
handleSocket(sock);
}
}
} catch (InterruptedException e) {
LOG.info("socketQueue interuppted - stopping");
if (!isStopping()) {
onAcceptError(e);
}
}
}
};
socketHandlerThread = new Thread(null, run, "ActiveMQ Transport Server Thread Handler: " + toString(), getStackSize());
socketHandlerThread.setDaemon(true);
socketHandlerThread.setPriority(ThreadPriorities.BROKER_MANAGEMENT - 1);
socketHandlerThread.start();
}
super.doStart();
}
Case 5: 模拟银行转帐存款
"withdraw" prio=10 tid=0x00007f3428110800 nid=0x2b6b waiting for monitor entry [0x00007f34155bb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.von.thread.research.DeadThread.depositMoney(DeadThread.java:13)
- waiting to lock <0x00000000d7fae540> (a java.lang.Object)
- locked <0x00000000d7fae530> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:28)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
"deposit" prio=10 tid=0x00007f342810f000 nid=0x2b6a waiting for monitor entry [0x00007f34156bc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.von.thread.research.DeadThread.withdrawMoney(DeadThread.java:21)
- waiting to lock <0x00000000d7fae530> (a java.lang.Object)
- locked <0x00000000d7fae540> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:29)
at java.lang.Thread.run(Thread.java:724)
Locked ownable synchronizers:
- None
Found one Java-level deadlock:
=============================
"withdraw":
waiting to lock monitor 0x00007f3400003620 (object 0x00000000d7fae540, a java.lang.Object),
which is held by "deposit"
"deposit":
waiting to lock monitor 0x00007f3400004b20 (object 0x00000000d7fae530, a java.lang.Object),
which is held by "withdraw"
Java stack information for the threads listed above:
===================================================
"withdraw":
at com.von.thread.research.DeadThread.depositMoney(DeadThread.java:13)
- waiting to lock <0x00000000d7fae540> (a java.lang.Object)
- locked <0x00000000d7fae530> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:28)
at java.lang.Thread.run(Thread.java:724)
"deposit":
at com.von.thread.research.DeadThread.withdrawMoney(DeadThread.java:21)
- waiting to lock <0x00000000d7fae530> (a java.lang.Object)
- locked <0x00000000d7fae540> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:29)
at java.lang.Thread.run(Thread.java:724)
Found 1 deadlock.
这里是一个非顺序加锁诱发的一个死锁场景。
好了,差不多了,总结一下在调优过程中需要重点关注的三类情况(grep java.lang.Thread.State dump.bin | awk '{print $2$3$4$5}' | sort | uniq -c):
1. waiting for monitor entry – thread state blocked。可能发生的问题: deadlock(sequential deadlock,starvation deadlock...)
2. waiting on condition – sleeping or timed_waiting。可能发生的问题: IO bottleneck
3. Object.wait – TIMED_WAITING。wait & notifyAll使用上需要明确其性能及其局限性问题,JCIP上也推荐尽可能使用JUC提供的高级并发原语AQS
参考文献:
-
http://architects.dzone.com/articles/how-analyze-java-thread-dumps
-
http://stackoverflow.com/questions/7698861/simple-java-example-runs-with-14-threads-why
-
http://www.longene.org/forum/viewtopic.php?f=5&t=94&p=399#p399
-
http://www.slideshare.net/Byungwook/analysis-bottleneck-in-j2ee-application
-
http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
-
JavaConcurrency in practice
-
http://stackoverflow.com/questions/37026/java-notify-vs-notifyall-all-over-again