线程一直等待或突然中断问题排查
问题描述:这两天经常收到“锁占用超时”的告警短信,第一次查找问题,是“批扣发送渠道任务”的锁占用超时,但未定位到原因,先暂时改了锁的状态,继续做业务。到第二天又发告警短信,这次是“批扣定时查询任务”的锁占用超时,意识到线程里某个地方肯定有问题,所以开始挤出时间排除问题。
问题查找:首先查找业务日志,发现定时任务线程业务日志未正常打印结束日志,首先想到的可能因素:1、业务量大导致线程执行时间过长;2、线程长时间等到;3、线程异常中断;
继续找业务运维导出这两天的线程日志,然后发现了端倪,部分线程日志如下:
Thread 160863: (state = BLOCKED
)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
- org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
- org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPo