Java - 死锁 Dead Lock 定位分析

Pre

JVM-11虚拟机性能监控与故障处理工具之【JDK的可视化工具-JConsole】

---

jstack

jstack主要用来查看某个Java进程内的线程堆栈信息 ，建议多杀几次线程dump ,每次都有某个事件，基本没跑了~

用法也很简单

---

Thread dump

Found one Java-level deadlock:
=============================
"T-ShortConn-4":
  waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
  which is held by "T-ShortConn-0"
"T-ShortConn-0":
  waiting to lock monitor 0x00007f875c00f188 (object 0x00000006c6f34238, a java.util.concurrent.ConcurrentHashMap),
  which is held by "localhost-startStop-1"
"localhost-startStop-1":
  waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
  which is held by "T-ShortConn-0"

Java stack information for the threads listed above:
===================================================
"T-ShortConn-4":
	at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:38)
	- waiting to lock <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
	at com.artisan.bus.tool.redis.RedisClusterHelper.listOs(RedisClusterHelper.java:328)
	at com.artisan.bus.tool.redis.RedisClusterHelper.lSize(RedisClusterHelper.java:193)
	at com.artisan.bus.service.comunication.CommunicationService.send(CommunicationService.java:1922)
	at com.artisan.bus.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$cacacc60.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.artisan.bus.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$8902c097.send(<generated>)
	at net.artisan.manager.DataProcessorManager.send(DataProcessorManager.java:850)
	at net.artisan.manager.DataProcessorManager.processShortLinkData(DataProcessorManager.java:833)
	at net.artisan.common.ShortLinkProcessorThread.run(ShortLinkProcessorThread.java:22)
	at java.lang.Thread.run(Thread.java:748)
"T-ShortConn-0":
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
	- waiting to lock <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:486)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:389)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1002)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1093)
	at com.artisan.bus.tool.utils.SpringApplicationContext.getBean(SpringApplicationContext.java:31)
	at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:40)
	- locked <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
	at com.artisan.bus.tool.redis.RedisClusterHelper.listOs(RedisClusterHelper.java:328)
	at com.artisan.bus.tool.redis.RedisClusterHelper.lSize(RedisClusterHelper.java:193)
	at com.artisan.bus.service.comunication.CommunicationService.send(CommunicationService.java:1922)
	at com.artisan.bus.service.comunication.CommunicationService$$FastClassBySpringCGLIB$$cacacc60.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.artisan.bus.service.comunication.CommunicationService$$EnhancerBySpringCGLIB$$8902c097.send(<generated>)
	at net.artisan.manager.DataProcessorManager.send(DataProcessorManager.java:850)
	at net.artisan.manager.DataProcessorManager.processShortLinkData(DataProcessorManager.java:833)
	at net.artisan.common.ShortLinkProcessorThread.run(ShortLinkProcessorThread.java:22)
	at java.lang.Thread.run(Thread.java:748)
"localhost-startStop-1":
	at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:38)
	- waiting to lock <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)
	at com.artisan.bus.tool.redis.RedisClusterHelper.del(RedisClusterHelper.java:63)
	at com.artisan.bus.dao.impl.settings.template.strategy.StrategyTemplateDao.cacheStrategyTemplate(StrategyTemplateDao.java:173)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy82.cacheStrategyTemplate(Unknown Source)
	at com.artisan.bus.service.impl.task.CheckTaskService.cacheStrategyTemplate(CheckTaskService.java:3658)
	at com.artisan.bus.service.impl.task.CheckTaskService$$FastClassBySpringCGLIB$$b794226f.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.artisan.bus.service.impl.task.CheckTaskService$$EnhancerBySpringCGLIB$$7b018424.cacheStrategyTemplate(<generated>)
	at com.artisan.bus.web.filter.ApplicationConfigue.setApplicationContext(ApplicationConfigue.java:74)
	at org.springframework.context.support.ApplicationContextAwareProcessor.invokeAwareInterfaces(ApplicationContextAwareProcessor.java:121)
	at org.springframework.context.support.ApplicationContextAwareProcessor.postProcessBeforeInitialization(ApplicationContextAwareProcessor.java:97)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:409)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1620)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	- locked <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
	- locked <0x00000006c6f655a8> (a java.lang.Object)
	at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:668)
	at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:634)
	at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:682)
	at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:553)
	at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:494)
	at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:138)
	at javax.servlet.GenericServlet.init(GenericServlet.java:158)
	at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1269)
	- locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper)
	at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
	- locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper)
	at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
	- locked <0x00000006c6d54a78> (a org.apache.catalina.core.StandardWrapper)
	at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
	- locked <0x00000006c54f5f78> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	- locked <0x00000006c54f5f78> (a org.apache.catalina.core.StandardContext)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

---

Dead Lock 分析

jstack 杀出来的线程堆栈， 划重点~~~

Found one Java-level deadlock:
=============================
"T-ShortConn-4":
  waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
  which is held by "T-ShortConn-0"
"T-ShortConn-0":
  waiting to lock monitor 0x00007f875c00f188 (object 0x00000006c6f34238, a java.util.concurrent.ConcurrentHashMap),
  which is held by "localhost-startStop-1"
"localhost-startStop-1":
  waiting to lock monitor 0x00007f87e1d95718 (object 0x00000006cc86c090, a java.lang.Class),
  which is held by "T-ShortConn-0"

Java stack information for the threads listed above:
===================================================

三个线程 T-ShortConn-4 、 T-ShortConn-0 和 localhost-startStop-1

看括号里面的 object xxxxxxx

• T-ShortConn-4 想去获取lock 【object 0x00000006cc86c090】 , 但这个lock被 T-ShortConn-0持有
• T-ShortConn-0 想去获取lock 【object 0x00000006c6f34238】 , 但这个lock被 localhost-startStop-1持有
• localhost-startStop-1 想去获取lock 【object 0x00000006cc86c090】 , 但这个lock被 T-ShortConn-0持有

====》得出结论

• T-ShortConn-0 持有的lock 【0x00000006cc86c090】 被 T-ShortConn-4 和 localhost-startStop-1 需要

• localhost-startStop-1 持有的lock 【0x00000006c6f34238】 被 T-ShortConn-0 需要

这不就产生deal lock了么？

---

分析代码

T-ShortConn-0 持有的lock 【0x00000006cc86c090】 那就去dump中查找这个关键字吧

at com.artisan.bus.tool.utils.SpringApplicationContext.getBean(SpringApplicationContext.java:31)
	at com.artisan.bus.tool.redis.RedisClusterHelper.getClient(RedisClusterHelper.java:40)
	- locked <0x00000006cc86c090> (a java.lang.Class for com.artisan.bus.tool.redis.RedisClusterHelper)

---

localhost-startStop-1 持有的lock 【0x00000006c6f34238】

DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187) 返回的是一个 ConcurrentHashMap ， lock类型就是 ConcurrentHashMap。

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	- locked <0x00000006c6f34238> (a java.util.concurrent.ConcurrentHashMap)

---

解决

解决方法： 提前初始化RedisClusterHelper这个单例