(实战篇)Java服务线上死锁日志怎么看?

前言

别嫌弃日志篇幅长哈, 只要是想让看到的同学可以完整的感受到死锁的排查过程.

一、死锁栈日志信息

Found one Java-level deadlock:
=============================
"pool-1-thread-5":
  waiting to lock monitor 0x00007efa20010788 (object 0x00000005f7800000, a com.alibaba.fastjson.util.ASMClassLoader),
  which is held by "catalina-exec-345"
"catalina-exec-345":
  waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
  which is held by "catalina-exec-311"
"catalina-exec-311":
  waiting for ownable synchronizer 0x00000005ed54a980, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "catalina-exec-57"
"catalina-exec-57":
  waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
  which is held by "catalina-exec-311"

Java stack information for the threads listed above:
===================================================
"catalina-exec-311":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000005ed54a980> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:70)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.<init>(InstMethodsInter.java:56)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:162)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhance(ClassEnhancePluginDefine.java:74)
	at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:75)
	at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:145)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:2335)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:8902)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:9306)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9269)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1300(AgentBuilder.java:9047)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9625)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9575)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9194)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:3148)
	- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:1388)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1876)
	- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1750)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:250)
	at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.<clinit>(HttpComponentsClientHttpRequestFactory.java:67)
	at com.lianjia.mls.data.processor.model.spring.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:33)
	at com.lianjia.mls.data.processor.model.spring.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:15)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
	at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
	- locked <0x00000005f8602888> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1081)
	at com.lianjia.mls.data.processor.api.business.utils.SearchSpringUtils.getBean(SearchSpringUtils.java:25)
	at com.lianjia.mls.data.processor.api.business.utils.SearchApiUtils.search(SearchApiUtils.java:43)
	at com.lianjia.mls.data.processor.api.business.utils.SearchApiUtils.mapSearchResult(SearchApiUtils.java:128)
	at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl.getBubbleData(DpMapSearchServiceImpl.java:194)
	at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl$$FastClassBySpringCGLIB$$2ccc99c4.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
	at com.alliance.house.service.search.map.impl.DpMapSearchServiceImpl$$EnhancerBySpringCGLIB$$5d130a26.getBubbleData(<generated>)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch$original$DZdTpsHH(AppMapSearchController.java:223)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch$original$DZdTpsHH$accessor$JqYP12Bi(AppMapSearchController.java)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController$auxiliary$YhLPQIyA.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:98)
	at com.alliance.house.ljweb.controller.app.search.AppMapSearchController.bubbleSearch(AppMapSearchController.java)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl