问题描述
生产环境告警线程池满了，线程池大小100
threadpool:tomcat-web, usage is too high, executing:100, occurs time:2020-06-01_13-58-26, and dump cost:490, pastBeforeDump cost:1
问题分析
线程是公司内部封装的，会自动dump堆栈快照。堆栈过滤筛选后如下（按照线程状态排序）：
"tomcat-web-12" Id=263 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-13" Id=264 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-14" Id=265 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-15" Id=266 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-16" Id=267 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-17" Id=268 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-18" Id=269 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-19" Id=270 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-1" Id=244 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-20" Id=271 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-21" Id=272 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-22" Id=273 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-23" Id=274 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-24" Id=275 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-27" Id=278 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-28" Id=279 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-2" Id=253 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-30" Id=281 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-31" Id=282 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-32" Id=283 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-34" Id=285 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-35" Id=286 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-37" Id=288 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-38" Id=289 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-39" Id=290 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-3" Id=254 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-40" Id=291 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-41" Id=292 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-43" Id=294 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-44" Id=295 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-45" Id=296 BLOCKED on java.lang.Object@7b8afebd owned by "tomcat-web-63" Id=318
"tomcat-web-46" Id=298 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-47" Id=297 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-48" Id=299 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-49" Id=300 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-4" Id=255 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-50" Id=301 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-51" Id=302 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-52" Id=303 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-54" Id=305 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-55" Id=309 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-56" Id=310 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-57" Id=311 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-58" Id=312 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-59" Id=314 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-5" Id=256 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-61" Id=316 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-62" Id=317 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-63" Id=318 BLOCKED on sun.misc.URLClassPath@a87f0db owned by "tomcat-web-26" Id=277
"tomcat-web-64" Id=319 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-65" Id=320 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-66" Id=321 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-67" Id=322 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-68" Id=323 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-69" Id=324 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-6" Id=257 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-70" Id=325 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-71" Id=326 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-72" Id=327 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-73" Id=328 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-74" Id=329 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-75" Id=330 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-77" Id=332 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-78" Id=333 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-79" Id=334 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-7" Id=258 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-80" Id=335 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-81" Id=336 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-82" Id=338 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-83" Id=337 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-84" Id=339 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-85" Id=340 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-86" Id=341 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-88" Id=343 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-89" Id=344 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-8" Id=259 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-90" Id=345 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-92" Id=347 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-94" Id=349 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-95" Id=350 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-97" Id=352 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-98" Id=353 BLOCKED on org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc owned by "tomcat-web-63" Id=318
"tomcat-web-9" Id=260 BLOCKED on java.lang.Object@181d205a owned by "tomcat-web-26" Id=277
"tomcat-web-0" Id=243 RUNNABLE
"tomcat-web-100" Id=355 RUNNABLE
"tomcat-web-101" Id=360 RUNNABLE
"tomcat-web-11" Id=262 RUNNABLE
"tomcat-web-25" Id=276 RUNNABLE
"tomcat-web-26" Id=277 RUNNABLE
"tomcat-web-42" Id=293 RUNNABLE
"tomcat-web-99" Id=354 RUNNABLE
"tomcat-web-10" Id=261 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50430c47
"tomcat-web-29" Id=280 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51198687
"tomcat-web-33" Id=284 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ecb324b
"tomcat-web-36" Id=287 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4e0f9f73
"tomcat-web-53" Id=304 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c5e9d6c
"tomcat-web-60" Id=315 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1febd5ff
"tomcat-web-76" Id=331 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526f9269
"tomcat-web-87" Id=342 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18c5cad7
"tomcat-web-91" Id=346 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90ecee9
"tomcat-web-93" Id=348 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@607ebea8
"tomcat-web-96" Id=351 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@556c07ba
"tomcat-web-interface-reporter-1" Id=246 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38820f46
"tomcat-web-interface-reporter-0" Id=245 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38820f46
"tomcat-web-interface-reporter-2" Id=306 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38820f46
- 大部分线程阻塞在"tomcat-web-63"持有的TomcatEmbeddedWebappClassLoader@69d8f6dc锁，数量70。java.lang.Object@7b8afebd锁，数量1
- "tomcat-web-63"与其他剩余BLOCKED线程都阻塞在"tomcat-web-26"的java.lang.Object@181d205a锁，数量11
tomcat-web-63
总共两把锁
- org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc：按照TomcatEmbeddedWebappClassLoader实例加锁
- java.lang.Object@7b8afebd：按照className加锁
"tomcat-web-63" Id=318 BLOCKED on sun.misc.URLClassPath@a87f0db owned by "tomcat-web-26" Id=277
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:479)
- blocked on sun.misc.URLClassPath@a87f0db
at sun.misc.URLClassPath.getResource(URLClassPath.java:248)
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked java.lang.Object@7b8afebd
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.loadFromParent(TomcatEmbeddedWebappClassLoader.java:93)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.doLoadClass(TomcatEmbeddedWebappClassLoader.java:66)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.loadClass(TomcatEmbeddedWebappClassLoader.java:50)
- locked org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137)
at org.apache.logging.log4j.util.LoaderUtil.loadClass(LoaderUtil.java:164)
at org.apache.logging.slf4j.Log4jLogger.createConverter(Log4jLogger.java:416)
at org.apache.logging.slf4j.Log4jLogger.<init>(Log4jLogger.java:54)
at org.apache.logging.slf4j.Log4jLoggerFactory.newLogger(Log4jLoggerFactory.java:37)
at org.apache.logging.slf4j.Log4jLoggerFactory.newLogger(Log4jLoggerFactory.java:29)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:52)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at com.....wireless.dubbo.filter.v2.InvokeResultLogFilter.invoke(InvokeResultLogFilter.java:49)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.dianwoba.wireless.dubbo.filter.v2.ConsumerTraceFilter.invoke(ConsumerTraceFilter.java:44)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:72)
at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:77)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56)
at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:78)
at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:244)
at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:75)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@10924317
第一把锁：at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader.loadClass(TomcatEmbeddedWebappClassLoader.java:50)
- locked org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedWebappClassLoader@69d8f6dc
查看源码发现内嵌的Tomcat的类加载器加载类时使用了同步锁，代码如下
@Override
public synchronized Class<?> loadClass(String name, boolean resolve)
throws ClassNotFoundException {
Class<?> result = findExistingLoadedClass(name);
result = (result != null) ? result : doLoadClass(name);
if (result == null) {
throw new ClassNotFoundException(name);
}
return resolveIfNecessary(result, resolve);
}
第二把锁：at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked java.lang.Object@7b8afebd
org.springframework.boot.loader.LaunchedURLClassLoader.loadClass，该类加载器为springboot封装的不在项目依赖中，打包完成后的BOOT-INF的org目录中，查看其源码此处是调用的父类ClassLoader.loadClass
protected Class<?> loadClass(String name, boolean resolve)
throws ClassNotFoundException
{
synchronized (getClassLoadingLock(name)) {
...
}
}
protected Object getClassLoadingLock(String className) {
Object lock = this;
if (parallelLockMap != null) {
Object newLock = new Object();
lock = parallelLockMap.putIfAbsent(className, newLock);
if (lock == null) {
lock = newLock;
}
}
return lock;
}
tomcat-web-26
两把锁
- java.lang.Object@181d205a
- sun.misc.URLClassPath@a87f0db
"tomcat-web-26" Id=277 RUNNABLE
at sun.misc.URLClassPath.getLoader(URLClassPath.java:504)
- locked sun.misc.URLClassPath@a87f0db
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:494)
- locked sun.misc.URLClassPath@a87f0db
at sun.misc.URLClassPath.getResource(URLClassPath.java:248)
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
- locked java.lang.Object@181d205a
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.alibaba.com.caucho.hessian.io.JavaSerializer.introspectWriteReplace(JavaSerializer.java:168)
at com.alibaba.com.caucho.hessian.io.JavaSerializer.<init>(JavaSerializer.java:77)
at com.alibaba.com.caucho.hessian.io.SerializerFactory.getDefaultSerializer(SerializerFactory.java:398)
at com.alibaba.com.caucho.hessian.io.SerializerFactory.getSerializer(SerializerFactory.java:369)
at com.alibaba.com.caucho.hessian.io.Hessian2Output.writeObject(Hessian2Output.java:389)
at com.alibaba.dubbo.common.serialize.hessian2.Hessian2ObjectOutput.writeObject(Hessian2ObjectOutput.java:88)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.encodeRequestData(DubboCodec.java:183)
at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encodeRequest(ExchangeCodec.java:235)
at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.encode(ExchangeCodec.java:72)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.encode(DubboCountCodec.java:38)
at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalEncoder.encode(NettyCodecAdapter.java:80)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:66)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:776)
at org.jboss.netty.channel.SimpleChannelHandler.writeRequested(SimpleChannelHandler.java:304)
at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.writeRequested(NettyHandler.java:98)
at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:266)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.jboss.netty.channel.Channels.write(Channels.java:611)
...
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@4d40ef8a
为什么不是LaunchedURLClassLoader类加载器？
为什么都是同一个类加载器实例：TomcatEmbeddedWebappClassLoader@69d8f6dc？
- springboot框架加载Servlet使用的是Thread context classloader
- 下面流程图中可以看到上下文加载Servlet前（loadOnStartup）设置的TomcatEmbeddedWebappClassLoader加载器
所以在spring容器刷新上下文完成后，tomcat启动期间，当前线程上下文的原类加载器不为空，并且restore恢复前，该期间业务请求进来走的是TomcatEmbeddedWebappClassLoader加载器
小结
对于TomcatEmbeddedWebappClassLoader@69d8f6dc锁，分析堆栈都是在公司内部封装的dubbo过滤器中InvokeResultLogFilter获取日志引起的阻塞
源代码如下：
Logger logger = LoggerFactory.getLogger(
String.format("dubbo.%s.resultlog.%s.%s", side, serviceName, methodName));
日志版本：log4j-slf4j-impl-2.11.0.jar
获取Logger过程中创建转换器
private static EventDataConverter createConverter() {
try {
LoaderUtil.loadClass("org.slf4j.ext.EventData");
return new EventDataConverter();
} catch (final ClassNotFoundException cnfe) {
return null;
}
}
加载EventData类
public static Class<?> loadClass(final String className) throws ClassNotFoundException {
if (isIgnoreTccl()) {
return Class.forName(className);
}
try {
return getThreadContextClassLoader().loadClass(className);
} catch (final Throwable ignored) {
return Class.forName(className);
}
}
当前线程上下文类加载器都是TomcatEmbeddedWebappClassLoader@69d8f6dc。所以大家都在竞争同一把锁产生了阻塞。
总结
- 阻塞原因是在自定义的dubbo过滤器（提供者端与消费者端）初始化时并发请求进来或消费调用外部接口情况。会为每一个side+接口+方法创建Logger，并发使用同一个TomcatEmbeddedWebappClassLoader实例的类加载器加载EventData类造成阻塞打满线程池
- 如果想要禁止Log4j2使用TCCL类加载器可以通过参数：log4j.ignoreTCL配置
新版本已经修复了该问题，例如：spring-boot-2.1.6.RELEASE，同步锁不再是实例，而是不同的class不同的锁，源码如下：
// org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader#loadClass
@Override
public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
synchronized (getClassLoadingLock(name)) {
Class<?> result = findExistingLoadedClass(name);
result = (result != null) ? result : doLoadClass(name);
if (result == null) {
throw new ClassNotFoundException(name);
}
return resolveIfNecessary(result, resolve);
}
}
附录1：springboot jar目录结构
BOOT-INF
+ - classes 项目class文件
+ - lib 第三方库
+ - META-INF maven相关的文件和MANIFEST.MF
\- org spring boot loader
附录2：springboot 内嵌tomcat初始化流程
- 创建Tomcat
- Tomcat设置baseDir
- 创建Connector（默认协议：org.apache.coyote.http11.Http11NioProtocol）
- 创建Service并添加Connector
- 自定义配置Connector：customizeConnector
- Tomcat设置Connector
- 创建Host，设置自动部署为false
- 创建Engine，配置Engine
- 为Service添加附加的Connector
- 创建上下文、配置上下文。并将上下文绑定至Tomcat的Host的child。
- 启动Tomcat
1069
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
