Tomcat DBCP连接池导致的线程阻塞问题

问题描述

最近在测试一个多线程任务时出现了我本地项目不报错，不停止，但是刷新任何的页面或者重新加载整个网站服务都无法继续执行操作的问题，也就是界面无响应，后台日志无输出且不报错卡死的情况。

解决流程

日志无法输出，因此查看了JVM线程堆栈的信息，怀疑时死锁的问题，在Jconsole中查看是否死锁现象。检测发现当前的所有线程都没有死锁存在，所以拖出来了日志，发现当前显示出来的所有状态不是 WAITING 就是 TIMED_WAITING ，不存在（运行状态的线程），不存在死锁那么就可能是线程阻塞了，日志显示大部分的线程都在等待 0x00000006d057ce08 这个资源，如下图所示：

"taskExecutor-3" #1158 prio=5 os_prio=0 tid=0x0000000032317800 nid=0x50e4 waiting on condition [0x000000008e72e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d057ce08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:586)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:359)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1543)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at com.hand.hap.task.service.impl.ExecuteServiceImpl$TaskThread.run(ExecuteServiceImpl.java:89)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"taskExecutor-2" #1156 prio=5 os_prio=0 tid=0x000000003230f800 nid=0x460 waiting on condition [0x000000008e62f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d057ce08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:586)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:359)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1543)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at com.hand.hap.task.service.impl.ExecuteServiceImpl$TaskThread.run(ExecuteServiceImpl.java:89)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"TASK-31825" #1154 prio=5 os_prio=0 tid=0x000000003232f800 nid=0x4208 waiting on condition [0x000000008e32d000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006d057ce08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:586)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
	at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:359)
	at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1543)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy731.updateProcessStatus(Unknown Source)
	at fms.xxmpl.biz.service.impl.MplTaskExecutionServiceImpl.batchManuscriptTaskExecute(MplTaskExecutionServiceImpl.java:169)
	at sun.reflect.GeneratedMethodAccessor3371.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:45005)
	at com.zeroturnaround.jrebelbase.facade.Forward.methodInvoke(SourceFile:247)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at com.hand.hap.core.impl.ServiceExecutionAdvice.invoke(ServiceExecutionAdvice.java:126)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy731.batchManuscriptTaskExecute(Unknown Source)
	at fms.xxmpl.biz.task.components.BatchManuscriptTask.execute(BatchManuscriptTask.java:32)
	at com.hand.hap.task.service.impl.ExecuteServiceImpl$TaskThread.operateExecutionInfo(ExecuteServiceImpl.java:225)
	at com.hand.hap.task.service.impl.ExecuteServiceImpl$TaskThread.executeTask(ExecuteServiceImpl.java:148)
	at com.hand.hap.task.service.impl.ExecuteServiceImpl$TaskThread.execute(ExecuteServiceImpl.java:120)
	at com.hand.hap.task.service.impl.ExecuteServiceImpl$TaskThread.run(ExecuteServiceImpl.java:96)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

但是并没有找到占用这个资源的而进程，不太方便定位问题的所在，看到下面的问题：

并且执行日志卡在了创建新的 sqlSession的日志处，之后就不在继续输出了，因此考虑是否是因为申请不到连接池的原因，导致线程阻塞的问题，再加上测试环境可以正常执行我的多线程任务，因此首先以为是数据库的原因：
1.是否是无法获取数据库的连接，查看了对应的数据库的当前连接数，以及最大连接数,使用量处于一半以下。不是该问题
2.是否是数据库锁表导致申请不到对应的资源，因此查看数据库是否缩表，也不是该问题

结合上面的日志参考测试环境的配置猜测可能是Tomcat配置的问题，Tomcat的连接池资源不够，因为Jconsle中执行程序的线程数上去之后就降不下来，线程本地的Context.xml文件中并没有配置配置相关的回收机制，和超时的处理机制，导致一直无限的 WAITING（线程的该状态表示无限期等地资源） 资源。

本地context.xml的配置是：

<Resource auth="Container" driverClassName="oracle.jdbc.driver.OracleDriver" url="jdbc:oracle:thin:@xx.xx.xx.xx:1531:SIT" name="jdbc/hfms_dev" type="javax.sql.DataSource" username="username" password="password"/>

看了sit的配置去了解了一下Tomcat使用的连接池： DBCP连接池，可以在里面配置数据库的连接以及对连接超时的处理情况，之后更改配置如下：

<Resource auth="Container" type="javax.sql.DataSource" name="jdbc/hfms_dev" 
        driverClassName="oracle.jdbc.driver.OracleDriver"
        url="jdbc:oracle:thin:@xx.xx.xx.xx:1531:SIT" 
        username="username" 
        password="password"
        maxWaitMillis="30000"
        maxTotal="3000"
        removeAbandonedOnBorrow="true" 
        removeAbandonedOnMaintenance="true" 
        removeAbandonedTimeout="180" />

maxWaitMillis=“30000” 连接最长的等待时间 ，单位是 毫秒 也就是 30 秒
maxTotal=“3000” 连接最大的总数: 3000

    **以下是用来配置数据库断开后自动连接的**
    removeAbandonedOnBorrow="true"  
    removeAbandonedOnMaintenance="true" 
    removeAbandonedTimeout="180"    等待移除的连接的超时时间 180s