dbcp连接池导致大量线程BLOCKED

一个线上老系统使用的dbcp1.4版本，近一段时间经常出现假死现象，客户端反应大量超时。

线上使用jstack获取线程堆栈，可以看到大量线程BLOCKED，

"pool-12-thread-5764" prio=10 tid=0x00007f5bac086800 nid=0x6eae waiting for monitor entry [0x00007f5c08bca000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:916)
	- waiting to lock <0x0000000680039cc8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.returnObject(AbandonedObjectPool.java:112)
	at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:87)
	- locked <0x000000068254b2b8> (a org.apache.commons.dbcp.PoolableConnection)
	at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
	at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:95)
	at org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:474)
	at org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:408)
	at org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:347)
	at org.hibernate.impl.SessionImpl.close(SessionImpl.java:326)
	at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:774)
	at org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:760)
	at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:400)
	at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:840)
	at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:832)
	at com.fuiou.cps.dao.hibernate.PfcBillLimitDaoHibernate.queryCreditTxnLimit(PfcBillLimitDaoHibernate.java:55)

pool-16-thread-8442" prio=10 tid=0x00007f5bc0073800 nid=0x6f39 waiting for monitor entry [0x00007f5c0a6e5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.getNumIdle(GenericObjectPool.java:911)
	- waiting to lock <0x0000000680039cc8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:78)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
	at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
	at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
	at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
	at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1354)
	at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:510)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:102)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
	at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:630)

   java.lang.Thread.State: RUNNABLE这些线程都在等待锁0x0000000680039cc8，但0x0000000680039cc8锁信息可以看到，是被另外一个线程持有，这个线程堆栈如下：

pool-16-thread-8441" prio=10 tid=0x00007f5bc0048800 nid=0x6f38 runnable [0x00007f5c087c5000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at com.ibm.db2.jcc.b.db.b(db.java:189)
	at com.ibm.db2.jcc.b.db.c(db.java:233)
	at com.ibm.db2.jcc.b.db.c(db.java:344)
	at com.ibm.db2.jcc.b.db.v(db.java:1130)
	at com.ibm.db2.jcc.b.gb.a(gb.java:38)
	at com.ibm.db2.jcc.b.w.a(w.java:34)
	at com.ibm.db2.jcc.b.vb.g(vb.java:139)
	at com.ibm.db2.jcc.c.fg.n(fg.java:1177)
	at com.ibm.db2.jcc.c.fg.a(fg.java:1832)
	at com.ibm.db2.jcc.c.fg.a(fg.java:473)
	at com.ibm.db2.jcc.c.fg.executeQuery(fg.java:457)
	- locked <0x0000000682655710> (a com.ibm.db2.jcc.b.b)
	at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
	at org.apache.commons.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:332)
	at org.apache.commons.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:855)
	- locked <0x0000000680039cc8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
	at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
	at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1577)
	at org.hibernate.loader.Loader.doQuery(Loader.java:696)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
	at org.hibernate.loader.Loader.doList(Loader.java:2232)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
	at org.hibernate.loader.Loader.list(Loader.java:2124)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
	at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
	at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
	at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1149)
	at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
	at org.springframework.orm.hibernate3.HibernateTemplate$29.doInHibernate(HibernateTemplate.java:849)
	at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:372)
	at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:840)
	at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:832)

分析以上堆栈可知，0x0000000680039cc8对象在这一行代码被占用：

at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:855)

我们查看代码，发现GenericObjectPool的borrowObject、returnObject方法是同步方法，

public synchronized Object borrowObject() throws Exception 
public synchronized void returnObject(Object obj) throws Exception

  而由于我们在获取连接时，设置了validationQuery，而线程在执行validateConnection方法时，socket读阻塞，导致AbandonedObjectPool对象锁一直被占用，进而引发大量线程阻塞，出现假死现象。

  方案： 我们最终的解决方法是用c3p0替换dbcp，连接池升级后，线上再无此类问题发生。