【新炬网络名师大讲堂】tomcat 环境下连接池泄露故障处理一例

某移动业务系统正常访问量就是50并发量，系统设置最大数据源连接数是200个，但每隔10分钟，监测数据源连接数就达到200个以上，导致系统无法正常使用，开发商尝试采用多台服务器进行负载处理，连接三天问题依旧。

接到这种问题，第一时间想到应该是连接泄露了，造成连接没有释放，可按照如下步骤进行排查:

1.检查数据源连接配置

数据源连接配置分两种，一种是采用中间件自带数据源配置，另外一种情况是应用程序自己使用第三方开源数据连接池。需要检查连接池是否配置了连接未使用超时设置，让系统自动关闭不活动连接

2.监测数据连接情况

利用第三方监测程序对数据源连接进行监测，可采用probe进行监测，如果程序使用了Proxool连接池，Proxoolw会自动提供监测数据源功能界面

3.线程情况

当数据源连接较高生成线程信息，看线程做什么，锁在哪

jrcmd.exe pid  print_threads >threaddump.log

“http-apr-8082-exec-114″ id=213 idx=0×374 tid=9344 prio=5 alive, native_blocked, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
at java/net/SocketInputStream.read(SocketInputStream.java:129)
at oracle/net/ns/Packet.receive()V(Unknown Source)
at oracle/net/ns/DataPacket.receive()V(Unknown Source)
at oracle/net/ns/NetInputStream.getNextPacket()V(Unknown Source)
at oracle/net/ns/NetInputStream.read([BII)I(Unknown Source)[inlined]
at oracle/net/ns/NetInputStream.read([B)I(Unknown Source)[inlined]
at oracle/net/ns/NetInputStream.read()I(Unknown Source)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:478)[optimized]
at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:213)[inlined]
at oracle/jdbc/driver/T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:796)[optimized]
at oracle/jdbc/driver/OracleStatement.executeMaybeDescribe(OracleStatement.java:1031)[optimized]
at oracle/jdbc/driver/T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:836)
at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1124)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3285)[optimized]
at oracle/jdbc/driver/OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3329)
    ^– Holding lock: oracle/jdbc/driver/T4CPreparedStatement@0x24562F78[biased lock]
    ^– Holding lock: oracle/jdbc/driver/T4CConnection@0×10884438[thin lock]
at net/sourceforge/jdbclogger/core/PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:402)
at sun/reflect/GeneratedMethodAccessor531.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/logicalcobwebs/proxool/ProxyStatement.invoke(ProxyStatement.java:100)[inlined]
at org/logicalcobwebs/proxool/ProxyStatement.intercept(ProxyStatement.java:57)[optimized]
at $java/sql/Statement$$EnhancerByProxool$$2fb99b38.executeQuery()Ljava/sql/ResultSet;()[optimized]
at org/hibernate/jdbc/AbstractBatcher.getResultSet(AbstractBatcher.java:208)[optimized]
at org/hibernate/loader/Loader.getResultSet(Loader.java:1808)[optimized]
at org/hibernate/loader/Loader.doQuery(Loader.java:697)[inlined]
at org/hibernate/loader/Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)[inlined]
at org/hibernate/loader/Loader.doList(Loader.java:2228)[inlined]
at org/hibernate/loader/Loader.listIgnoreQueryCache(Loader.java:2125)[inlined]
at org/hibernate/loader/Loader.list(Loader.java:2120)[inlined]
at org/hibernate/loader/custom/CustomLoader.list(CustomLoader.java:312)[inlined]
at org/hibernate/impl/SessionImpl.listCustomQuery(SessionImpl.java:1722)[optimized]
at org/hibernate/impl/AbstractSessionImpl.list(AbstractSessionImpl.java:165)[optimized]
at org/hibernate/impl/SQLQueryImpl.list(SQLQueryImpl.java:175)[optimized]
at com/gpdi/infores/pipeline/statistics/dao/hibernate/ComprehensiveDAOHibernate$4.doInHibernate(ComprehensiveDAOHibernate.java:139)[optimized]
at org/springframework/orm/hibernate3/HibernateTemplate.doExecute(HibernateTemplate.java:419)[inlined]
at org/springframework/orm/hibernate3/HibernateTemplate.execute(HibernateTemplate.java:339)[inlined]
at com/gpdi/infores/pipeline/statistics/dao/hibernate/ComprehensiveDAOHibernate.getQueryMapList(ComprehensiveDAOHibernate.java:121)[optimized]
at com/gpdi/infores/pipeline/flow/service/impl/InspectionScheduleServiceImpl.getAlsoArrange(InspectionScheduleServiceImpl.java:54)
at com/gpdi/infores/pipeline/flow/service/impl/InspectionScheduleServiceImpl$$FastClassByCGLIB$$889ae678.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;()
at net/sf/cglib/proxy/MethodProxy.invoke(MethodProxy.java:149)

从上面可以看出连接可能是锁在数据源ojdbc 上，可以尝试采用替换最新ojdbc jar 版本

 

4.检查系统代码连接是怎么获取及关闭

检查这个方面，取决于你是否有丰富的开发经验，因为目前开发一个应用系统，都会采用很多开源框架快速实现业务功能。以本次故障为例，开发商使用的spring 开源框架 管理数据源连接：

public List findList(String hsql) {
  Session session =this.getSession();
Query q =session.createQuery(hsql);
return q.list();
}

如果采用getSession这种方式获取连接，然后查询数据时，Spring并不能够自动管理连接，也就是说，每调用一次就申请一个连接，直到 连接池中的连接耗尽现象，Spring 建议采用getHibernateTemplate()采用这种方式进行操作连接，具体可参考如下spring 官方资料:

http://docs.spring.io/spring/docs/2.0.x/api/org/springframework/orm/hibernate3/support/HibernateDaoSupport.html#getSession()

Note that this is not meant to be invoked from HibernateTemplate code but rather just in plain Hibernate code. Either rely on a thread-bound Session or use it in combination with releaseSession(org.hibernate.Session).

spring 官方说了很清楚了getSession 连接事务spring 不管理，如果使用这种方法需使用releaseSession()方法才能关闭连接
protected final void releaseSession(Session session) Close the given Hibernate Session, created via this DAO’s SessionFactory, if it isn’t bound to the thread (i.e. isn’t a transactional Session).
Typically used in plain Hibernate code, in combination with getSession() and convertHibernateAccessException(org.hibernate.HibernateException).

解决办法

1.上面代码加入releaseSession()这个方法，进行手动关闭连接

public List findList(String hsql) {
Session session =this.getSession();
try {
Query q =session.createQuery(hsql);
return q.list();
}catch(Exception e){
System.out.print(e.getMessage());
}
finally {
   releaseSession(session);
}
return null;
}

2.在spring事务配置里面加入此方面事务配置，让不受管理的getSession 方法，自动被spring 事务进行管理

PROPAGATION_REQUIRED
PROPAGATION_REQUIRED
PROPAGATION_REQUIRED
PROPAGATION_REQUIRED
PROPAGATION_REQUIRED
PROPAGATION_REQUIRED
 PROPAGATION_REQUIRED
PROPAGATION_REQUIRED,readOnly



class=”org.springframework.orm.hibernate3.HibernateTransactionManager”>

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/29960155/viewspace-1369708/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/29960155/viewspace-1369708/