Oracle连接数满了

项目逐渐做大，分布式微服务越部署越多，会遇到 数据库连接数不够的情况。

image.png

采用直连的方式报错如下：

00:00:22.670 |Weekday_Thread-220523178|ERROR| c.e.m.b.t.WeekdayManageThread - 线程中断运行：Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.S
QLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot 
get a connection, pool error Timeout waiting for idle object
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:238) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at com.sun.proxy.$Proxy2417.doWeekdayControl(Unknown Source) ~[na:na]
    at com.xxx.matrix.base.thread.xxxx.run(xxxx.java:43) ~[its.xxx.xxxxx-1.70.jar:1.70]
    at java.lang.Thread.run(Thread.java:784) [na:1.6.0]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104) ~[commons-dbcp-1.2.2.jar:1.2.2]
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) ~[commons-dbcp-1.2.2.jar:1.2.2]
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:200) ~[spring-2.5.6.jar:2.5.6.SEC03]
    ... 10 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) ~[commons-dbcp-1.2.2.jar:1.2.2]
    ... 12 common frames omitted

使用websphereb（was）报错如下：

10:39:51.130 |Weekday_Thread878128082|ERROR| c.e.m.b.t.xx - 线程中断运行：Could not open JDBC Connection for transaction; nested exception is com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:238) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:374) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:263) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) ~[spring-2.5.6.jar:2.5.6.SEC03]
    at com.sun.proxy.$Proxy109.doWeekdayControl(Unknown Source) ~[na:na]
    at com.xxx.matrix.base.thread.xxx.run(WeekdayManageThread.java:43) ~[its.xxx.xxx.xx-1.70.jar:1.70]
    at java.lang.Thread.run(Thread.java:784) [na:1.6.0]
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
    at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1680) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:661) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611) ~[com.ibm.ws.runtime.jar:na]
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:200) ~[spring-2.5.6.jar:2.5.6.SEC03]
    ... 10 common frames omitted
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: CWTE_NORMAL_J2CA1009
    at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1782) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3871) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3091) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1548) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:1031) ~[com.ibm.ws.runtime.jar:na]
    at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:644) ~[com.ibm.ws.runtime.jar:na]
    ... 12 common frames omitted

通常遇到这样的问题，排查方向：

• 1、检查应用的连接数，是不是配置的小了。
• 2、如果使用数据源，检查数据源的连接数是不是配置小了。
• 3、检查数据库的最大连接数和当前连接数，是否是公用的数据库，被别人占用了
• 4、如果上面均正常，那么查询时间最长的sql，是否是有锁死等待的sql，一直在创建，但是一直锁等待。

---- 当前连接数
select count(*) from v$process ;
--- 最大连接数
select value from v$parameter where name ='processes';  
--- 当前所有用户，按照连接数排序
 select username,count(username) as con from v$session where username is not null group by username  order by con desc;
 
---- 按执行时间排序
 SELECT osuser, a.username,cpu_time/executions/1000000||'s', sql_fulltext,machine
from v$session a, v$sqlarea b
where a.sql_address =b.address order by cpu_time/executions desc;