问题描述
有一个很特殊的case,使用Agile PLM 9.3产品每隔2~3天,用户无法登录,而已经登录的用户无法继续执行任何操作,用户界面hang住。检查opmn日志,没有有用的信息,错误太杂乱不明确。检查Oracle Database,CPU, Memory,连接都没有任何问题。重新启动SOA,问题临时不会出现,但2~3天后,问题继续。
Oracle环境
- Middleware: Win2003 + Oracle SOA 10.1.3.4
- Database: Win2003 + Oracle Database 10.2.0.4
- Application: Agile PLM 9.3.0
分析线程
在Windows2003平台上,抓出opmn的线程dump如下:
"AJPRequestHandler-HTTPThreadGroup-226" prio=6 tid=0x637dfd20 nid=0x18bc waiting for monitor entry [0x6c3be000..0x6c3bfc1c]
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:178)
- waiting to lock <0x194cf728> (a oracle.jdbc.pool.OracleDataSource)
at oracle.oc4j.sql.DataSourceConnectionPoolDataSource.getPooledConnection(DataSourceConnectionPoolDataSource.java:87)
at oracle.oc4j.sql.xa.EmulatedXADataSource.getXAConnection(EmulatedXADataSource.java:117)
at oracle.oc4j.sql.spi.ManagedConnectionFactoryImpl.createXAConnection(ManagedConnectionFactoryImpl.java:209)
at oracle.oc4j.sql.spi.ManagedConnectionFactoryImpl.createManagedConnection(ManagedConnectionFactoryImpl.java:170)
at com.evermind.server.connector.ApplicationConnectionManager.createManagedConnection(ApplicationConnectionManager.java:1398)
at oracle.j2ee.connector.ConnectionPoolImpl.createManagedConnectionFromFactory(ConnectionPoolImpl.java:327)
at oracle.j2ee.connector.ConnectionPoolImpl.access$800(ConnectionPoolImpl.java:98)
at oracle.j2ee.connector.ConnectionPoolImpl$NonePoolingScheme.getManagedConnection(ConnectionPoolImpl.java:1211)
at oracle.j2ee.connector.ConnectionPoolImpl.getManagedConnection(ConnectionPoolImpl.java:785)
at oracle.oc4j.sql.ConnectionPoolImpl.getManagedConnection(ConnectionPoolImpl.java:45)
at com.evermind.server.connector.ApplicationConnectionManager.getConnectionFromPool(ApplicationConnectionManager.java:1