weblogic大量功能慢导致的问题

  现场报节点不能使用。

####<2018-1-2 上午09时41分49秒 CST> <Error> <WebLogicServer> <AQSCGLXT15> <FWMS_23_8004> <[ACTIVE] ExecuteThread: '155' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1514857309373> <BEA-000337> <[STUCK] ExecuteThread: '59' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@1b831c3d[POST /web/forinstance/instrument/add]

", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
java.lang.Object.wait(Native Method)
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:685)
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:650)
weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1305)
weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)
weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1677)
weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2243)
weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:290)
weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:250)
org.glassfish.transaction.TransactionManagerImplCommon.commit(TransactionManagerImplCommon.java:535)
org.springframework.transaction.jta.JtaTransactionManager.doCommit(JtaTransactionManager.java:1011)
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:755)
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724)
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:475)
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:270)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)

从jstack可以看到由于13号线程堵塞了接近400个线程，如下所示：
"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fe3cc008800 nid=0x15b03 runnable [0x00007fe3e8b7f000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at oracle.net.ns.Packet.receive(Packet.java:300)
at oracle.net.ns.DataPacket.receive(DataPacket.java:106)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1119)
at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:375)
- locked <0x000000070113ad78> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:279)
- locked <0x000000070113ad78> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
at gg.bpms.db.EnginePostJdbcAdapterImpl.queryUsersByPostAndOrgCode(EnginePostJdbcAdapterImpl.java:321)
at gg.bpms.db.EnginePostJdbcAdapterImpl.recycleGetUsersByPostIdAndOrgId(EnginePostJdbcAdapterImpl.java:285)


"[ACTIVE] ExecuteThread: '157' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007fe3cc0e1800 nid=0x3878 waiting for monitor entry [0x00007fe053e6d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:342)
- waiting to lock <0x000000070113ad78> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.jta.DataSource.end(DataSource.java:953)
at weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1306)
at weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:424)
at weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:344)
at weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1689)
at weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2082)
at weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3238)
at weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:59)
at weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3480)
at weblogic.transaction.internal.PlatformHelperImpl$3.run(PlatformHelperImpl.java:1052)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:312)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:264)

总结：这个功能查询比较慢，执行一次大概在60s，当出现大量这样的请求时，weblogic就出现过载了。打一个基线：正常的weblogic是1一百多个，出问题的时候，线程多大405个（不算GC的线程）。