诊断weblogic告警必须要有jstack_jstack分析weblogic报错-CSDN博客

本文链接：https://blog.csdn.net/stevendbaguo/article/details/79024041

上午10点收到GX现场weblogic告警，由于server日志是每天生成一份，所以看到的是今天的堵塞，连接池不够用了。

####<2018-1-10 上午07时47分03秒 CST> <Error> <WebLogicServer> <zcyy71> <mm_app01_8001> <[ACTIVE] ExecuteThread: '110' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1515541623723> <BEA-000337> <[STUCK] ExecuteThread: '100' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "651" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@56e1f809[

POST /web/gg/controlquality/common/getReadyData.sc HTTP/1.1
X-Requested-With: XMLHttpRequest
Accept: application/json, text/javascript, */*; q=0.01
Referer: http://10.100.81.120/web/gg/purchase/contracteventtrack/ContractEventTrackToSignNew.jsp?tSession=1515540971309
Accept-Language: zh-cn
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; BOIE9;ZHCN)
Content-Length: 0
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: DWRSESSIONID=nxVxEsqGULlin2EyUpDZ2NBht4m; JSESSIONIDMINI=vv70hVRMJyQJmMh29fFJBQGWSPC2GF5LhZCkDnvDGZXJ49mSBVj1!-1775855159; ComtopSessionSID=<SNAID>mJ1xhVRfFMGbGkwHqfvrLwwM7125Q4mGTFDt0VV81hJyF50Ll2JJ!-2095660264!1515540863114</SNAID>; cs_cStorage_=1; JSESSIONID=HJTdRl9KgW1m34fvJJ-FvpiQUAs02L2XUTFqCkTMxmTcinSf-KzC!-189191805
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.lang.Object.wait(Native Method)
weblogic.transaction.internal.XAResourceDescriptor.startResourceUse(XAResourceDescriptor.java:643)
weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1230)
weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1164)
weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:285)
weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:522)
weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:449)
weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:349)
weblogic.jdbc.jts.Driver.connect(Driver.java:173)
weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:364)
org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:84)
org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:58)
org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:271)
org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:69)
org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:56)
org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)
org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)

org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)

随手做了一份jstack，发现了有一个stuck的进程,持有了锁，堵塞了几个进程：

"[STUCK] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f54f8008800 nid=0x628b runnable [0x00007f53cdf89000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:293)
at oracle.net.ns.DataPacket.receive(DataPacket.java:104)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1006)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1315)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3677)
- locked <0x000000074602ea10> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4714)
- locked <0x000000074602ea10> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1374)
at weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:99)
at org.apache.ibatis.executor.statement.CallableStatementHandler.update(CallableStatementHandler.java:45)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:66)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:45)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:100)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:59)
at sun.proxy.$Proxy1263.update(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:358)
at sun.proxy.$Proxy1203.update(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.update(SqlSessionTemplate.java:254)
at com.gg.common.dao.CommonDAO.execUpdate(CommonDAO.java:220)
at com.gg.material.controlquality.workflow.arrivalsampling.appservice.impl.CqWorkflowASAppService.postASDataToCSG(CqWorkflowASAppService.java:406)