Weblogic读取blob堵塞的原因

  很久之前的一个问题今天终于得到了答案。
  之前是有这么一个stuck请求，它上了一个锁导致后来的数据库链接都在等待这个锁，上次是没有找到原因，因为只是读取一条数据的一个blob字段，这个字段并不大。
"[STUCK] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f6108009800 nid=0x46bd runnable [0x00007f60d1ad0000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:293)
at oracle.net.ns.DataPacket.receive(DataPacket.java:104)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)
at oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1504)
at oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:450)
at oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:767)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:358)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)
at oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2346)
- locked <0x0000000703ccd180> (a oracle.jdbc.driver.T4CConnection)
at oracle.sql.BLOB.getBytes(BLOB.java:331)
at oracle.sql.BLOB.getBytes(BLOB.java:217)
at oracle.jdbc.driver.T4CBlobAccessor.getBytes(T4CBlobAccessor.java:464)
at oracle.jdbc.driver.OracleResultSetImpl.getBytes(OracleResultSetImpl.java:708)
- locked <0x0000000703ccd180> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleResultSet.getBytes(OracleResultSet.java:398)
at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.getBytes(Unknown Source)
at com.gg.bpms.engine.deploy.model.DefinitionModelVOConvert.voConvert(DefinitionModelVOConvert.java:80)
.................................................................


今天有一个服务发生了告警，这是server日志的报的错，是读取一个外系统的功能，由于外系统的功能出了问题，所以一直都连不上，stuck也是相关的错误。
<BEA-000000> <2017/12/29 01:05:05:740 CST [ERROR] CommonUtils - Tried all: '1' addresses, but could not connect over HTTP to server: '10.10.181.132', port: '80' <java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: '10.10.181.132', port: '80'>java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: '10.10.181.132', port: '80'
at weblogic.net.http.HttpClient.openServer(HttpClient.java:333)
at weblogic.net.http.HttpClient.openServer(HttpClient.java:425)
at weblogic.net.http.HttpClient.New(HttpClient.java:252)
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:189)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:374)
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
at org.jasig.cas.client.javafilter.util.CommonUtils.getResponseFromServer(CommonUtils.java:392)

<[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "645" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@7719c697[
GET /web/gg/Login.jsp?verifyCode=123 HTTP/1.1
User-Agent: Jodd HTTP
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:224)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:148)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:377)
weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
org.jasig.cas.client.javafilter.util.CommonUtils.getResponseFromServer(CommonUtils.java:392)

下面是这个服务jstack的结果：
"[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f99e4005000 nid=0x400f runnable [0x00007f99947c1000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x0000000716c42dd8> (a java.io.BufferedInputStream)
at weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:224)
at weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:148)
at weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:377)
- locked <0x0000000716c43048> (a weblogic.net.http.SOAPHttpURLConnection)
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)
at org.jasig.cas.client.javafilter.util.CommonUtils.getResponseFromServer(CommonUtils.java:392)

可以看到两类错误上面的堆栈都是一样，所以第一个问题就是当时网络出了问题，导致连接一直hang住。