WebLogic Server (WLS) Hangs With Deadlocked Socket Close() and Poll() Threads on AIX 【转】

最新推荐文章于 2018-10-29 14:31:38 发布

小腿vs15

最新推荐文章于 2018-10-29 14:31:38 发布

阅读量2.4k

点赞数

分类专栏： Weblogic 文章标签： weblogic aix socket server patch ibm

Weblogic 专栏收录该内容

12 篇文章 0 订阅

订阅专栏

Applies to:

Oracle Weblogic Server – Version: 8.1 to 10.3
IBM PS/2 AIX
IBM IA64 AIX
IBM AIX on POWER Systems (64-bit)
IBM ES9000 AIX/ESA
Bull Escala AIX
IBM RT/PC AIX
IBM 370 AIX
IBM AIX on POWER Systems (32-bit)
AIX Based Systems

Symptoms

On the IBM AIX platform, all the servers hang. The hang is not related to load, and usually occurs within 10 minutes or so of server startup. A thread dump reveals two deadlocked threads, one doing a socket close() and one doing a socket poll(). For example:

'[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'' (TID:0x438F2000, sys_thread_t:0x438F9EF8, state:R, native ID:0x0037F035) prio=5
 at java/net/PlainSocketImpl.socketClose0(Native Method)
 at java/net/PlainSocketImpl.socketPreClose(PlainSocketImpl.java:744(Compiled Code))
 at java/net/PlainSocketImpl.close(PlainSocketImpl.java:568(Compiled Code))
 at java/net/SocksSocketImpl.close(SocksSocketImpl.java:1049(Compiled Code))
 at java/net/Socket.close(Socket.java:1341(Compiled Code))
 at weblogic/socket/WeblogicSocket.close(WeblogicSocket.java:80(Compiled Code))
 at weblogic/socket/SocketMuxer.closeSocket(SocketMuxer.java:449(Compiled Code))
 at weblogic/socket/SocketMuxer.cancelIo(SocketMuxer.java:773)
 at weblogic/socket/SocketMuxer$TimerListenerImpl.timerExpired(SocketMuxer.java:976(Compiled Code))
 at weblogic/timers/internal/TimerImpl.run(TimerImpl.java:273(Compiled Code))
 at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464(Compiled Code))
 at weblogic/work/ExecuteThread.execute(ExecuteThread.java:197(Compiled Code))
 at weblogic/work/ExecuteThread.run(ExecuteThread.java:172(Compiled Code))

'ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'' (TID:0x3511B500, sys_thread_t:0x34FAD6C8, state:R, native ID:0x003E80CB) prio=5
at weblogic/socket/PosixSocketMuxer.poll(Native Method)
at weblogic/socket/PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

Cause

Internally, WebLogic Server polls the sockets to see if there is any new activity. The trouble is we are also calling socket.close() to try and clear the sockets of existing clients at the same time.

According to IBM engineering:

The underlying TCP implementation of socket close was changed between IBM JDK SR5 and SR7. Part of that change is that dup2 is now used.
Because of the dup2 change, the poll() and close() methods cannot be called at the same time on the same socket.

This causes the deadlock.

Bug 8173442 addresses this issue.

Solution

Patches are available for bug 8173442.

Patches for WLS 8.x can be found in My Oracle Support. Open the Patches & Updates tab. Search for patch ID 8173442 for the patches for WLS 8.1mp3, 8.1mp4, and 8.1mp5. Search for patch ID 8179792 for the patch for WLS 8.1mp6.

Patches for WLS 9.x and higher can be downloaded from Smart Update using these patch IDs and passcodes:

Fixed in: 10.0.2, 10.3.1

To apply one of these patches, download the patch using Smart Update and apply: see Note 885851.1 for step-by-step instructions. If your target system is offline, please refer to Note 876004.1 for additional assistance. All the details about SmartUpdate are available in the SmartUpdate documentation.

Patches are specifically tied to a particular release and maintenance pack of WebLogic Server (WLS). A patch for WLS 9.2.2, for example, very likely would not work on WLS 9.2.3. In a few cases, patches are also specific to particular operating systems. If you think you are experiencing the problem outlined in this note, but your WLS version is not included in the list of patches provided here, please contact support and ask for a version of the patch appropriate for you. Please reference this note as well as this bug number to help speed our service for you.