应用程序死锁导致服务器挂起的介绍

Problem Description

An inadvertent deadlock in the application code can cause a server to hang. For example, a situation in which thread1 is waiting for resource1 and is holding a lock on resource2, while thread2 needs resource2 and is holding the lock on resource1. Neither thread can progress.

Problem Troubleshooting

This Application Deadlock pattern should be used only after doing all the steps in the Generic Server Hang pattern. One indicator that this is an application deadlock problem is that thread dumps will show the threads are in the application methods. Several thread dumps taken a few seconds apart will show that the threads are not progressing. Troubleshooting this problem will involve reviewing the application code. There exists a thread analyzer tool at BEA dev2dev which has proven useful in analysis of the thread dumps.

Quick Links

• Why does the problem occur?
• Known WebLogic Server Issues
• External Resources

 

Why does the problem occur?

Fundamentaly, this problem happens because the design and implementation of the application has introduced the possibility of deadlocks. These types of problems may only show up under heavy load. Therefore, these applications often pass through QA testing and become problems in production.

Coding problems to look for:

• Unnecessary use of synchronized java classes, e.g., using Hashtable (synchronized) versus the use of HashMap(unsynchronized)
• Application has a synchronized method that contains synchronized object method calls. See example below.

  import java.util.Vector;   <– Vector is a synchronized java class
  Public class Employee {
             Vector names = new Vector();
              Employees () {
                     Object object = new Object();
                      synchronized (object) {
                              names.add("al");
                              names.add("Saganich");
                       }
              }
              synchronized String getName (int index) {
                        String name = (String) names.elementAt(index);
                         return name;
              }

• Using synchronization around long running complex code.
• Threads are waiting on resources that will never become available.

Application Design

• The application uses up all of the configured number of threads. This can happen when an executing thread reaches a point where it must wait for work done by another thread to complete. The timing may be that the waited for method which this thread is trying to enter is long running. Eventually, all the threads must reach this long running method. After running awhile, the application will find that the threads will be lined up waiting for this long running method. No new work can be introduced because the allocated number of threads is all used up. See example below.

  import java…..;
  import java…..;
  public class myAppMethods {
            public String getName(String name) {
                       String lastname =  getLastName(name);
                       return lastname;
            }
            public synchronized String getLastName (String name) {
                       do a DB Lookup    <———— takes mucho time to get a last name
                        return lastname;
           }

  If the database if very slow, the server can appear to be hung because the threads will line up trying to get access to the database and all the available threads could eventually be used up.

• The application running inside a WebLogic Server invokes a service on another WebLogic instance on a remote machine. The remote service invoked on the remote machine makes a call back to the first server. This sets up the opportunity for a deadlock on the first server especially under heavy load. The first server has an execution thread that is tied up waiting for an inbound response. This inbound response will require a thread from the same execute pool as the thread that is waiting to receive the response.

  If the first server is faster than the remote server, eventually all the threads in the execute pool will be exhausted by the server making outbound requests with fewer threads available for processing inbound responses. As the load grows, the number of outgoing requests that cannot complete their work grows while they wait for an inbound response to complete. Below is an example of a thread in the waitForDataResponseImpl.java method of.

  "ExecuteThread: ‘52′ for queue: ‘default’" daemon prio=5 tid=0×4b3e40b0 nid=0×1170 waiting on monitor [0x4c74f000..0x4c74fdbc] 
  at java.lang.Object.wait(Native Method)
  at 
  weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)

Top of Page

Known WebLogic Server Issues

WebLogic Server cannot detect deadlocked threads. Some JVM’s are able to do so. See External Resources. There is a tool available for thread analysis as well as good information about thread dumps on BEA dev2dev.

Top of Page

External Resources

If you suspect a deadlock, it is helpful to go to the site of the JVM vendor to learn if there are clues provided for you in their thread dumps. If you are using JDK 1.3.1 you can add - XX:+JavaMonitorInStack Trace to show locking more explicitly/builtin. Here is an example of an HP JVM thread dump in which it is clearly marked that the threads’ state are waiting on monitor, and the monitor is identified. See example below.

"msg 0-941667944865" (TID:0×7b1a5ba0, sys_thread_t:0×2a4290,
  state:Waiting on Monitor,
  thread_t: t@108, stack_base:0×7a76e000, stack_size:0×20000,
pc: 0xc01ea178, monitor = 0×25334) prio=8
  MsgThread.rest(Compiled Code)
  MsgThread.run(Compiled Code)
"msg 3-941667944865" (TID:0×7b1a5a80, sys_thread_t:0×263278,
  state:Waiting on Monitor,
  thread_t: t@105, stack_base:0×7a95d000, stack_size:0×20000,
pc: 0xc01ea178, monitor = 0×25334) prio=8
  MsgThread.rest(Compiled Code)
  MsgThread.run(Compiled Code)
"msg 5-941667944864" (TID:0×7b1a5f08, sys_thread_t:0×2c42d8,
  state:Waiting on Monitor,
  thread_t: t@106, stack_base:0×7aa65000, stack_size:0×20000,
pc: 0xc01ea178, monitor = 0×25334) prio=8
  MsgThread.rest(Compiled Code)
  MsgThread.run(Compiled Code)

 

 

 

备注：

本文转载自：http://www.hashei.me/2009/08/serverhang_application_deadlock.html