MustGather: No response (hang) or performance degradation on AIX
Technote (troubleshooting)
Problem(Abstract)
MustGather for when your IBM® WebSphere® Application Server is not responding (hangs) on the AIX® platform. Gathering this information before calling IBM® support helps familiarize you with the troubleshooting process and saves you time.
Resolving the problem
Do you want to automate the collection of MustGather data?
No response (hang) or performance degradation specific MustGather information To troubleshoot a hung Java™ virtual machine (JVM™), collect the following information when a hang occurs.
Follow these instructions for initial environment set up before you re-create the hang problem:
Synchronize clocks on all Web server and WebSphere Application Server systems. If clocks on the Web server and WebSphere Application Systems were not synchronized, state which clock was faster and the difference between the two times.
If possible, clear all Web server, plug-in, WebSphere Application Servers and FFDC logs.
When you experience the hang, enter the following commands from a command-line prompt:
netstat -an > netstat.out
If the Web server is remote, gather netstat output on the Application Server system and the Web server system.
vmstat 5 12 > vmstat.out
Collect CPU usage statistics by entering the following command:
tprof -k -s -e -x sleep 60
Generate a set of three Javacores for the hung JVM by issuing the following commands:
A. kill -3 [PID_of_hung_JVM]
B. Wait two minutes.
C. kill -3 [PID_of_hung_JVM]
D. Wait two minutes
E. kill -3 [PID_of_hung_JVM]
./dbxtrace_aix.sh -a [PID_of_hung_JVM] > dbx-hung.out
If you are serving static content from a WebServer, please determine whether your WebServer is responding to requests for that information. Determine whether you can serve dynamic documents such as servlets or JSPs when bypassing the WebServer and WebSphere Application Server plug-in.
Collect the following information:
For WebSphere Application Server V6.0 and v6.1 releases:
The server.xml file located in the install_root/profiles/profile_name/config/cells/cell_name/nodes/ node_name/servers/server_name
plugin-cfg.xml and http_plugin.log
Everything in the install_root/profiles/profile_name/logs/server_name directory
Everything in the install_root/profiles/profile_name/logs/ffdc directory
Send all files generated by tprof, particularly _prof.all. These files are in the directory from which the tprof command was run and they begin with an underscore "_". Some versions of AIX might produce only a sleep.prof file.
Send the configuration and log files from the Web server system. For example, for IBM HTTP Server and Apache Web, server provide the httpd.conf, access and error logs. If you are running the Sun ONE™ 6 Web server, provide magnus.conf, obj.conf, access and error logs.
If you want to analyze the Java thread dumps yourself, download the IBM Thread and Monitor Dump Analyzer for Java Technology. ThreadAnalyzer is a technology preview that can analyze thread dumps from WebSphere Application Server. It is useful for identifying deadlocks, contention, and bottlenecks, as well as for summarizing the state of threads within WebSphere Application Server.
For a listing of all technotes, downloads, and educational materials specific to the Hangs/Performance Degradation component, search the WebSphere Application Server support site.