降低java虚拟机cpu使用率_java虚拟机CPU占用率过高排查方法

问题描述

一台M4000主机，操作系统是solaris10，上面的resin进程cpu占用率过高，达到了70%,如下：

-bash-3.00$ ps -ef -o pid,pcpu,args|grep java

1511  0.1 /usr/java/bin/java -Dwebview.htdocs=/etc/opt/FJSVwvcnf/htdocs/FJSVwvbs -mx128m

2135  0.0 /usr/java/bin/java -server -Xmx128m -XX:+BackgroundCompilation -XX:PermSize=32m

15945  0.0 sh -c /svi/jdk150/jdk1.5.0_06/bin/java  -server -Xms512m -Xmx3072m -XX:MaxPe

15946 70.7 /svi/jdk150/jdk1.5.0_06/bin/java -server -Xms512m -Xmx3072m -XX:MaxPermSize=

排查过程

1. 首先需要查找cpu占用率过高的LWP

-bash-3.00$ prstat -L -p 15946

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/LWPID

15946 slview 3336M 3301M sleep 15 0 3:56:27 2.2% java/49

15946 slview 3336M 3301M sleep 8 0 3:33:17 2.2% java/52

15946 slview 3336M 3301M sleep 12 0 3:32:20 2.2% java/50

15946 slview 3336M 3301M sleep 13 0 3:29:43 2.2% java/51

15946 slview 3336M 3301M sleep 13 0 3:30:54 2.2% java/47

15946 slview 3336M 3301M sleep 12 0 1:24:19 2.2% java/64

15946 slview 3336M 3301M sleep 15 0 1:07:55 2.1% java/144

2. 查找LWP与java线程的对应关系

-bash-3.00$ pstack 15946|grep lwp

----------------- lwp# 47 / thread# 47 --------------------

ff2c49fc _lwp_start (0, 0, 0, 0, 0, 0)

----------------- lwp# 48 / thread# 48 --------------------

ff2c5cd0 lwp_cond_wait (1704928, 1704910, 0, 0)

ff2c49fc _lwp_start (0, 0, 0, 0, 0, 0)

----------------- lwp# 49 / thread# 49 --------------------

ff2c49fc _lwp_start (0, 0, 0, 0, 0, 0)

----------------- lwp# 50 / thread# 50 --------------------

ff2c49fc _lwp_start (0, 0, 0, 0, 0, 0)

----------------- lwp# 51 / thread# 51 --------------------

ff2c49fc _lwp_start (0, 0, 0, 0, 0, 0)

----------------- lwp# 52 / thread# 52 --------------------

ff2c49fc _lwp_start (0, 0, 0, 0, 0, 0)

3. used the jstack find the callstack of thread

$ jstack -m 15946 获取所有线程的调用堆桟

hread t@50: (state = IN_VM)

- java.lang.AbstractStringBuilder.expandCapacity(int) @bci=28, line=99 (Compiled frame; information may be imprecise)

- per.xwnmp.flux.report.RptFluxHisQuery.GetFluxData(java.lang.String[], java.util.HashMap, java.lang.String, java.lang.Stri

ng, java.lang.String, java.lang.String, java.lang.String) @bci=480, line=509 (Interpreted frame)

- per.xwnmp.flux.report.RptFluxHisQuery.GenFluxReport(java.lang.String, java.lang.String[], java.lang.String, java.lang.Str

ing, java.lang.String, java.lang.String, java.lang.String) @bci=124, line=82 (Interpreted frame)

- _nos._flux._flux._FluxPerfView_0Excel__jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletRespo

nse) @bci=930, line=162 (Interpreted frame)

- com.caucho.jsp.JavaPage.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=9, line=75 (Interpreted frame)

- com.caucho.jsp.Page.subservice(com.caucho.server.http.CauchoRequest, com.caucho.server.http.CauchoResponse) @bci=214, line=506 (I

- com.caucho.server.TcpConnection.run() @bci=73, line=139 (Interpreted frame)

- java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)

Thread t@52: (state = IN_VM)

- per.xwnmp.flux.report.RptFluxHisQuery.GetFluxData(java.lang.String[], java.util.HashMap, java.lang.String, java.lang.Stri

ng, java.lang.String, java.lang.String, java.lang.String) @bci=435, line=508 (Compiled frame; information may be imprecise)

- per.xwnmp.flux.report.RptFluxHisQuery.GenFluxReport(java.lang.String, java.lang.String[], java.lang.String, java.lang.Str

ing, java.lang.String, java.lang.String, java.lang.String) @bci=124, line=82 (Interpreted frame)

- _nos._flux._flux._FluxPerfView_0Excel__jsp._jspService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletRespo

nse) @bci=930, line=162 (Interpreted frame)

- com.caucho.jsp.JavaPage.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=9, line=75 (Interpreted frame)

- com.caucho.jsp.Page.subservice(com.caucho.server.http.CauchoRequest, com.caucho.server.http.CauchoResponse) @bci=214, line=506 (I

nterpreted frame)