目前我们的监控,可以发现消耗较高CPU的进程(阀值为3个CPU),通过监控我们可以找到消耗较高CPU的进程号;
通过进程号pid,我们在linux上可以通过top –H –p <pid>命令,显示该进程中每个线程的CPU资源消耗情况。
然后通过threadump命令,可以打印出某个应用JVM中某时刻所有线程的调用情况,通过线程号我们就可以对应找到线程调用的程序了。
另:
使用jrcmd命令也可以进行threadump和进程执行线程情况的查看,但该命令为jrockit 5.0 新带的命令,在wls81用户下,我们没有执行权限,但可以联系中间件帮忙执行。
1. 进入到jrockit/bin目录下
2. 打./jrcmd 回车,会看到当前所有在执行中的java进程的PID号
3. 打./jrcmd [PID] help 回车,会看到指定java进程支持的操作
4. 打./jrcmd [PID] print_threads > xxxFile,可以把当前java进程中的所有线程堆栈轨迹打印到指定文件中
示例:
[serviceop@CNSZ031330 wls]$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
228961 wls81 15 0 714m 637m 1804 S 755.7 0.2 383:24.53 java
[serviceop@CNSZ031330 wls]$ top -H -p 228961
top - 13:37:01 up 156 days, 16:11, 8 users, load average: 14.24, 13.82, 12.88
Tasks: 140 total, 23 running, 117 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.6%us, 1.6%sy, 0.0%ni, 72.6%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 264120292k total, 217896772k used, 46223520k free, 4547804k buffers
Swap: 3145720k total, 464k used, 3145256k free, 153619224k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
229077 wls81 15 0 714m 637m 1804 R 32.6 0.2 7:11.79 ExecuteThread:
229093 wls81 15 0 714m 637m 1804 R 32.6 0.2 23:41.47 ExecuteThread:
229073 wls81 15 0 714m 637m 1804 R 32.2 0.2 2:24.30 ExecuteThread:
229076 wls81 15 0 714m 637m 1804 R 32.2 0.2 2:22.75 ExecuteThread:
229078 wls81 15 0 714m 637m 1804 R 32.2 0.2 7:21.38 ExecuteThread:
229082 wls81 15 0 714m 637m 1804 R 32.2 0.2 10:14.74 ExecuteThread:
229084 wls81 15 0 714m 637m 1804 R 32.2 0.2 8:43.72 ExecuteThread:
229085 wls81 15 0 714m 637m 1804 R 32.2 0.2 12:45.62 ExecuteThread:
229087 wls81 15 0 714m 637m 1804 R 32.2 0.2 19:11.79 ExecuteThread:
229089 wls81 15 0 714m 637m 1804 R 32.2 0.2 23:22.67 ExecuteThread:
229075 wls81 15 0 714m 637m 1804 R 31.9 0.2 5:06.52 ExecuteThread:
229088 wls81 15 0 714m 637m 1804 R 31.9 0.2 13:19.55 ExecuteThread:
229090 wls81 15 0 714m 637m 1804 R 31.9 0.2 23:53.55 ExecuteThread:
229092 wls81 15 0 714m 637m 1804 R 31.9 0.2 24:20.10 ExecuteThread:
229094 wls81 15 0 714m 637m 1804 R 31.9 0.2 33:25.41 ExecuteThread:
229095 wls81 15 0 714m 637m 1804 R 31.9 0.2 33:41.20 ExecuteThread:
229096 wls81 15 0 714m 637m 1804 R 31.9 0.2 36:22.72 ExecuteThread:
229081 wls81 15 0 714m 637m 1804 R 31.6 0.2 7:28.80 ExecuteThread:
229083 wls81 15 0 714m 637m 1804 R 31.6 0.2 11:05.48 ExecuteThread:
229086 wls81 15 0 714m 637m 1804 R 31.6 0.2 11:40.36 ExecuteThread:
229091 wls81 15 0 714m 637m 1804 R 31.6 0.2 17:53.92 ExecuteThread:
229097 wls81 15 0 714m 637m 1804 R 31.6 0.2 32:41.28 ExecuteThread:
229079 wls81 15 0 714m 637m 1804 R 31.2 0.2 7:33.44 ExecuteThread:
229080 wls81 15 0 714m 637m 1804 S 1.3 0.2 0:14.13 ExecuteThread:
228963 wls81 15 0 714m 637m 1804 S 0.7 0.2 0:13.44 (GC Main Thread
229070 wls81 15 0 714m 637m 1804 S 0.7 0.2 0:05.97 ExecuteThread:
229664 wls81 15 0 714m 637m 1804 S 0.7 0.2 0:08.69 pool-1-thread-7
Threadump信息片段:
229083:
"ExecuteThread: '35' for queue: 'weblogic.kernel.Default'" id=44 idx=0xd4 tid=229083 prio=5 alive, daemon
at java/util/HashMap.get(HashMap.java:319)
at com/paic/abbs/biz/service/common/impl/HardCodeServiceImpl.getString(HardCodeServiceImpl.java:115)
at com/paic/abbs/biz/service/util/impl/UtilServicesImpl.isExpected(UtilServicesImpl.java:45)
at com/paic/abbs/biz/service/common/impl/AbbsInternetServiceImpl.policyIssuing(AbbsInternetServiceImpl.java:3156)
at com/paic/abbs/biz/service/common/impl/AbbsInternetServiceImpl.procRequest(AbbsInternetServiceImpl.java:218)
at com/paic/abbs/biz/service/common/impl/AbbsInternetServiceImpl.handleRequest(AbbsInternetServiceImpl.java:122)
at com/paic/abbs/biz/service/common/impl/AbbsInternetService_gwxgqo_EOImpl.handleRequest(AbbsInternetService_gwxgqo_EOImpl.java:262)
at sun/reflect/GeneratedMethodAccessor287.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodA