AIX/WEBLOGIC81_CPU高实验

[img]http://dl.iteye.com/upload/attachment/210658/9a0e063a-2fdc-35c9-a933-7e6fcd4fcde6.jpg[/img]


1、查看“CP”列（表示 CPU 占用率），看其中哪些线程的此项值比较高并从中挑选一个线程。
ps -mp 458960 -o THREAD > thd.txt
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
weblogic 458960 417824 - A 120 60 49 * 202001 pts/4 - /usr/java14/bin/java -Xms128M -Xmx256M -Dweblogic.Name=myserver -Dweblogic.ProductionModeEnabled= -Djava.security.policy=/home/weblogic/bea/weblogic81/server/lib/weblogic.policy weblogic.Server
- - - 188655 S 0 60 1 f100070f10002e40 8410400 - - -
- - - 430329 S 0 60 1 f100070f10006940 8410400 - - -
- - - 458909 S 0 60 1 f100070f10007040 8410400 - - -
- - - 471059 S 0 60 1 4159cfa0 c10400 - - -
- - - 635059 S 0 60 1 f100070f10009b40 8410400 - - -
- - - 651471 S 0 60 1 f100070f10009f40 8410400 - - -
- - - 663785 S 0 60 1 f100070f1000a240 8410400 - - -
- - - 667763 S 0 60 1 f100070f1000a340 8410400 - - -
- - - 671917 S 0 60 1 f100070f1000a440 8410400 - - -
- - - 708691 S 0 60 1 f100070f1000ad40 8410400 - - -
- - - 733415 S 0 60 1 f100070f1000b340 8410400 - - -
- - - 753773 S 0 60 1 f100070f1000b840 8410400 - - -
- - - 757787 S 0 60 1 f100070f1000b940 8410400 - - -
- - - 823453 S 0 60 1 f100070f1000c940 8410400 - - -
- - - 827635 S 0 60 1 f100070f1000ca40 8410400 - - -
- - - 831717 S 1 60 1 f100070f1000cb40 8410400 - - -
- - - 839881 S 0 60 1 f100070f1000cd40 8410400 - - -
- - - 843823 S 0 60 1 f10001001e4ebeb0 410400 - - -
- - - 847939 S 0 60 1 f100070f1000cf40 8410400 - - -
- - - 856157 S 0 60 1 f100070f1000d140 8410400 - - -
- - - 860185 S 0 60 1 f100070f1000d240 8410400 - - -
- - - 876683 S 0 60 1 f100070f1000d640 8410400 - - -
- - - 905271 S 0 60 1 f100070f1000dd40 8410400 - - -
- - - 913639 S 0 60 1 f100070f1000df40 8410400 - - -
- - - 917655 S 0 60 1 4159cfa0 c10400 - - -
- - - 921761 S 0 60 1 f100070f1000e140 8410400 - - -
- - - 937999 S 0 60 1 f100070f1000e540 8410400 - - -
- - - 946407 S 0 60 1 f100070f1000e740 8410400 - - -
- - - 966767 S 0 60 1 f100070f1000ec40 8410400 - - -
- - - 995523 S 0 60 1 f100070f1000f340 8410400 - - -
- - - 999661 S 0 60 1 f100070f1000f440 8410400 - - -
- - - 1036539 S 0 60 1 f100070f1000fd40 8410400 - - -
- - - 1056917 S 0 60 1 f100070f10010240 8410400 - - -
- - - 1065187 S 0 60 1 f100070f10010440 8410400 - - -
- - - [b][color=red]1073241 [/color][/b]R 119 119 0 - 400000 - - -
- - - 1081519 S 0 60 1 f100070f10010840 8410400 - - -
- - - 1089721 S 0 60 1 f100070f10010a40 8410400 - - -
- - - 1142941 S 0 60 1 f100070f10011740 8410400 - - -
- - - 1147049 S 0 60 1 f100070f10011840 8410400 - - -
- - - 1175749 Z 0 60 1 - c00001 - - -
- - - 1192135 S 0 60 1 f100070f10012340 8410400 - - -
- - - 1302777 S 0 60 1 f100070f10013e40 8410400 - - -
- - - 1343695 S 0 60 1 f100070f10014840 8410400 - - -
- - - 1351853 S 0 60 1 f100070f10014a40 8410400 - - -
- - - 1368219 S 0 60 1 f100070f10014e40 8410400 - - -
- - - 1372413 S 0 60 1 f100070f10014f40 8410400 - - -
- - - 1392711 S 0 60 1 f100070f10015440 8410400 - - -
- - - 1409063 S 0 60 1 f100070f10015840 8410400 - - -
- - - 1417347 S 0 60 1 f1000200020ba208 400400 - - -
- - - 1433711 S 0 60 1 f100070f10015e40 8410400 - - -

2. 通过执行以下命令对服务器进行 Thread Dump：
kill -3 <WLS_PID>

3. 运行： dbx -a <WLS_PID>
4. 在 dbx 中时，运行 thread命令（以列出所有线程）。
5. 查找与您通过 ps -mp <PID -o THREAD 命令获取的 TID 匹配的行。 
该行中的号码应当采用“$t<NUM>”格式，其中“NUM”是一个号码。


-bash-4.0$ dbx -a 458960
正在等待连接至进程 458960...
已成功连接至 java。
警告: 无法确定包含 java 的目录。
应用“use”命令来初始化源路径。

输入“help”以获取帮助。
正在读符号信息 ...警告: 未使用 -g 编译源文件

stopped in _p_nsleep at 0xd0124254 ($t2)
0xd0124254 (_p_nsleep+0x10) 80410014 lwz r2,0x14(r1)
(dbx) thread
thread state-k wchan state-u k-tid mode held scope function
$t1 wait 0xf100070f1000a240 blocked 663785 k no sys _event_sleep
>$t2 run running 843823 k no sys _p_nsleep
$t3 wait 0xf100070f1000b940 blocked 757787 k no sys _event_sleep
$t4 wait 0xf100070f1000e140 blocked 921761 k no sys _event_sleep
$t5 wait 0xf100070f10014a40 blocked 1351853 k no sys _event_sleep
$t6 wait 0xf100070f1000ec40 blocked 966767 k no sys _event_sleep
$t7 wait 0xf100070f1000df40 blocked 913639 k no sys _event_sleep
$t8 wait 0xf100070f10015440 blocked 1392711 k no sys _event_sleep
$t9 wait 0xf100070f10002e40 blocked 188655 k no sys _event_sleep
$t10 wait 0xf100070f1000fd40 blocked 1036539 k no sys _event_sleep
$t11 wait 0xf100070f10015e40 blocked 1433711 k no sys _event_sleep
$t12 wait 0xf100070f10009f40 blocked 651471 k no sys _event_sleep
$t13 wait 0xf100070f10013e40 blocked 1302777 k no sys _event_sleep
$t14 wait 0xf100070f1000c940 blocked 823453 k no sys _event_sleep
$t15 wait 0xf100070f10007040 blocked 458909 k no sys _event_sleep
$t16 wait 0xf100070f1000b340 blocked 733415 k no sys _event_sleep
$t17 wait 0xf100070f1000ca40 blocked 827635 k no sys _event_sleep
$t18 wait 0xf100070f1000d240 blocked 860185 k no sys _event_sleep
$t19 wait 0xf100070f10014840 blocked 1343695 k no sys _event_sleep
$t20 wait 0xf100070f10015840 blocked 1409063 k no sys _event_sleep
$t21 wait 0xf100070f1000ad40 blocked 708691 k no sys _event_sleep
[b][color=red]$t22 run running 1073241 [/color][/b]u no sys __ieee754_exp
$t23 wait 0xf100070f10014e40 blocked 1368219 k no sys _event_sleep
$t24 wait 0xf100070f10010840 blocked 1081519 k no sys _event_sleep
$t25 wait 0xf100070f1000f340 blocked 995523 k no sys _event_sleep
$t26 wait 0xf100070f1000a340 blocked 667763 k no sys _event_sleep
$t27 wait 0xf100070f1000a440 blocked 671917 k no sys _event_sleep
$t28 wait 0xf100070f1000d140 blocked 856157 k no sys _event_sleep
$t29 wait 0xf100070f1000b840 blocked 753773 k no sys _event_sleep
$t30 wait 0xf100070f10014f40 blocked 1372413 k no sys _event_sleep
$t31 wait 0xf100070f1000e540 blocked 937999 k no sys _event_sleep
$t32 wait 0xf100070f1000dd40 blocked 905271 k no sys _event_sleep
$t33 wait 0xf100070f1000d640 blocked 876683 k no sys _event_sleep
$t34 wait 0xf100070f1000e740 blocked 946407 k no sys _event_sleep
$t35 wait 0xf100070f1000f440 blocked 999661 k no sys _event_sleep
$t36 wait 0xf100070f1000cb40 blocked 831717 k no sys _event_sleep
$t39 wait 0xf100070f1000cf40 blocked 847939 k no sys _event_sleep
$t38 wait 0xf100070f10010a40 blocked 1089721 k no sys _event_sleep
$t40 wait 0xf100070f10011740 blocked 1142941 k no sys _event_sleep
$t41 wait 0xf100070f10010240 blocked 1056917 k no sys _event_sleep
$t42 wait 0xf100070f10011840 blocked 1147049 k no sys _event_sleep
$t43 wait 0xf1000200020ba208 running 1417347 k no sys closableAccept
$t44 wait 0x4159cfa0 blocked 471059 k no sys _mutex_lock
$t45 wait 0x4159cfa0 blocked 917655 k no sys _mutex_lock
$t46 wait 0xf100070f10012340 blocked 1192135 k no sys _event_sleep
$t47 wait 0xf100070f1000cd40 blocked 839881 k no sys _event_sleep
$t48 wait 0xf100070f10010440 blocked 1065187 k no sys _event_sleep
$t49 wait 0xf100070f10006940 blocked 430329 k no sys _event_sleep
$t50 wait 0xf100070f10009b40 blocked 635059 k no sys _event_sleep
(dbx)

6. 在 dbx 中时，运行 dbx 命令 th info <TID>（此 TID 来自上一步骤，该步骤在 $t<NUM>后面列出号码）以获取关于该线程的信息。


(dbx) th info [color=red][b]22[/b][/color]
thread state-k wchan state-u k-tid mode held scope function
$t22 run running 1073241 u no sys __ieee754_exp

general:
pthread addr = 0x43bb4cf0 size = 0x290
vp addr = 0x43bb7800 size = 0x2d8
thread errno = 4
start pc = 0xf17abfc0
joinable = no
pthread_t = [color=red][b]1516[/b][/color]
scheduler:
kernel =
user = 1 (other)
nice = 60
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x43b34000 size = 0x80000
limit = 0x43bb4b98
sp = 0x43bb2d38
(dbx)

7、在“general”下查找“pthread_t”，并记录该十六进制号码。
8. 非常重要说明：在 dbx 提示符下，您需要在完成操作时在 dbx 命令行键入“detach”，否则，如果您在连接到进程时只要一退出，dbx 将终止该进程！
9. 记下“p_thread_t”输出中的十六进制值，并在 Thread Dump 中搜索其中哪个线程的“native ID”等于该值。 
这将为您揭示造成高 CPU 占用率问题的线程。

在JAVACORE可以对应到native ID为[color=red][b]151[/b][/color][color=red][b]6[/b][/color]的线程
ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" (TID:0x300CB938, sys_thread_t:0x43B312A8, state:R, native ID:0x1516) prio=5
4XESTACKTRACE at java.lang.StrictMath.exp(Native Method)
4XESTACKTRACE at java.lang.Math.exp(Math.java(Compiled Code))
4XESTACKTRACE at tdy218.alg.DeadLoop.service(DeadLoop.java(Compiled Code))
4XESTACKTRACE at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
4XESTACKTRACE at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java(Compiled Code))
4XESTACKTRACE at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java(Compiled Code))
4XESTACKTRACE at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java(Compiled Code))
4XESTACKTRACE at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7047)
4XESTACKTRACE at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java(Compiled Code))
4XESTACKTRACE at weblogic.security.service.SecurityManager.runAs(SecurityManager.java(Compiled Code))
4XESTACKTRACE at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
4XESTACKTRACE at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
4XESTACKTRACE at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java(Compiled Code))