进程分析工具 pstack pstree strace

1. 当进程卡住不动或者死锁时，pstack可以把当前进程的代码栈打出来，方便我们排查。
   用法非常简单，后面直接加进程号即可。如果是多线程的，则会打印每个线程的堆栈信息。
   man pstack 可查看帮助

[root@localhost ~]# pstack 7383
Thread 8 (Thread 0x7fcc0429c700 (LWP 7384)):
#0  0x00007fcc0d322a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcc08f5f299 in cnd_timedwait (cond=<optimized out>, mtx=<optimized out>, ts=<optimized out>) at tinycthread.c:462
#2  0x00007fcc08f5f67d in cnd_timedwait_abs (cnd=cnd@entry=0x25570b8, mtx=mtx@entry=0x2557090, tspec=tspec@entry=0x7fcc04299df0) at tinycthread_extra.c:100
#3  0x00007fcc08f2852e in rd_kafka_q_serve (rkq=0x2557090, timeout_ms=<optimized out>, max_cnt=max_cnt@entry=0, cb_type=cb_type@entry=RD_KAFKA_Q_CB_CALLBACK, callback=callback@entry=0x0, opaque=opaque@entry=0x0) at rdkafka_queue.c:462
#4  0x00007fcc08ef5dbc in rd_kafka_thread_main (arg=arg@entry=0x2556390) at rdkafka.c:1836
#5  0x00007fcc08f5f0f7 in _thrd_wrapper_function (aArg=<optimized out>) at tinycthread.c:576
#6  0x00007fcc0d31edc5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007fcc0a3e221d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fcc03a9b700 (LWP 7385)):
#0  0x00007fcc0d322a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcc08f5f299 in cnd_timedwait (cond=<optimized out>, mtx=<optimized out>, ts=<optimized out>) at tinycthread.c:462
#2  0x00007fcc08f5f67d in cnd_timedwait_abs (cnd=cnd@entry=0x25580d8, mtx=mtx@entry=0x25580b0, tspec=tspec@entry=0x7fcc03a98ad0) at tinycthread_extra.c:100
#3  0x00007fcc08f2814c in rd_kafka_q_pop_serve (rkq=0x25580b0, timeout_ms=<optimized out>, version=version@entry=0, cb_type=cb_type@entry=RD_KAFKA_Q_CB_RETURN, callback=callback@entry=0x0, opaque=opaque@entry=0x0) at rdkafka_queue.c:390
#4  0x00007fcc08f28220 in rd_kafka_q_pop (rkq=<optimized out>, timeout_ms=<optimized out>, version=version@entry=0) at rdkafka_queue.c:416
...　太多了，省略

２.　查看进程树
pstree 常用参数也是后面加pid

[root@nice ~]# ps -ef|grep Demo
root      30731  10338  2 12:38 pts/0    00:00:00 ./Demo
root      30781  30696  0 12:38 pts/1    00:00:00 grep --color=auto Demo
[root@nice ~]# pstree -p 30731 
Demo(30731)─┬─{Demo}(30733)
               ├─{Demo}(30734)
               ├─{Demo}(30735)
               ├─{Demo}(30736)
               ├─{Demo}(30737)
               ├─{Demo}(30738)

3. strace 的功能比较强大
   一般这么用-o指定输出文件，-T 统计系统函数耗时，–tt 时间戳（微妙）

strace -o output.txt -T -tt -e trace=all -p 28979

内容如下

21:58:56.981775 read(18, "\26\3\3\0A", 5) = 5 <0.003039>
21:58:56.985280 read(18, "\2\0\0=\3\3\346\351\20\300\201\305D7K}.\2107\"\237\375\266\241\301q\r\267\231R\337Z"..., 65) = 65 <0.000121>
21:58:56.985642 read(18, "\26\3\3\6\"", 5) = 5 <0.000131>
21:58:56.985953 read(18, "\v\0\6\36\0\6\33\0\3\v0\202\3\0070\202\1\357\240\3\2\1\2\2\1\0020\r\6\t*\206"..., 1570) = 1570 <0.000131>
21:58:56.986544 stat("/usr/local/mysql/ssl/certs/e86656db.0", 0x7ffcdee82ef0) = -1 ENOENT (没有那个文件或目录) <0.000148>
21:58:56.987298 stat("/usr/local/mysql/ssl/certs/e86656db.0", 0x7ffcdee82ef0) = -1 ENOENT (没有那个文件或目录) <0.000127>
21:58:56.987674 stat("/usr/local/mysql/ssl/certs/e86656db.0", 0x7ffcdee82ef0) = -1 ENOENT (没有那个文件或目录) <0.000147>
21:58:56.988023 read(18, "\26\3\3\1,", 5) = 5 <0.000134>
21:58:56.988368 read(18, "\f\0\1(\3\0\35 \261o\0R\226\177yN<\16v\305!\25\213\30\266\364\24^\301\341\255\""..., 300) = 300 <0.000134>
21:58:56.989207 read(18, "\26\3\3\0:", 5) = 5 <0.000108>
21:58:56.989555 read(18, "\r\0\0006\3\1\2@\0.\4\3\5\3\6\3\10\7\10\10\10\t\10\n\10\v\10\4\10\5\10\6"..., 58) = 58 <0.000131>
21:58:56.989889 read(18, "\26\3\3\0\4", 5) = 5 <0.000060>

以上三个工具结合起来可以分析进程在什么地方耗时过多，进而有目的的优化程序。更多详细的使用方法，可以参考man手册。