linux查看线程 挂起,查找线程死锁或挂起的原因

分享一个之前整理的查找线程死锁或挂起的原因；

注：服务器环境 linux ,用于C/C++编写的进程，JAVA原理类似。

常见由线程挂起导致的现象

程序处理速度由慢到严重超时，最后全部超时，重启程序会循环这一现象，那90%是线程被挂起了。

常见的线程挂起或死锁有

线程锁里面出现死循环，锁不能被释放，导致其它线程一直等待；

锁里加锁，即双重锁；

多线程编程里，共享资源没有加线程锁，造成多线程共同强夺资源而挂起。

判断进程是否挂起

使用pstree命令查看某进程的线程数：pstree -p |grep [进程名]。

例如下：

yuejctest:[/yuejc]pstree -p |grep Ywdeal

|-Ywdeal(8969)-+-{Ywdeal}(9013) [线程1]

| `-{Ywdeal}(9016) [线程2]

如果在次执行此函数，发现线程数一直在增加(程序中有限制，达到限制时不在增加也不减)说明线程无法释放，可能被挂起。

什么是pstack

此命令可显示每个进程的栈跟踪，使用 pstack 来确定进程挂起的位置。此命令的唯一选项是‘要检查进程的 PID’。

pstack pid,你会得到很多信息:

例如下：

yuejctest:[/yuejc]pstack 8969

Thread 3 (Thread 0x42e88940 (LWP 9013)):

#0 0x00000039e329a0b1 in nanosleep () from /lib64/libc.so.6

#1 0x00000039e3299f99 in sleep () from /lib64/libc.so.6

#2 0x0000000000406dc9 in pthread_mdb_keepconnect ()

#3 0x00000039e3e064a7 in start_thread () from /lib64/libpthread.so.0

#4 0x00000039e32d3c2d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x44e89940 (LWP 9016)):

#0 0x00000039e329a0b1 in nanosleep () from /lib64/libc.so.6

#1 0x00000039e3299f99 in sleep () from /lib64/libc.so.6

#2 0x0000000000406b49 in pthread_db_keepconnect ()

#3 0x00000039e3e064a7 in start_thread () from /lib64/libpthread.so.0

#4 0x00000039e32d3c2d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2b2518adcdd0 (LWP 8969)):

#0 0x00000039e32d4f52 in msgrcv () from /lib64/libc.so.6

#1 0x000000000045eab4 in msgRcv ()

#2 0x00000000004056d0 in main ()

现实中遇到的问题，帮你如何从pstack信息中找到挂起原因：

yuejcapp2:[/yuejc]pstack 23677

Thread 12 (Thread 0x43d8b940 (LWP 23686)):

#0 0x00000032ec00d91b in read () from /lib64/libpthread.so.0

#1 0x00000000004a3735 in _NetReadSocket ()

#2 0x00000000004a3d1e in _dci_recv_msg ()

#3 0x0000000000493ca9 in _dci_query_buf ()

#4 0x0000000000494365 in _dci_send_query ()

#5 0x0000000000494daf in si_dci_query_p ()

#6 0x000000000049388a in dci_query_p ()

#7 0x000000000041df0d in mdb_stream::excuteSql() ()

#8 0x000000000041f283 in mdb_stream::open(mdb_connect&, char const*, int, int) ()

#9 0x0000000000406bfa in pthread_mdb_keepconnect(void*) ()

#10 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0

#11 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x45d8c940 (LWP 23687)):

#0 0x00000032eb49a1a1 in nanosleep () from /lib64/libc.so.6

#1 0x00000032eb49a089 in sleep () from /lib64/libc.so.6

#2 0x0000000000406b3d in pthread_db_keepconnect(void*) ()

#3 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0

#4 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x47d8d940 (LWP 19145)):

#0 0x00000032ec00d91b in read () from /lib64/libpthread.so.0

#1 0x00000000004a3735 in _NetReadSocket ()

#2 0x00000000004a3d1e in _dci_recv_msg ()

#3 0x0000000000493ca9 in _dci_query_buf ()

#4 0x0000000000494365 in _dci_send_query ()

#5 0x0000000000494daf in si_dci_query_p ()

#6 0x000000000049388a in dci_query_p ()

#7 0x000000000041df0d in mdb_stream::excuteSql() ()

#8 0x000000000048ee6b in mdb_stream::operator<

#9 0x000000000046c345 in mdb_select_userinfo_W ()

#10 0x0000000000426d2c in GetUserInfo(char*, _USER_INFO*) ()

#11 0x0000000000443005 in UserAbilityDeal(int&) ()

#12 0x000000000044f48f in ServiceOPenNewAdd(_SERVICE_OPEN_REQ&) ()

#13 0x0000000000408160 in pthread_service_open(void*) ()

#14 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0

#15 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x49d8e940 (LWP 19375)):

#0 0x00000032ec00d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0

#1 0x00000032ec008e1a in _L_lock_1034 () from /lib64/libpthread.so.0

#2 0x00000032ec008cdc in pthread_mutex_lock () from /lib64/libpthread.so.0

#3 0x00000000004304e9 in GetOrderInfo(char*, int, _ORDER_QUERY_INFO*) ()

#4 0x0000000000441007 in GetServiceOpenMoreInfo(_SERVICE_OPEN_REQ&) ()

#5 0x0000000000407ce6 in pthread_service_open(void*) ()

#6 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0

#7 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x4dd90940 (LWP 19717)):

#0 0x00000032ec00d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0

#1 0x00000032ec008e1a in _L_lock_1034 () from /lib64/libpthread.so.0

#2 0x00000032ec008cdc in pthread_mutex_lock () from /lib64/libpthread.so.0

#3 0x00000000004463ed in ServiceOpenChangePayType(_SERVICE_OPEN_REQ&) ()

#4 0x0000000000408444 in pthread_service_open(void*) ()

#5 0x00000032ec00673d in start_thread () from /lib64/libpthread.so.0

#6 0x00000032eb4d3d1d in clone () from /lib64/libc.so.6

yuejcapp2:[/yuejc/log]

对以上线程信息的分析，#0表示最底层的那个函数正在处理：

<1>.线程 Thread 12 正在read ()资源，

线程 Thread 11 在nanosleep ()暂停某个线程，

线程 Thread 10 正在read ()资源，

线程 Thread 9 在__lll_lock_wait ()对资源加锁等待，

线程 Thread 8 在__lll_lock_wait ()对资源加锁等待，

<2>.根据对以上线程的分析结果，检查Thread 11 是守护进程，人为正常暂停，且此线程锁正常。

而Thread 9和Thread 8等待加锁锁定资源，是正常的等待。是什么原因让这两个线程一直等待呢？

在看一下Thread 12和Thread 10两个线程同时在read资源，造成了资源强夺现象而被挂起。

<3>.根据以上分析，检查Thread 12和Thread 10信息中的pthread_mdb_keepconnect()函数中的mdb_stream::open()函数和mdb_select_userinfo_W()函数。

发现线程Thread 12提示的mdb_stream::open()函数，在代码中没有加线程锁，增加线程锁后，程序运行正常，挂起现象解决。