gdb定位死锁问题

       本案例借着gdb调试死锁的问题，演示在多线程场景下如何使用gdb调试多线程死锁的

调试过程：

1.为了重现死锁现象，自己写了个死锁demo

[root@localhost bin]# ./deadLock 
thread_routine_two:lock mutex two
thread_routine_one:lock mutex one
thread_routine_one:lock mutex two
thread_routine_two:lock mutex one

2. 使用pstack和gdb命令配合来定位问题具体出在哪里

[root@localhost ~]# pstack
Usage: pstack <process-id>
[root@localhost ~]# ps -ef|grep deadLock |grep -v grep 
root     14759 10973  0 10:06 pts/0    00:00:00 ./deadLock
[root@localhost ~]# pstack 14759        #这里可以看到进程有一个主线程加两个子线程，判断是否卡死可以多次执行，如果有线程堆栈一直没变化，基本可以确定是该线程卡死
Thread 3 (Thread 0x7f0e309c0700 (LWP 14760)):
#0  0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f0e30d97dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f0e30d97c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400940 in thread_routine_one (arg=0x0) at /opt/testgdb/src/deadLock.c:19
#4  0x00007f0e30d95dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0e30abeead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0e301bf700 (LWP 14761)):
#0  0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f0e30d97dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f0e30d97c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004009cb in thread_routine_two (arg=0x0) at /opt/testgdb/src/deadLock.c:42
#4  0x00007f0e30d95dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0e30abeead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f0e311b3740 (LWP 14759)):
#0  0x00007f0e30d96f47 in pthread_join () from /lib64/libpthread.so.0
#1  0x0000000000400a85 in main () at /opt/testgdb/src/deadLock.c:69
[root@localhost ~]#
[root@localhost bin]# gdb deadLock 14759      #gdb调试正在运行的进程
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/testgdb/bin/deadLock...done.
Attaching to program: /opt/testgdb/bin/deadLock, process 14759
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 14761]
[New LWP 14760]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f0e30d96f47 in pthread_join () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7.x86_64
(gdb) info thread                   #打印线程信息，可以看到线程 2 3都在等待加锁
  Id   Target Id         Frame 
  3    Thread 0x7f0e309c0700 (LWP 14760) "deadLock" 0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
  2    Thread 0x7f0e301bf700 (LWP 14761) "deadLock" 0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
* 1    Thread 0x7f0e311b3740 (LWP 14759) "deadLock" 0x00007f0e30d96f47 in pthread_join () from /lib64/libpthread.so.0
(gdb) 
  Id   Target Id         Frame 
  3    Thread 0x7f0e309c0700 (LWP 14760) "deadLock" 0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
  2    Thread 0x7f0e301bf700 (LWP 14761) "deadLock" 0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
* 1    Thread 0x7f0e311b3740 (LWP 14759) "deadLock" 0x00007f0e30d96f47 in pthread_join () from /lib64/libpthread.so.0
(gdb) 
  Id   Target Id         Frame 
  3    Thread 0x7f0e309c0700 (LWP 14760) "deadLock" 0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
  2    Thread 0x7f0e301bf700 (LWP 14761) "deadLock" 0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
* 1    Thread 0x7f0e311b3740 (LWP 14759) "deadLock" 0x00007f0e30d96f47 in pthread_join () from /lib64/libpthread.so.0
(gdb) thread 2                       #切换到编号为2的线程
[Switching to thread 2 (Thread 0x7f0e301bf700 (LWP 14761))]
#0  0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt                            #打印线程堆栈
#0  0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f0e30d97dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f0e30d97c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000004009cb in thread_routine_two (arg=0x0) at /opt/testgdb/src/deadLock.c:42
#4  0x00007f0e30d95dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0e30abeead in clone () from /lib64/libc.so.6
(gdb) f 3                         #打印堆栈信息编号为3的位置的代码
#3  0x00000000004009cb in thread_routine_two (arg=0x0) at /opt/testgdb/src/deadLock.c:42
42	    pthread_mutex_lock(&gMutex1);
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f0e309c0700 (LWP 14760))]
#0  0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f0e30d9c4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f0e30d97dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f0e30d97c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x0000000000400940 in thread_routine_one (arg=0x0) at /opt/testgdb/src/deadLock.c:19
#4  0x00007f0e30d95dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0e30abeead in clone () from /lib64/libc.so.6
(gdb) f 3                 #在这里查看了线程2 3的堆栈信息再结合代码，很轻松就判断出来是死锁的问题
#3  0x0000000000400940 in thread_routine_one (arg=0x0) at /opt/testgdb/src/deadLock.c:19
19	    pthread_mutex_lock(&gMutex2);
(gdb) quit
A debugging session is active.

	Inferior 1 [process 14759] will be detached.

Quit anyway? (y or n) y
Detaching from program: /opt/testgdb/bin/deadLock, process 14759
[root@localhost bin]#