linux cpu占有率居高不下 调试

今天调试程序，使用top命令后，发现程序的cpu占有率很高，一直在99，这很可怕，所以来调试。

使用top命令，得如下结果

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1997 root 20 0 358m 71m 3208 S 99.1 7.2 81:53.50 test
1 root 20 0 24332 2044 1176 S 0.0 0.2 0:01.36 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:46.51 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:38.53 kworker/0:0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0

可知test的PID为1997

然后具体查看test里线程的cpu使用情况

使用 top -H -p 1997 命令

root@slk:~# top -H -p 1997

top - 17:19:47 up 15 days, 34 min, 4 users, load average: 1.02, 1.06, 1.06
Tasks: 8 total, 1 running, 7 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.3%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.3%st
Mem: 1017924k total, 887500k used, 130424k free, 85928k buffers
Swap: 0k total, 0k used, 0k free, 351280k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2005 root 20 0 358m 71m 3208 R 99.3 7.2 83:44.47 test
2001 root 20 0 358m 71m 3208 S 0.3 7.2 0:08.66 test 
2004 root 20 0 358m 71m 3208 S 0.3 7.2 0:19.61 test 
1997 root 20 0 358m 71m 3208 S 0.0 7.2 0:26.24 test 
1999 root 20 0 358m 71m 3208 S 0.0 7.2 0:00.06 test 
2000 root 20 0 358m 71m 3208 S 0.0 7.2 0:00.00 test
2002 root 20 0 358m 71m 3208 S 0.0 7.2 0:05.86 Server Listen(2
2003 root 20 0 358m 71m 3208 S 0.0 7.2 0:00.00 Server Accept(2

可得线程中CPU占有率最高的线程的PID是2005

使用gdb icdn 2005 命令

root@slk:~# gdb icdn 2005
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
icdn: No such file or directory.
Attaching to process 2005

warning: process 2005 is a cloned process
Reading symbols from /home/slk/test/snmptrapd...done.
Reading symbols from /usr/lib/libnetsnmp.so.30...done.
Loaded symbols for /usr/lib/libnetsnmp.so.30
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
Reading symbols from /usr/local/lib/libstd.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libstd.so
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /usr/local/lib/libudt.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libudt.so
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
execute_search () at test.c:310

从最后一行可知，线程2005运行的是execute_search这个函数，然后查看该函数，发现有这样一段

 

while(g_search_state == 1)
{
　　tmp = g_data_list->phead->next;

　　if (tmp)
　　{
　　　　pthread_mutex_lock(&g_mysql_mutex);
　　　　res.result = search_mysql(tmp->sn, &(res.machine));
　　　　pthread_mutex_unlock(&g_mysql_mutex);

　　　　snprintf(res.sn, sizeof(res.sn), "%s", tmp->sn);
　　}
}

由于tmp长期为NULL，所以这变成了一个while的死循环，而死循环极为占用内存，于是再下面加上usleep(50000)，问题解决