阿里云服务器cpu连续n天使用率为100%问题解决方案！

硬件配置:
阿里云服务器(CPU:4核 内存:4GB 数据盘:450G 带宽:5Mbps).




系统配置: 
iluckysi@ILUCKYSI-PC:/etc# cat issue
Ubuntu 12.04.5 LTS \n \l




异常信息:
cpu连续n天使用率为100%.




查找原因:
查看系统cpu和memory使用情况.
iluckysi@ILUCKYSI-PC:/etc# top
top - 11:27:43 up 303 days,  3:31,  2 users,  load average: 5.06, 5.03, 5.05
Tasks: 180 total,   6 running, 174 sleeping,   0 stopped,   0 zombie
Cpu(s): 69.8%us, 30.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   4051260k total,  3592692k used,   458568k free,   227332k buffers
Swap:  2187260k total,    70940k used,  2116320k free,  2351156k cached


  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                     
19608 daemon    20   0 29948 2768 1060 R   97  0.1 160766:24 perl                                                                                                                         
26598 daemon    20   0 23740 1964  708 R   85  0.0 168071:12 perl                                                                                                                         
 4020 daemon    20   0   832    8    0 R   80  0.0 237454:42 pdflush                                                                                                                      
19612 daemon    20   0 29948 2524  864 R   72  0.1 177923:31 perl                                                                                                                         
31348 daemon    20   0 23740  728  400 R   62  0.0 167897:47 perl                                                                                                                         
25444 root      20   0 3321m 550m  12m S    2 13.9  45:24.95 java                                                                                                                         
11739 daemon    20   0 30216  344  192 S    1  0.0  24:40.73 perl                                                                                                                         
然后按c,查看COMMAND对应的实际进程.
iluckysi@ILUCKYSI-PC:/etc# top
top - 11:29:20 up 303 days,  3:32,  2 users,  load average: 5.01, 5.02, 5.05
Tasks: 185 total,   6 running, 179 sleeping,   0 stopped,   0 zombie
Cpu(s): 65.3%us, 34.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.2%si,  0.0%st
Mem:   4051260k total,  3594584k used,   456676k free,   227332k buffers
Swap:  2187260k total,    70940k used,  2116320k free,  2351272k cached


  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                     
 4020 daemon    20   0   832    8    0 R   99  0.0 237456:05 [pdflush]                                                                                                                    
31348 daemon    20   0 23740  728  400 R   78  0.0 167898:59 klogd -x                                                                                                                     
19608 daemon    20   0 29948 2768 1060 R   76  0.1 160767:42 /usr/sbin/sshd -D                                                                                                            
19612 daemon    20   0 29948 2524  864 R   74  0.1 177924:48 /usr/sbin/acpid                                                                                                              
26598 daemon    20   0 23740 1964  708 R   68  0.0 168072:28 klogd -x                                                                                                                     
25444 root      20   0 3321m 550m  12m S    2 13.9  45:25.68 /usr/lib/jdk/jdk1.7.0_71/jre/bin/java -Djava.util.logging.config.file=/usr/lib/tomcat/apache-tomcat-7.0.32/conf/logging.prope
  398 root      20   0  679m  12m 8224 S    1  0.3 405:19.89 /usr/local/aegis/aegis_client/aegis_00_73/AliYunDun                                                                          
12731 root      20   0  882m 9000 6284 S    1  0.2 124:16.60 /usr/local/aegis/alihids/AliHids  
更多关于top指令的使用方法,请上网查询......




异常分析:
查看cpu使用率过高的这几个进程关联的操作:ls -al:查看某个进程打开的文件的权限.
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/4020/fd/
total 0
dr-x------ 2 daemon daemon  0 Jun 12 16:45 .
dr-xr-xr-x 7 daemon daemon  0 May  1 09:56 ..
lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[68038509]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> /dev/null (deleted)
l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> /dev/null (deleted)
lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> /var/tmp/.fontUnix (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/31348/fd/
total 0
dr-x------ 2 daemon daemon  0 Jun 12 16:45 .
dr-xr-xr-x 7 daemon daemon  0 Jan 23 16:02 ..
lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[634666134]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> pipe:[634666138]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 Jun 12 16:45 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 Jun 12 16:45 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> pipe:[634666141]
lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> socket:[637677664]
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/19608/fd/
total 0
dr-x------ 2 daemon daemon  0 May  6 00:44 .
dr-xr-xr-x 7 daemon daemon  0 May  1 09:56 ..
lr-x------ 1 daemon daemon 64 May  6 00:44 0 -> pipe:[1849120182]
l-wx------ 1 daemon daemon 64 May  6 00:44 1 -> pipe:[1849120186]
l-wx------ 1 daemon daemon 64 May  6 00:44 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 May  6 00:44 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 May  6 00:44 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 May  6 00:44 2 -> pipe:[1849120188]
lrwx------ 1 daemon daemon 64 May  6 00:44 3 -> socket:[1335075729]
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/19612/fd/
total 0
dr-x------ 2 daemon daemon  0 May  6 00:44 .
dr-xr-xr-x 7 daemon daemon  0 May  1 09:56 ..
lr-x------ 1 daemon daemon 64 May  6 00:44 0 -> pipe:[1849120182]
l-wx------ 1 daemon daemon 64 May  6 00:44 1 -> pipe:[1849120186]
l-wx------ 1 daemon daemon 64 May  6 00:44 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 May  6 00:44 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 May  6 00:44 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 May  6 00:44 2 -> pipe:[1849120188]
lrwx------ 1 daemon daemon 64 May  6 00:44 3 -> socket:[2163495078]
iluckysi@ILUCKYSI-PC:/var/tmp# ls -al /proc/26598/fd/
total 0
dr-x------ 2 daemon daemon  0 Jun 12 16:45 .
dr-xr-xr-x 7 daemon daemon  0 Jan 23 16:02 ..
lr-x------ 1 daemon daemon 64 Jun 12 16:45 0 -> pipe:[3453697476]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 1 -> pipe:[3453697477]
l-wx------ 1 daemon daemon 64 Jun 12 16:45 11 -> /opt/httpd-2.2.21/logs/mod_jk.log
lrwx------ 1 daemon daemon 64 Jun 12 16:45 12 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
lrwx------ 1 daemon daemon 64 Jun 12 16:45 13 -> /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
l-wx------ 1 daemon daemon 64 Jun 12 16:45 2 -> pipe:[3453697478]
lrwx------ 1 daemon daemon 64 Jun 12 16:45 3 -> socket:[525581618]
iluckysi@ILUCKYSI-PC:/var/tmp# 
同时可以使用如下指令进行分析: lsof -p pid:查看正在运行中的进程打开了哪些文件,目录和套接字,是系统监测工具之一.
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 4020
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
pdflush 4020 daemon  cwd    DIR    3,1    12288  1835016 /usr/sbin
pdflush 4020 daemon  rtd    DIR    3,1     4096        2 /
pdflush 4020 daemon  txt    REG    3,1   562008   264410 /var/tmp/pdflush (deleted)
pdflush 4020 daemon    0r  FIFO    0,8      0t0 68038509 pipe
pdflush 4020 daemon    1w   CHR    1,3      0t0     4782 /dev/null (deleted)
pdflush 4020 daemon    2w   CHR    1,3      0t0     4782 /dev/null (deleted)
pdflush 4020 daemon    3u   REG    3,1        0   263762 /var/tmp/.fontUnix (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 31348
COMMAND   PID   USER   FD   TYPE    DEVICE  SIZE/OFF      NODE NAME
perl    31348 daemon  cwd    DIR       3,1      4096   2097153 /tmp
perl    31348 daemon  rtd    DIR       3,1      4096         2 /
perl    31348 daemon  txt    REG       3,1     10352   1841631 /usr/bin/perl
perl    31348 daemon  mem    REG       3,1     26968   1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl    31348 daemon  mem    REG       3,1     22840   1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl    31348 daemon  mem    REG       3,1     43288   1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl    31348 daemon  mem    REG       3,1   1811128   1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl    31348 daemon  mem    REG       3,1    135366   1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl    31348 daemon  mem    REG       3,1   1030512   1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl    31348 daemon  mem    REG       3,1     14768   1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl    31348 daemon  mem    REG       3,1   1479112   1841632 /usr/lib/libperl.so.5.10.1
perl    31348 daemon  mem    REG       3,1    149280   1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl    31348 daemon    0r  FIFO       0,8       0t0 634666134 pipe
perl    31348 daemon    1w  FIFO       0,8       0t0 634666138 pipe
perl    31348 daemon    2w  FIFO       0,8       0t0 634666141 pipe


perl    31348 daemon    3u  IPv4 637677664       0t0       TCP 110.76.39.140:44833->209.92.176.14:http (ESTABLISHED)
perl    31348 daemon   11w   REG       3,1 138468379    661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl    31348 daemon   12u   REG       3,1       448    661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl    31348 daemon   13u   REG       3,1         1    688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 19608
COMMAND   PID   USER   FD   TYPE     DEVICE  SIZE/OFF       NODE NAME
perl    19608 daemon  cwd    DIR        3,1      4096    2097153 /tmp
perl    19608 daemon  rtd    DIR        3,1      4096          2 /
perl    19608 daemon  txt    REG        3,1     10352    1841631 /usr/bin/perl
perl    19608 daemon  mem    REG        3,1    105288    1314564 /lib/x86_64-linux-gnu/libresolv-2.15.so
perl    19608 daemon  mem    REG        3,1     31104    1314576 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
perl    19608 daemon  mem    REG        3,1     52120    1314561 /lib/x86_64-linux-gnu/libnss_files-2.15.so
perl    19608 daemon  mem    REG        3,1     26968    1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl    19608 daemon  mem    REG        3,1     22840    1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl    19608 daemon  mem    REG        3,1     43288    1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl    19608 daemon  mem    REG        3,1   1811128    1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl    19608 daemon  mem    REG        3,1    135366    1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl    19608 daemon  mem    REG        3,1   1030512    1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl    19608 daemon  mem    REG        3,1     14768    1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl    19608 daemon  mem    REG        3,1   1479112    1841632 /usr/lib/libperl.so.5.10.1
perl    19608 daemon  mem    REG        3,1    149280    1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl    19608 daemon    0r  FIFO        0,8       0t0 1849120182 pipe
perl    19608 daemon    1w  FIFO        0,8       0t0 1849120186 pipe
perl    19608 daemon    2w  FIFO        0,8       0t0 1849120188 pipe
perl    19608 daemon    3u  IPv4 1335075729       0t0        TCP 110.76.39.140:53110->119.68.205.1:smtp (ESTABLISHED)
perl    19608 daemon   11w   REG        3,1 138471419     661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl    19608 daemon   12u   REG        3,1       448     661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl    19608 daemon   13u   REG        3,1         1     688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 19612
COMMAND   PID   USER   FD   TYPE     DEVICE  SIZE/OFF       NODE NAME
perl    19612 daemon  cwd    DIR        3,1      4096    2097153 /tmp
perl    19612 daemon  rtd    DIR        3,1      4096          2 /
perl    19612 daemon  txt    REG        3,1     10352    1841631 /usr/bin/perl
perl    19612 daemon  mem    REG        3,1    105288    1314564 /lib/x86_64-linux-gnu/libresolv-2.15.so
perl    19612 daemon  mem    REG        3,1     31104    1314576 /lib/x86_64-linux-gnu/libnss_dns-2.15.so
perl    19612 daemon  mem    REG        3,1     52120    1314561 /lib/x86_64-linux-gnu/libnss_files-2.15.so
perl    19612 daemon  mem    REG        3,1     26968    1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl    19612 daemon  mem    REG        3,1     22840    1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl    19612 daemon  mem    REG        3,1     43288    1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl    19612 daemon  mem    REG        3,1   1811128    1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl    19612 daemon  mem    REG        3,1    135366    1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl    19612 daemon  mem    REG        3,1   1030512    1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl    19612 daemon  mem    REG        3,1     14768    1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl    19612 daemon  mem    REG        3,1   1479112    1841632 /usr/lib/libperl.so.5.10.1
perl    19612 daemon  mem    REG        3,1    149280    1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl    19612 daemon    0r  FIFO        0,8       0t0 1849120182 pipe
perl    19612 daemon    1w  FIFO        0,8       0t0 1849120186 pipe
perl    19612 daemon    2w  FIFO        0,8       0t0 1849120188 pipe
perl    19612 daemon    3u  IPv4 2163495078       0t0        TCP 110.76.39.140:43416->119.68.205.1:smtp (ESTABLISHED)
perl    19612 daemon   11w   REG        3,1 138471723     661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl    19612 daemon   12u   REG        3,1       448     661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl    19612 daemon   13u   REG        3,1         1     688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# lsof -p 26598
COMMAND   PID   USER   FD   TYPE    DEVICE  SIZE/OFF       NODE NAME
perl    26598 daemon  cwd    DIR       3,1      4096    2097153 /tmp
perl    26598 daemon  rtd    DIR       3,1      4096          2 /
perl    26598 daemon  txt    REG       3,1     10352    1841631 /usr/bin/perl
perl    26598 daemon  mem    REG       3,1     26968    1841662 /usr/lib/perl/5.10.1/auto/Socket/Socket.so
perl    26598 daemon  mem    REG       3,1     22840    1841660 /usr/lib/perl/5.10.1/auto/IO/IO.so
perl    26598 daemon  mem    REG       3,1     43288    1314543 /lib/x86_64-linux-gnu/libcrypt-2.15.so
perl    26598 daemon  mem    REG       3,1   1811128    1314563 /lib/x86_64-linux-gnu/libc-2.15.so
perl    26598 daemon  mem    REG       3,1    135366    1314539 /lib/x86_64-linux-gnu/libpthread-2.15.so
perl    26598 daemon  mem    REG       3,1   1030512    1314566 /lib/x86_64-linux-gnu/libm-2.15.so
perl    26598 daemon  mem    REG       3,1     14768    1314577 /lib/x86_64-linux-gnu/libdl-2.15.so
perl    26598 daemon  mem    REG       3,1   1479112    1841632 /usr/lib/libperl.so.5.10.1
perl    26598 daemon  mem    REG       3,1    149280    1310795 /lib/x86_64-linux-gnu/ld-2.15.so
perl    26598 daemon    0r  FIFO       0,8       0t0 3453697476 pipe
perl    26598 daemon    1w  FIFO       0,8       0t0 3453697477 pipe
perl    26598 daemon    2w  FIFO       0,8       0t0 3453697478 pipe
perl    26598 daemon    3u  IPv4 525581618       0t0        TCP 110.76.39.140:34176->210.253.114.69:81 (ESTABLISHED)
perl    26598 daemon   11w   REG       3,1 138472179     661852 /opt/httpd-2.2.21/logs/mod_jk.log
perl    26598 daemon   12u   REG       3,1       448     661951 /opt/httpd-2.2.21/logs/mod_jk.shm.30138 (deleted)
perl    26598 daemon   13u   REG       3,1         1     688338 /opt/httpd-2.2.21/logs/mod_jk.shm.30138.lock (deleted)
iluckysi@ILUCKYSI-PC:/var/tmp# 
由上面的分析得知,除了4020进程,其余进程都和httpd有关系.
并且除了4020进程,其余所有的进程都对外有一个TCP连接,查看连接的ip,都是来自国外.




深入分析:
在上面的输出中,我们看到了.fontUnix,查询apache的error.log,看到如下记录.
[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] Connecting to 85.236.52.116:80... 
[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] connected.
[Sat Nov 22 22:31:30 2014] [error] [client 166.78.138.102] HTTP request sent, awaiting response... 
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 200 OK
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] Length: 1018 [text/plain]
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] Saving to: `/var/tmp/.font-unix.sh'
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102]      0K                                                       100%  199M=0s
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 
[Sat Nov 22 22:31:33 2014] [error] [client 166.78.138.102] 2014-11-22 22:31:33 (199 MB/s) - `/var/tmp/.font-unix.sh' saved [1018/1018]\




解决方法:
将上面分析出的的有问题的进程强制杀死.
iluckysi@ILUCKYSI-PC:~# kill -9 4020
iluckysi@ILUCKYSI-PC:~# kill -9 31348
iluckysi@ILUCKYSI-PC:~# kill -9 26598
iluckysi@ILUCKYSI-PC:~# kill -9 19612
iluckysi@ILUCKYSI-PC:/tmp# kill -9 19608
此时到阿里云平台查看服务器cpu使用率,发现cpu使用率降下来了.




最终方案:
阿里云Linux Bash严重漏洞修复紧急通知:http://bbs.aliyun.com/read/176977.html
升级系统版本,升级bash版本.