思科 6509交换机 - High CPU - 原因排查

1、查看cpu利用率

vss-6513#show processes cpu | e 0.00
CPU utilization for five seconds: 85%/49%; one minute: 69%; five minutes: 68%
PID 5Sec 1Min 5Min Process
1 0.0% 0.3% 0.3% kernel
3 0.0% 0.0% 0.0% devc-pty
4 0.0% 0.0% 0.0% devc-mistral.proc
5 0.0% 0.0% 0.0% pipe
4102 0.0% 0.0% 0.0% dumper.proc
4103 0.0% 0.0% 0.0% pcmcia_driver.proc
4104 0.0% 0.0% 0.0% bflash_driver.proc
12297 0.0% 0.0% 0.0% mqueue
12298 0.0% 0.0% 0.0% flashfs_hes.proc
12299 0.0% 0.0% 0.0% dfs_bootdisk.proc
12300 0.0% 0.0% 0.0% ldcache.proc
12301 0.0% 0.0% 0.0% watchdog.proc
12302 0.0% 0.0% 0.0% syslogd.proc
12303 0.0% 0.0% 0.0% name_svr.proc
12304 0.1% 0.0% 0.0% wdsysmon.proc
12305 0.0% 0.0% 0.0% sysmgr.proc
16386 0.0% 0.0% 0.0% chkptd.proc
16402 0.0% 0.0% 0.0% sysmgr.proc
16403 0.0% 0.0% 0.0% syslog_dev.proc
16404 0.0% 0.0% 0.0% itrace_exec.proc
16405 0.0% 0.0% 0.0% packet.proc
16406 78.0% 60.5% 59.9% ios-base
16407 0.0% 0.0% 0.0% fh_fd_rf.proc
16408 0.0% 0.0% 0.0% fh_fd_rpc.proc
16409 0.0% 0.0% 0.0% fh_fd_oir.proc
16410 0.0% 0.0% 0.0% fh_fd_env.proc
16411 0.0% 0.0% 0.0% fh_fd_cli.proc
16412 0.0% 0.0% 0.0% fh_metric_dir.proc
16413 0.0% 0.0% 0.0% fh_fd_test.proc
16414 0.0% 0.0% 0.0% fh_fd_snmp.proc
16415 0.0% 0.0% 0.0% fh_fd_none.proc
16416 0.0% 0.0% 0.0% fh_fd_intf.proc
16417 0.0% 0.0% 0.0% fh_fd_track.proc
16418 0.0% 0.0% 0.0% fh_fd_timer.proc
16419 0.0% 0.0% 0.0% fh_fd_ioswd.proc
16420 0.0% 0.0% 0.0% fh_fd_counter.proc
16421 0.0% 0.0% 0.0% fh_fd_gold.proc
16422 0.0% 0.0% 0.0% fh_server.proc
16423 0.0% 0.0% 0.0% fh_policy_dir.proc
16424 0.0% 0.0% 0.0% call_home.proc
16425 0.0% 0.0% 0.0% ipfs_daemon.proc
16426 0.0% 0.0% 0.0% fh_fd_config.proc
16427 1.2% 1.3% 1.3% raw_ip.proc
16428 0.0% 0.0% 0.0% inetd.proc
16429 0.5% 0.4% 0.4% tcp.proc
16430 0.1% 0.2% 0.1% udp.proc
PID 5Sec 1Min 5Min Process
16431 4.8% 6.3% 6.0% iprouting.iosproc
16432 0.5% 0.4% 0.4% cdp2.iosproc

2、查看cpu某一个进程利用率

vss-6513#show processes cpu detailed 16406
CPU utilization for five seconds: 84%/45%; one minute: 71%; five minutes: 69%
PID/TID 5Sec 1Min 5Min Process Prio STATE CPU
16406 68.6% 59.6% 59.0% ios-base 1h29m
0.0% 0.0% 0.1% [dead threads]
1 2.2% 2.1% 1.7% 10 Receive 63.667
2 1.0% 1.3% 1.3% 5 Ready 3m17s
3 3.9% 1.6% 1.9% 10 Receive 89.528
4 2.3% 3.0% 2.4% 10 Receive 3m31s
5 0.0% 0.0% 0.0% 11 Nanosleep 0.449
6 33.4% 32.3% 32.9% 21 Intr 44m04s
7 5.8% 3.4% 3.3% 22 Intr 6m13s
8 1.1% 1.1% 1.1% 23 Intr 2m02s
9 0.0% 0.0% 0.0% 25 Intr 0.000
10 0.0% 0.0% 0.0% 10 Reply 0.623
11 0.2% 0.2% 0.2% 10 Receive 26.414
12 0.0% 0.0% 0.0% 10 Condvar 0.108
13 0.0% 0.0% 0.0% 20 Sigwaitin 0.000
14 2.4% 1.6% 2.0% 10 Receive 3m14s
15 1.0% 1.6% 1.7% 10 Receive 3m13s
16 7.1% 3.1% 2.6% 10 Reply 3m41s
17 2.6% 2.5% 2.3% 10 Receive 19.136
18 2.5% 2.7% 2.6% 10 Receive 3m46s
21 1.1% 0.8% 0.7% 10 Reply 11.918
22 2.0% 2.3% 2.2% 10 Receive 2m45s

。。。

3、交换引擎cpu利用率

vss-6513#remote login switch

vss-6513-sp#show processes cpu | e 0.00
CPU utilization for five seconds: 72%/43%; one minute: 33%; five minutes: 29%
PID 5Sec 1Min 5Min Process
1 10.3% 1.9% 1.6% kernel
3 0.0% 0.0% 0.0% devc-pty
4 0.0% 0.0% 0.0% devc-mistral.proc
5 0.0% 0.0% 0.0% pipe
4102 0.0% 0.0% 0.0% dumper.proc
4103 0.0% 0.0% 0.0% bflash_driver.proc
8200 0.9% 1.0% 1.0% pcmcia_driver.proc
8201 0.0% 0.0% 0.0% mqueue
8203 0.0% 0.0% 0.0% dfs_disk1.proc
8204 0.0% 0.0% 0.0% dfs_disk0.proc
8205 0.1% 0.1% 0.1% dfs_bootdisk.proc
8206 0.0% 0.0% 0.0% ldcache.proc
8207 0.0% 0.0% 0.0% watchdog.proc
8208 0.0% 0.0% 0.0% syslogd.proc
8209 0.0% 0.0% 0.0% name_svr.proc
8210 0.0% 0.0% 0.0% wdsysmon.proc
8211 0.0% 0.0% 0.0% sysmgr.proc
12290 0.0% 0.0% 0.0% chkptd.proc
12308 0.0% 0.0% 0.0% sysmgr.proc
12309 0.0% 0.0% 0.0% itrace_exec.proc
12310 0.0% 0.0% 0.0% packet.proc
PID 5Sec 1Min 5Min Process
12311 63.0% 29.7% 26.4% ios-base
12312 0.0% 0.0% 0.0% tcp.proc
12313 0.0% 0.0% 0.0% udp.proc

4、运行IOS版本

vss-6513#show version | in boot
System image file is "sup-bootdisk:s72033-adventerprisek9-vz.122-33.SXI3.bin"

5、故障诊断

1.This should be a SW bug.

A somewhat similar bug is the following:

CSCsy61956 Bug Details

Crash in ios-base when running 'show ip route' or 'show bgp' commands

Symptom:

A Cat6K running modular IOS version 12.2(33)SXI may crash when running 'sh ip route vrf ..' or 'show bgp ..' commands.

Conditions:

Must be running modular IOS. Not seen in non-modular

Workaround:

None

详见：http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsy61956

 

2，另外原因：死路由原因，排查路由

6、解决办法

升级s72033-adventerprisek9_wan-mz.122-33.SXI4a.bin版本后，CPU利用率恢复正常。