登录用户的UCS Manager，发现Chassis 2第6个槽位的风扇报警，指示灯不亮，将该槽位的风扇与其它正常槽位的风扇互换，该风扇又工作正常了，可是原来好用的风扇插在第6个槽位后，仍然报警，指示灯不亮。初步判断应该是刀箱存在问题。

于是向思科开了一个case，思科TAC工程师远程连到6248上，通过命令查看风扇确实工作异常，信息如下：

magic:                0x486f7403        # OK
valid:                1
pid:                1554
interval:        15                # seconds
write_ts:        1451028436        # Fri Dec 25 07:27:16 2015
stale_ts:        1451028458        # Fri Dec 25 07:27:38 2015 OK
now:                1451028443        # Fri Dec 25 07:27:23 2015
status:                1                # ACTIVE
policy_state:        1                # COOL
xreading:        1                # DEVELOPER_MODE: FALSE
hwconf_valid:        1
maxfans:                8
fan[1].fault/read/req:        0/30/30        # OK
fan[2].fault/read/req:        0/30/30        # OK
fan[3].fault/read/req:        0/30/30        # OK
fan[4].fault/read/req:        0/30/30        # OK
fan[5].fault/read/req:        0/30/30        # OK
fan[6].fault/read/req:        1/0/30        # MISSING
fan[7].fault/read/req:        0/30/30        # OK
fan[8].fault/read/req:        0/30/30        # OK

通过上面的信息可以看到第6个槽位的风扇缺失。

在阅读日志等相关信息后，思科TAC工程师怀疑问题出现在UCS5108的内部总线上，于是给出解决建议如下：

第一步：

Remove PSU1 let sit for 2 minutes replace, wait 10 secondsconfirm PSU1 has power, Move to PSU2

Remove PSU2 let sit for 2 minutes replace, wait 10 seconds haspower, Move to PSU3

Remove PSU3 let sit for 2 minutes replace, wait 10 seconds PSU3has power, Move to PSU4

Remove PSU4 let sit for 2 minutes replace, wait 10 seconds PSU4has power, Move to Fan1

 第二步：

Basically, the power supply reseat did not have business impact,you could do it now.

If the issue still exist, please take the action below in amaintenance window:

Remove Fan1 let sit for 30 seconds replace, wait 10 secondsconfirm Fan1 has power, Move to Fan2

Remove Fan2 let sit for 30 seconds replace, wait 10 secondsconfirm Fan2 has power, Move to Fan3

Remove Fan3 let sit for 30 seconds replace, wait 10 secondsconfirm Fan3 has power, Move to Fan4

Remove Fan4 let sit for 30 seconds replace, wait 10 secondsconfirm Fan4 has power, Move to Fan5

Remove Fan5 let sit for 30 seconds replace, wait 10 secondsconfirm Fan5 has power, Move to Fan6

Remove Fan6 let sit for 30 seconds replace, wait 10 secondsconfirm Fan6 has power, Move to Fan7

Remove Fan7 let sit for 30 seconds replace, wait 10 seconds confirmFan7 has power, Move to Fan8

Remove Fan8 let sit for 30 seconds replace, wait 10 secondsconfirm Fan8 has power

第三步：

Remove right IOM,  let sit for 5 minutes replace, confirmthat IO MOD is UP and Running before you reseat left IOM

Once right IOM is Up and Running finally reseat left IOM let sitfor 5 minutes, and place it back into the chassis.

最后一步：

If all the above did not fix the issue, then you need to power-cycle thewhole chassis 2.

当我执行到第三步的时候，风扇上面的***报警框已经消失了，但是通过命令查看风扇仍然工作不正常，于是执行了最后一步，当刀箱重启完成之后通过命令查看风扇工作正常。

magic:                0x486f7403        # OK
valid:                1
pid:                1511
interval:        15                # seconds
write_ts:        1451289000        # Mon Dec 28 07:50:00 2015
stale_ts:        1451289022        # Mon Dec 28 07:50:22 2015 OK
now:                1451289010        # Mon Dec 28 07:50:10 2015
status:                1                # ACTIVE
policy_state:        1                # COOL
xreading:        1                # DEVELOPER_MODE: FALSE
hwconf_valid:        1
maxfans:                8
fan[1].fault/read/req:        0/30/30        # OK
fan[2].fault/read/req:        0/30/30        # OK
fan[3].fault/read/req:        0/30/30        # OK
fan[4].fault/read/req:        0/30/30        # OK
fan[5].fault/read/req:        0/30/30        # OK
fan[6].fault/read/req:        0/30/30        # OK
fan[7].fault/read/req:        0/30/30        # OK
fan[8].fault/read/req:        0/30/30        # OK


至此故障解决。

*************************************************************************************

后记：虽然替换法可以快速定位故障，但是有时也并不一定准确，问题不一定就是出现在硬件上，也有可能是软件出了问题。

感谢思科TAC赵工！

*************************************************************************************