网卡bond和信息查看 EDACDIMMCEError错误导致服务器重启

https://blog.csdn.net/weixin_45720992?spm=1001.2101.3001.5343

EDACDIMMCEError错误导致服务器重启
现象：

最近几天一个华为RH2285服务器一直不定时自动重启，基本每天一两次，查看系统日志报下面的错误，每秒记录一条错误日志

OS：OEL 6.5

$ more /var/log/message

Jul 21 08:54:32 customerkernel: EDAC MC1: 5486 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:33 customerkernel: EDAC MC1: 11480 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:34 customerkernel: EDAC MC1: 11330 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:35 customerkernel: EDAC MC1: 6584 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:36 customerkernel: EDAC MC1: 27428 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:37 customerkernel: EDAC MC1: 30113 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:38 customerkernel: EDAC MC1: 4453 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:39 customerkernel: EDAC MC1: 6269 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:40 customer kernel:EDAC MC1: 15720 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1 page:0x0offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:41 customerkernel: EDAC MC1: 16107 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

分析解决：

这个是EDAC (Error Detection AndCorrection) 的日志.

CE Error 是 Correctable Error 的简称，另外还有 UE(Uncorrectable Error)

按照上面的文档, 找出错误的DIMM:

[root@customer log]# grep “[0-9]” /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count

/sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow0/ch3_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow0/ch4_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow1/ch3_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow1/ch4_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow0/ch0_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow0/ch3_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow0/ch4_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow1/ch0_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow1/ch3_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow1/ch4_ce_count:554836518

count：不为0的行即代表存在内存错误。
mc:第几个CPU。
csrow：内存通道。
ch：通道内的第几根内存。*

查到是 /mc1/csrow1/ch4, 根据结构图:

Channel 0 Channel 1

===================================

csrow0 | DIMM_A0 | DIMM_B0 |

csrow1 | DIMM_A0 | DIMM_B0 |

===================================

===================================

csrow2 | DIMM_A1 | DIMM_B1 |

csrow3 | DIMM_A1 | DIMM_B1 |

===================================

然后通过dmidecode查看：

[root@customer log]# dmidecode -t memory |grep ‘Locator: DIMM’

Locator: DIMM_D0

Locator: DIMM_D1

Locator: DIMM_E0

Locator: DIMM_E1

Locator: DIMM_F0

Locator: DIMM_F1

Locator: DIMM_A0

Locator: DIMM_A1

Locator: DIMM_B0

Locator: DIMM_B1

Locator: DIMM_C0

Locator: DIMM_C1

通过服务器控制台查看内存：

结合报错日志：kernel: EDAC MC1: 16107 CE error on CPU#1Channel#2_DIMM#1 (channel:2slot:1

应该是内存插槽DIMM_F1的问题。

解决：

最后我们要做的就是，把有问题的F1插槽上的内存拔出来或是更换到其它的内存插槽上面，之后系统启动后不再报错。

参考：

http://blog.tankywoo.com/2014/12/02/edac-dimm-ce-error.html

http://serverfault.com/questions/648240/how-can-i-find-which-memory-have-ce-error

http://www.taodudu.cc/news/show-3468474.html?action=onClick