EDAC DIMM CE Error错误导致服务器重启

 现象：

 

最近几天一个华为RH2285服务器一直不定时自动重启，基本每天一两次，查看系统日志报下面的错误，每秒记录一条错误日志

 

OS：OEL 6.5

 

$ more /var/log/message

Jul 21 08:54:32 customerkernel: EDAC MC1: 5486 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:33 customerkernel: EDAC MC1: 11480 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:34 customerkernel: EDAC MC1: 11330 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:35 customerkernel: EDAC MC1: 6584 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:36 customerkernel: EDAC MC1: 27428 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:37 customerkernel: EDAC MC1: 30113 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:38 customerkernel: EDAC MC1: 4453 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:39 customerkernel: EDAC MC1: 6269 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:40 customer kernel:EDAC MC1: 15720 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1 page:0x0offset:0x0 grain:8 syndrome:0x0)

Jul 21 08:54:41 customerkernel: EDAC MC1: 16107 CE error on CPU#1Channel#2_DIMM#1 (channel:2 slot:1page:0x0 offset:0x0 grain:8 syndrome:0x0)

 

 

分析解决：

 

这个是[EDAC (Error Detection AndCorrection)](https://www.kernel.org/doc/Documentation/edac.txt) 的日志.

 

CE Error 是 Correctable Error 的简称，另外还有 UE(Uncorrectable Error)

 

按照上面的文档, 找出错误的DIMM:

[root@customer log]# grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count

/sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow0/ch1_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow0/ch2_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow1/ch1_ce_count:0

/sys/devices/system/edac/mc/mc0/csrow1/ch2_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow0/ch0_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow0/ch1_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow0/ch2_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow1/ch0_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow1/ch1_ce_count:0

/sys/devices/system/edac/mc/mc1/csrow1/ch2_ce_count:554836518

 

查到是 /mc1/csrow1/ch2, 根据结构图:

 

       Channel 0   Channel 1

===================================

csrow0 | DIMM_A0   | DIMM_B0 |

csrow1 | DIMM_A0   | DIMM_B0 |

===================================

===================================

csrow2 | DIMM_A1   | DIMM_B1 |

csrow3 | DIMM_A1   | DIMM_B1 |

===================================

 

然后通过dmidecode查看：

[root@customer log]# dmidecode -t memory |grep 'Locator: DIMM'

       Locator: DIMM_D0

       Locator: DIMM_D1

       Locator: DIMM_E0

       Locator: DIMM_E1

       Locator: DIMM_F0

       Locator: DIMM_F1

       Locator: DIMM_A0

       Locator: DIMM_A1

       Locator: DIMM_B0

       Locator: DIMM_B1

       Locator: DIMM_C0

       Locator: DIMM_C1

       

通过服务器控制台查看内存：

 

主板上内存插槽的分布：

 

结合报错日志：kernel: EDAC MC1: 16107 CE error on CPU#1Channel#2_DIMM#1 (channel:2slot:1

应该是内存插槽DIMM_F1的问题。

 

解决：

最后我们要做的就是，把有问题的F1插槽上的内存拔出来或是更换到其它的内存插槽上面，之后系统启动后不再报错。

 

参考：

http://blog.tankywoo.com/2014/12/02/edac-dimm-ce-error.html

http://serverfault.com/questions/648240/how-can-i-find-which-memory-have-ce-error