Linux服务器EDAC CE memory read error

之前在大数据集群中，有一台服务器的CPU占用总是莫名其妙飙高，就算执行简单任务也会耗费很长时间，且reboot不能解决问题。检查了各种可能的问题之后，最终在查看dmesg命令的设备信息时，发现大量如下的日志：

......
[1180532.573917] EDAC MC1: 128 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x2b67afa offset:0xb40 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573925] EDAC MC1: 147 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x30b6e9a offset:0x540 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573930] EDAC MC1: 236 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x236e7bb offset:0xd00 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.573935] EDAC MC1: 189 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x365669d offset:0xf40 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573940] EDAC MC1: 255 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x3d2953a offset:0x540 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573944] EDAC MC1: 177 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x212b93c offset:0x540 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573949] EDAC MC1: 136 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x31d099d offset:0xb40 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573953] EDAC MC1: 88 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x228e5be offset:0x100 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.573958] EDAC MC1: 111 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x34e5e7b offset:0xf00 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.573962] EDAC MC1: 149 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x314bb9f offset:0xd00 grain:32 syndrome:0x0 -  area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.611035] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180532.611045] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180533.460534] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180533.460544] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180533.547047] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
......

EDAC即error detection and correction（错误检测与纠正），是Linux系统内部的机制。在上面的日志中，可以清楚地看出是内存读错误。MC即memory controller（内存控制器）。CE则代表correctable error，是ECC内存中可以纠正的错误，相对地还有UE（uncorrectable error）。
为了摸清是哪些内存出了问题，找出所有内存的CE计数：

~ grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count
/sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch1_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch2_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch3_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch1_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch2_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch3_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch2_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch3_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch0_ce_count:198276530
/sys/devices/system/edac/mc/mc1/csrow1/ch1_ce_count:233266714
/sys/devices/system/edac/mc/mc1/csrow1/ch2_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch3_ce_count:0

可见是mc1/csrow1/ch0和mc1/csrow1/ch1两个通道上的内存出了问题。mc、csrow（chip-select row）与ch之间的关系类似下图：

mc0        ch0       ch1
==============================
csrow0 | DIMM_A0 | DIMM_B0 |
csrow1 | DIMM_A0 | DIMM_B0 |
==============================
mc1        ch0       ch1
==============================
csrow2 | DIMM_A1 | DIMM_B1 |
csrow3 | DIMM_A1 | DIMM_B1 |
==============================

通过dmidecode -t memory命令，可以查看每个DIMM（也就是常说的内存条）对应的位置。然后根据映射关系，去服务器上将出现问题的两根内存条更换，问题就解决了。