mysql 脑裂的问题,DRBD脑裂问题故障处理

环境：Mysql+heartbeat+drbd 架构

问题：mysql-主 宕机 mysql-从接替之后，再恢复Mysql-主之后，发现主的drbd启动不了，而且从上面也无法辨识对方，从的状态是Primary/Unknown ，主的状态是Secondary/Unknown

查看从节点的日志发现脑裂：[root@node2 ~]# tail -n 12 /var/log/messagesSep 21 11:19:53 lab4 kernel: block drbd0: Split-Brain detected but unresolved, dropping connection!Sep 21 11:19:53 lab4 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0Sep 21 11:19:53 lab4 kernel: block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)Sep 21 11:19:53 lab4 kernel: block drbd0: conn( WFReportParams -> Disconnecting )Sep 21 11:19:53 lab4 kernel: block drbd0: error receiving ReportState, l: 4!Sep 21 11:19:53 lab4 kernel: block drbd0: meta connection shut down by peer.Sep 21 11:19:53 lab4 kernel: block drbd0: asender terminatedSep 21 11:19:53 lab4 kernel: block drbd0: Terminating drbd0_asenderSep 21 11:19:53 lab4 kernel: block drbd0: Connection closedSep 21 11:19:53 lab4 kernel: block drbd0: conn( Disconnecting -> StandAlone )Sep 21 11:19:53 lab4 kernel: block drbd0: receiver terminatedSep 21 11:19:53 lab4 kernel: block drbd0: Terminating drbd0_receiver

此时的环境是从节点的heartbeat、mysql启动，顶替主的工作，从节点的drbd状态是Primary/Unknown，并且已经挂载着  /dev/drbd0  /data  ，主节点的状态是未启动heartbeat和mysql 并且启动drbd的时候报错。

恢复过程：

将主节点设置为从节点并丢弃资源数据

[root@node1 ~]# drbdadm secondary r0

[root@node1 ~]# drbdadm -- --discard-my-data connect r0

在从节点上手动连接资源

[root@node2 ~]# drbdadm connect r0

最后查看节点状态，连接已恢复正常[root@node1 ~]# cat /proc/drbdversion: 8.3.15 (api:88/proto:86-97)GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by mockbuild@builder17.centos.org, 2013-03-27 16:04:080: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----ns:36 nr:0 dw:24 dr:185 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

此时再启动主节点的heartbeat 即可！