心跳丢失造成RAC节点驱逐

杨俊峰的博客

于 2021-10-11 15:48:04 发布

阅读量289

点赞数

文章标签：网络交换机 gnome openstack 分布式存储

本文链接：https://blog.csdn.net/yangjunfeng0604/article/details/120705971

版权

博客内容讲述了在一次迁库过程中，一套再建生产业务库遭遇主机重启的问题。通过分析alert日志和CRS中的超时告警，怀疑是由于心跳网卡丢失导致的节点脑裂，进而触发了节点驱逐。网络同事检查交换机后确认了异常情况。

摘要由CSDN通过智能技术生成

赶上了迁库风，我们的一套再建生产业务库发生了主机重启，什么原因呢

[root@nqzeyddb2 ~]# uptime

15:42:46 up 5 days, 22:29, 4 users, load average: 0.08, 0.09, 0.10

alert日志里面没有明显告警，crs里面是有明显的超时告警的，怀疑是心跳网卡丢失造成脑裂进行的节点驱逐

2021-10-09 00:47:55.743: [ CSSD][580302592]clssnmPollingThread: node nqzeyddb1 (1) at 50% heartbeat fatal, removal in 14.910 seconds

2021-10-09 00:47:55.743: [ CSSD][580302592]clssnmPollingThread: node nqzeyddb1 (1) is impending reconfig, flag 2491406, misstime 15090

2021-10-09 00:47:55.743: [ CSSD][580302592]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)

2021-10-09 00:47:55.743: [ CSSD][586610432]clssnmvDHBValidateNcopy: node 1, nqzeyddb1, has a disk HB, but no network HB, DHB has rcfg 528920094, wrtcnt, 880951, LATS 286157624, lastSeqNo 880924, uniqueness 1633579205, timestamp 1633711660/286147104

2021-10-09 00:47:55.863: [ CSSD][589764352]clssnmvDiskPing: Writing with status 0x3, timestamp 1633711675/286157744

2021-10-09 00:47:56.144: [ CSSD][594495232]clssnmvDiskPing: Writing with status 0x3, timestamp 1633711676/286158024

2021-10-09 00:47:56.306: [ CSSD][943818496]clssgmpcBuildNodeList: nodename for node 0 is NULL

2021-10-09 00:48:07.745: [ CSSD][580302592]clssnmPollingThread: node nqzeyddb1 (1) at 90% heartbeat fatal, removal in 2.910 seconds,

2021-10-09 00:48:10.656: [ CSSD][580302592]clssnmMarkNodeForRemoval: node 1, nqzeyddb1 marked for removal

后找网络同事帮忙查看一下交换机有无异常

心跳丢失造成RAC节点驱逐_超时