模拟私网问题导致节点宕机无法启动

最新推荐文章于 2021-10-22 14:04:33 发布
最新推荐文章于 2021-10-22 14:04:33 发布
阅读量2.6k 收藏 2
点赞数 1
分类专栏： 故障解决 文章标签： RAC 私网问题
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wanbin6470398/article/details/86493885
版权

模拟私网问题导致节点宕机无法启动

目的

本文章通过模拟私网问题，导致集群节点宕机，来进行日志分析。

# ifconfig eth1 down

分析过程

GI alert日志

<GI_HOME>/log/<节点名称>/alert<节点名称>.log

  • 节点1

    2019-01-16 13:44:16.839
[cssd(28522)]CRS-1612:Network communication with node rac2 (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.400 seconds
2019-01-16 13:44:23.855
[cssd(28522)]CRS-1611:Network communication with node rac2 (2) missing for 75% of timeout interval.  Removal of this node from cluster in 7.380 seconds
2019-01-16 13:44:28.865
[cssd(28522)]CRS-1610:Network communication with node rac2 (2) missing for 90% of timeout interval.  Removal of this node from cluster in 2.370 seconds

    与节点2 rac2的网络通信在50％的超时间隔内丢失。 在14秒内从群集中删除此节点
    与节点2 rac2的网络通信在75％的超时间隔内丢失。 在7秒内从群集中删除此节点
    与节点2 rac2的网络通信在90％的超时间隔内丢失。 在2秒内从群集中删除此节点

    2019-01-16 13:44:31.243
[cssd(28522)]CRS-1607:Node rac2 is being evicted in cluster incarnation 322157220; details at (:CSSNM00007:) in /u01/11.2.0/grid/log/rac1/cssd/ocssd.log.
2019-01-16 13:44:54.468
[ohasd(28301)]CRS-8011:reboot advisory message from host: rac2, component: mo093358, with time stamp: L-2019-01-16-13:44:53.124
[ohasd(28301)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
2019-01-16 13:45:02.236
[cssd(28522)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 .
2019-01-16 13:45:02.393
[ctssd(28691)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac1.
2019-01-16 13:45:27.304
[crsd(28825)]CRS-5504:Node down event reported for node 'rac2'.
2019-01-16 13:45:32.086
[crsd(28825)]CRS-2773:Server 'rac2' has been removed from pool 'Generic'.
2019-01-16 13:45:32.086
[crsd(28825)]CRS-2773:Server 'rac2' has been removed from pool 'ora.orcl'.
2019-01-16 13:46:37.328
[ctssd(28691)]CRS-2406:The Cluster Time Synchronization Service timed out on host rac1. Details in /u01/11.2.0/grid/log/rac1/ctssd/octssd.log.

    在集群中驱逐rac2节点，要求rac2重启，集群重新配置完成，从服务池中删除rac2节点

  • 节点2

    2019-01-16 13:44:17.512
[cssd(24201)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.400 seconds
2019-01-16 13:44:24.529
[cssd(24201)]CRS-1611:Network communication with node rac1 (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.380 seconds
2019-01-16 13:44:29.539
[cssd(24201)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.370 seconds

    与节点1 rac1的网络通信在50％的超时间隔内丢失。 在14秒内从群集中删除此节点
    与节点1 rac1的网络通信在75％的超时间隔内丢失。 在7秒内从群集中删除此节点
    与节点1 rac1的网络通信在90％的超时间隔内丢失。 在2秒内从群集中删除此节点

      2019-01-16 13:44:31.915
 [cssd(24201)]CRS-1609:This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /u01/11.2.0/grid/log/rac2/cssd/ocssd.log.
 2019-01-16 13:44:32.025
 [cssd(24201)]CRS-1608:This node was evicted by node 1, rac1; details at (:CSSNM00005:) in /u01/11.2.0/grid/log/rac2/cssd/ocssd.log.
 2019-01-16 13:47:33.459
 [ohasd(2892)]CRS-2112:The OLR service started on node rac2.
 2019-01-16 13:47:34.870
 [ohasd(2892)]CRS-8011:reboot advisory message from host: rac2, component: ag125511, with time stamp: L-2019-01-15-16:20:15.283
 [ohasd(2892)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS
 2019-01-16 13:47:35.804
 [ohasd(2892)]CRS-8011:reboot advisory message from host: rac2, component: mo093358, with time stamp: L-2019-01-16-13:44:53.124
 [ohasd(2892)]CRS-8013:reboot advisory message text: clsnomon_status: need to reboot, unexpected failure 8 received from CSS

    以上信息看出节点2无法与群集中的其他节点通信，并且正在关闭以保持群集完整性; 节点2被节点1驱逐。

os日志

/var/log/messages

  • 节点1
    Jan 16 13:45:02 rac1 kernel: [Oracle OKS] Cluster Membership change - Current  incarn 0x19
Jan 16 13:45:02 rac1 kernel: [Oracle OKS] Nodes in cluster:
Jan 16 13:45:02 rac1 kernel: [Oracle OKS]   Node 1 (IP 0xa0a0a0a) 
Jan 16 13:45:02 rac1 kernel: [Oracle OKS] Node count 1
Jan 16 13:45:02 rac1 kernel: [Oracle ADVM] Cluster reconfiguration started.
Jan 16 13:45:02 rac1 kernel: [Oracle ADVM] Cluster reconfiguration completed.
Jan 16 13:45:02 rac1 kernel: [Oracle ADVM] Cluster reconfiguration completed.
Jan 16 13:45:02 rac1 kernel: [Oracle OKS] Cluster Membership change setup complete
Jan 16 13:45:30 rac1 avahi-daemon[2647]: Registering new address record for 172.18.4.187 on eth0.
Jan 16 13:45:30 rac1 avahi-daemon[2647]: Withdrawing address record for 172.18.4.187 on eth0.
Jan 16 13:45:30 rac1 avahi-daemon[2647]: Registering new address record for 172.18.4.187 on eth0.
Jan 16 13:45:30 rac1 avahi-daemon[2647]: Registering new address record for 172.18.4.172 on eth0.
Jan 16 13:45:30 rac1 avahi-daemon[2647]: Withdrawing address record for 172.18.4.172 on eth0.
Jan 16 13:45:30 rac1 avahi-daemon[2647]: Registering new address record for 172.18.4.172 on eth0.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Withdrawing address record for fe80::a00:27ff:feb1:3373 on eth1.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Withdrawing address record for 10.10.10.10 on eth1.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Withdrawing address record for fe80::a00:27ff:fe2e:e4e on eth0.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Withdrawing address record for 172.18.4.172 on eth0.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Withdrawing address record for 172.18.4.186 on eth0.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Withdrawing address record for 172.18.4.182 on eth0.
Jan 16 13:45:31 rac1 avahi-daemon[2647]: Host name conflict, retrying with <rac1-3>
  • 节点2
    没有故障时间段信息

    根据上面信息，可以看出集群重新配置，节点2的上vip与scanip都注册到到节点1的eth0网卡上。

ocssd.log 日志

<GI_HOME>/log/<节点名称>/cssd/ocssd.log

  • 节点1

    2019-01-16 13:44:16.839: [    CSSD][1218210112]clssnmPollingThread: node rac2 (2) at 50% heartbeat fatal, removal in 14.400 seconds
2019-01-16 13:44:16.839: [    CSSD][1218210112]clssnmPollingThread: node rac2 (2) is impending reconfig, flag 394254, misstime 15600

    以上信息显示：节点1发现节点2已经在连续一段时间内丢失网络心跳。如果这种情况继续下》去，集群会在14.400s之后发生重新配置，11Gr2 misscount=30s，通过命令crsctl get css misscount可以查询。

  • 节点2

    2019-01-16 13:44:17.512: [    CSSD][1222531392]clssnmPollingThread: node rac1 (1) at 50% heartbeat fatal, removal in 14.400 seconds
2019-01-16 13:44:17.512: [    CSSD][1222531392]clssnmPollingThread: node rac1 (1) is impending reconfig, flag 132108, misstime 15600

    节点2出现了和节点1类似的消息。由于这是一个双节点集群，所以这是正常情况。当丢失网络心跳的时间达到misscount时间后，集群就需要重新配置了。

  • 节点1

    2019-01-16 13:44:16.839: [    CSSD][1218210112]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
2019-01-16 13:44:16.926: [    CSSD][1249679680]clssnmvSchedDiskThreads: DiskPingThread for voting file /dev/raw/raw1 sched delay 950 > margin 750 cur_ms 59003974 lastalive 59003024
2019-01-16 13:44:18.914: [    CSSD][1228699968]clssnmSendingThread: sending status msg to all nodes
2019-01-16 13:44:18.914: [    CSSD][1228699968]clssnmSendingThread: sent 5 status msgs to all nodes
2019-01-16 13:44:20.946: [    CSSD][1099692352]clssscSelect: cookie accept request 0xa430f48
2019-01-16 13:44:20.946: [    CSSD][1099692352]clssgmAllocProc: (0xad8a000) allocated
2019-01-16 13:44:20.946: [    CSSD][1099692352]clssgmClientConnectMsg: properties of cmProc 0xad8a000 - 1,2,3,4
2019-01-16 13:44:20.946: [    CSSD][1099692352]clssgmClientConnectMsg: Connect from con(0x5a2bd) proc(0xad8a000) pid(9116) version 11:2:1:4, properties: 1,2,3,4
2019-01-16 13:44:20.946: [    CSSD][1099692352]clssgmClientConnectMsg: msg flags 0x0000
2019-01-16 13:44:20.947: [    CSSD][1099692352]clssgmExecuteClientRequest: Node name request from client ((nil))
2019-01-16 13:44:20.950: [    CSSD][1099692352]clssgmDeadProc: proc 0xad8a000
2019-01-16 13:44:20.950: [    CSSD][1099692352]clssgmDestroyProc: cleaning up proc(0xad8a000) con(0x5a2bd) skgpid  ospid 9116 with 0 clients, refcount 0
2019-01-16 13:44:20.950: [    CSSD][1099692352]clssgmDiscEndpcl: gipcDestroy 0x5a2bd
2019-01-16 13:44:23.855: [    CSSD][1218210112]clssnmPollingThread: node rac2 (2) at 75% heartbeat fatal, removal in 7.380 seconds
2019-01-16 13:44:23.917: [    CSSD][1228699968]clssnmSendingThread: sending status msg to all nodes
2019-01-16 13:44:23.917: [    CSSD][1228699968]clssnmSendingThread: sent 5 status msgs to all nodes
2019-01-16 13:44:28.865: [    CSSD][1218210112]clssnmPollingThread: node rac2 (2) at 90% heartbeat fatal, removal in 2.370 seconds, seedhbimpd 1
2019-01-16 13:44:28.921: [    CSSD][1228699968]clssnmSendingThread: sending status msg to all nodes
2019-01-16 13:44:28.921: [    CSSD][1228699968]clssnmSendingThread: sent 5 status msgs to all nodes
2019-01-16 13:44:30.731: [    CSSD][1099692352]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 850 > margin 750 cur_ms 59017784 lastalive 59016934
2019-01-16 13:44:31.242: [    CSSD][1218210112]clssnmPollingThread: Removal started for node rac2 (2), flags 0x6040e, state 3, wt4c 0
2019-01-16 13:44:31.242: [    CSSD][1218210112]clssnmDiscHelper: rac2, node(2) connection failed, endp (0x331), probe(0x100000000), ninf->endp 0x331
2019-01-16 13:44:31.242: [    CSSD][1218210112]clssnmDiscHelper: node 2 clean up, endp (0x331), init state 5, cur state 5

    节点1认为节点2持续丢失网络心跳，决定从集群中清除节点2

  • 节点2

    2019-01-16 13:44:17.512: [    CSSD][1222531392]clssnmPollingThread: local diskTimeout set to 27000 ms, remote disk timeout set to 27000, impending reconfig status(1)
2019-01-16 13:44:18.247: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 840 > margin 750 cur_ms 20599214 lastalive 20598374
2019-01-16 13:44:19.232: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 820 > margin 750 cur_ms 20600194 lastalive 20599374
2019-01-16 13:44:20.232: [    CSSD][1138612544]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 820 > margin 750 cur_ms 20601194 lastalive 20600374
2019-01-16 13:44:21.239: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 830 > margin 750 cur_ms 20602204 lastalive 20601374
2019-01-16 13:44:22.236: [    CSSD][1233021248]clssnmSendingThread: sending status msg to all nodes
2019-01-16 13:44:22.236: [    CSSD][1233021248]clssnmSendingThread: sent 5 status msgs to all nodes
2019-01-16 13:44:22.243: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 830 > margin 750 cur_ms 20603204 lastalive 20602374
2019-01-16 13:44:23.238: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 830 > margin 750 cur_ms 20604204 lastalive 20603374
2019-01-16 13:44:24.238: [    CSSD][1138612544]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 830 > margin 750 cur_ms 20605204 lastalive 20604374
2019-01-16 13:44:24.529: [    CSSD][1222531392]clssnmPollingThread: node rac1 (1) at 75% heartbeat fatal, removal in 7.380 seconds
2019-01-16 13:44:25.237: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 830 > margin 750 cur_ms 20606204 lastalive 20605374
2019-01-16 13:44:26.246: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 840 > margin 750 cur_ms 20607214 lastalive 20606374
2019-01-16 13:44:27.230: [    CSSD][1233021248]clssnmSendingThread: sending status msg to all nodes
2019-01-16 13:44:27.230: [    CSSD][1233021248]clssnmSendingThread: sent 5 status msgs to all nodes
2019-01-16 13:44:27.232: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 820 > margin 750 cur_ms 20608194 lastalive 20607374
2019-01-16 13:44:28.231: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 820 > margin 750 cur_ms 20609194 lastalive 20608374
2019-01-16 13:44:29.236: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 820 > margin 750 cur_ms 20610194 lastalive 20609374
2019-01-16 13:44:29.539: [    CSSD][1222531392]clssnmPollingThread: node rac1 (1) at 90% heartbeat fatal, removal in 2.370 seconds, seedhbimpd 1
2019-01-16 13:44:30.241: [    CSSD][1254000960]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 820 > margin 750 cur_ms 20611204 lastalive 20610384
2019-01-16 13:44:31.229: [    CSSD][1138612544]clssnmvSchedDiskThreads: DiskPingMonitorThread sched delay 810 > margin 750 cur_ms 20612194 lastalive 20611384
2019-01-16 13:44:31.914: [    CSSD][1222531392]clssnmPollingThread: Removal started for node rac1 (1), flags 0x2040c, state 3, wt4c 0
2019-01-16 13:44:31.914: [    CSSD][1222531392]clssnmDiscHelper: rac1, node(1) connection failed, endp (0x256), probe(0x100000000), ninf->endp 0x256
2019-01-16 13:44:31.914: [    CSSD][1222531392]clssnmDiscHelper: node 1 clean up, endp (0x256), init state 5, cur state 5

    节点2同样认为节点1持续丢失网络心跳，决定从集群中清除节点1.

  • 节点1

    2019-01-16 13:44:31.242: [GIPCXCPT][1218210112]gipcInternalDissociate: obj 0xab2fe10 [0000000000000331] { gipcEndpoint : localAddr 'gipc://rac1:3f93-7c9d-5dcd-3d8d#10.10.10.10#43951', remoteAddr 'gipc://rac2:nm_raccluster#10.10.10.20#56183', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x5e89, pidPeer 0, flags 0x2616, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2019-01-16 13:44:31.242: [GIPCXCPT][1218210112]gipcDissociateF [clssnmDiscHelper : clssnm.c : 3215]: EXCEPTION[ ret gipcretFail (1) ]  failed to dissociate obj 0xab2fe10 [0000000000000331] { gipcEndpoint : localAddr 'gipc://rac1:3f93-7c9d-5dcd-3d8d#10.10.10.10#43951', remoteAddr 'gipc://rac2:nm_raccluster#10.10.10.20#56183', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x5e89, pidPeer 0, flags 0x2616, usrFlags 0x0 }, flags 0x0
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmDoSyncUpdate: Initiating sync 322157220
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssscUpdateEventValue: NMReconfigInProgress  val 1, changes 3
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmDoSyncUpdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 322157220
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmSetupAckWait: Ack message type (11) 
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmSetupAckWait: node(1) is ALIVE
2019-01-16 13:44:31.242: [    CSSD][1239189824]clssnmSendSync: syncSeqNo(322157220), indicating EXADATA fence initialization complete
2019-01-16 13:44:31.242: [    CSSD][1239189824]List of nodes that have ACKed my sync: NULL

  • 节点2

    2019-01-16 13:44:31.914: [GIPCXCPT][1222531392]gipcInternalDissociate: obj 0x4ab8b60 [0000000000000256] { gipcEndpoint : localAddr 'gipc://rac2:34a4-9438-68a9-4123#10.10.10.20#56183', remoteAddr 'gipc://10.10.10.10:3f4c-bc6d-9424-f406#10.10.10.10#43951', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x6f6a, pidPeer 0, flags 0x102616, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
2019-01-16 13:44:31.914: [GIPCXCPT][1222531392]gipcDissociateF [clssnmDiscHelper : clssnm.c : 3215]: EXCEPTION[ ret gipcretFail (1) ]  failed to dissociate obj 0x4ab8b60 [0000000000000256] { gipcEndpoint : localAddr 'gipc://rac2:34a4-9438-68a9-4123#10.10.10.20#56183', remoteAddr 'gipc://10.10.10.10:3f4c-bc6d-9424-f406#10.10.10.10#43951', numPend 5, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x6f6a, pidPeer 0, flags 0x102616, usrFlags 0x0 }, flags 0x0
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmDoSyncUpdate: Initiating sync 322157220
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssscUpdateEventValue: NMReconfigInProgress  val 1, changes 9
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmDoSyncUpdate: local disk timeout set to 27000 ms, remote disk timeout set to 27000
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 322157220
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmSetupAckWait: Ack message type (11) 
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmSetupAckWait: node(2) is ALIVE
2019-01-16 13:44:31.914: [    CSSD][1243511104]clssnmSendSync: syncSeqNo(322157220), indicating EXADATA fence initialization complete
2019-01-16 13:44:31.914: [    CSSD][1243511104]List of nodes that have ACKed my sync: NULL

    每个节点都向自己认为没问题的节点发送重新配置消息，但是对于两个节点的集群来说，不会有其他节点需要接收这个消息的。接下来，需要通过磁盘心跳信息决定每个节点的状态。

  • 节点1

    2019-01-16 13:44:31.243: [    CSSD][1239189824]clssnmCheckDskInfo: Checking disk info...
2019-01-16 13:44:31.243: [    CSSD][1239189824]clssnmCheckSplit: Node 2, rac2, is alive, DHB (1547617471, 20612054) more than disk timeout of 27000 after the last NHB (1547617441, 20582204)

  • 节点2

    2019-01-16 13:44:31.915: [    CSSD][1243511104]clssnmCheckDskInfo: Checking disk info...
2019-01-16 13:44:31.915: [    CSSD][1243511104]clssnmCheckSplit: Node 1, rac1, is alive, DHB (1547617471, 59018214) more than disk timeout of 27000 after the last NHB (1547617441, 58988964)

    看起来脑裂要发生了，因为每个节点都发现对方的状态是正常的。接下来，集群需要通过自己处理脑裂的规则来决定哪一个节点离开集群。

    集群处理脑裂规则：
    对于Oracle集群，脑裂是指集群的某些节点间的网络心跳丢失，但是节点的磁盘心跳是正常的情况。当脑裂出现后，集群会分裂成若干个子集群（Corhort）。对于这种情况的出现，集群需要进行重新配置，基本原则是：节点数多的子集群存活，如果子集群包含的节点数相同，那么包含最小编号节点的子集群存活。

  • 节点1

    2019-01-16 13:44:31.243: [    CSSD][1239189824]clssnmCheckDskInfo: My cohort: 1
2019-01-16 13:44:31.243: [    CSSD][1239189824]clssnmEvict: Start
2019-01-16 13:44:31.243: [    CSSD][1239189824](:CSSNM00007:)clssnmrEvict: Evicting node 2, rac2, from the cluster in incarnation 322157220, node birth incarnation 322157217, death incarnation 322157220, stateflags 0x64000
2019-01-16 13:44:31.243: [    CSSD][1239189824]clssnmrFenceSage: Fenced node rac2, number 2, with EXADATA, handle 1239186920
2019-01-16 13:44:31.243: [    CSSD][1239189824]clssnmSendShutdown: req to node 2, kill time 59018294

  • 节点2

    2019-01-16 13:44:31.915: [    CSSD][1243511104]clssnmCheckDskInfo: My cohort: 2
2019-01-16 13:44:31.915: [    CSSD][1243511104]clssnmCheckDskInfo: Surviving cohort: 1
2019-01-16 13:44:31.915: [    CSSD][1243511104](:CSSNM00008:)clssnmCheckDskInfo: Aborting local node to avoid splitbrain. Cohort of 1 nodes with leader 2, rac2, is smaller than cohort of 1 nodes led by node 1, rac1, based on map type 2

    节点2为避免脑裂，离开了集群，并重启了节点。从11.2.0.2开始，由于新特性rebootless restart的原因，节点不会被重启，而是集群管理软件进行重启。

参考文档

《RAC核心技术详解》

DB-Engineer
关注
  • 1
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
oracle acfsctl,模拟RAC两个节点内联网不通
weixin_39707725的博客
04-08 327
模拟RAC两个节点内联网不通环境：RHEL5.8 RAC 11.2.0.3.01：2节点集群是正常的[grid@rac1 ~]$ crs_stat -tName Type Target State Host------------------------------------------------------------ora.CRSDAT...
oracle补丁下载需要什么_Oracle 19c RAC打补丁过程避坑指南
weixin_39573781的博客
11-06 780
墨天轮原文链接：https://www.modb.pro/db/22393 导读：记录19.3 rac 打补丁过程中遇到的一些问题19.3升19.5、19.5升19.6。新装的19.3 rac 需要安装补丁，目前最新的RU是19.6，由于最新的可能不稳定，选择了次新的19.5，打第一套比较顺利，后面的几套都出现些大大小小的问题 ，记录一下。19.3存在一个比较严重的crs-6015错误，是个bug...
RAC ORA-16019 报错
上帝之手&传奇 - MartinLee
01-22 650
报错信息: [grid@dominic1 ~]$ srvctl start database -d dominic  PRCR-1079 : Failed to start resource ora.dominic.db  CRS-5017: The resource action "ora.dominic.db start" encountered the following er
模拟RAC两个节点内联网不通
wonder_2012的专栏
07-16 3743
环境：RHEL5.8 RAC 11.2.0.3.0 1：2节点集群是正常的 [grid@rac1 ~]$ crs_stat -t Name           Type           Target    State     Host         ------------------------------------------------------------ ora.
oel6.3上 oracle RAC 上节点驱逐检查过程。
DATA
04-18 1745
RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，上面跑的库是11.2.0.4 发现节点2挂掉了 1.检查节点1的alert [root@racnode1 racnode1]# pwd /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1 [root@racnode1 racnode1]# tail -1
一点通路由模拟
11-14
5. **故障排查**：模拟网络故障，如链路断开、设备宕机等，学习如何使用ping、traceroute等工具定位问题。 6. **安全设置**：模拟配置访问控制列表（ACL）以实现网络安全策略，防止未授权访问。 7. **网络优化**：...
MongoDB副本集丢失数据的测试实例教程
09-09
6. **原主节点重新加入**：当原主节点再次启动并加入副本集时，它会发现自己落后于新主节点，因此触发回滚以同步状态。 7. **数据回滚**：原主节点回滚到与新主节点相同的状态，进一步确认了数据丢失。 测试步骤...
网络游戏-视频监控网络的视频诊断和运维管理系统及方法.zip
09-20
以及制定应急预案，应对突发事件，如服务器宕机、网络攻击等。 这套方法可能包括以下几个步骤： 1. **数据采集**：通过部署在各个节点的监控设备，收集视频流的实时数据和系统运行状态信息。 2. **数据分析**：...
redis announce的分析及检测
02-28
4. **故障模拟**：模拟节点宕机，观察Sentinel的响应，验证故障检测和恢复机制是否正常工作。 5. **客户端配置**：确保客户端配置的主节点地址与Sentinel发现的主节点地址一致。 通过上述步骤，我们可以深入理解...
MongoDB学习以及集群搭建的实践全纪录
12-16
然而，这种架构存在单点故障的问题，如果主节点宕机，系统将无法自动切换到从节点，导致服务中断。为了解决这个问题，MongoDB引入了副本集（Replica Set）模式，从MongoDB 3.0开始，副本集允许在主节点故障时自动...
ologgerd进程导致数据库负载高
congmo2314的博客
05-04 574
客户一台主机load超过50，响应很慢，发现进程27941消耗比率高 。 经过metalink 查询发现这个是oracle 的一个bug,数据库版本是11...
Oracle 12.2新特性 | 基于权重的节点驱逐
Enmotech的博客
01-26 827
前言在Oracle Clusterware需要从集群中逐出特定节点或一组节点的情况时，基于服务器权重的节点驱逐可充当断裂机制，通过服务器上的负载附加信息来识别要驱逐的节点或节点组。其中所有节点被驱逐的可能性是一样的。本课程视频，点击阅读原文即可下载基于权重的集群驱逐介绍基于权重的集群驱逐（Server Weight-Based Node Eviction）是一项Oracle 12.2版本引入的一项
CRS-0184: Cannot communicate with the CRS daemon.之节点脱离集群问题解决
siyanyanyanyai的专栏
04-27 3671
1、系统环境：                         OEL6.5企业版                         Oracle11.2.0.2 RAC + ASM 2、报错信息： $ crs_stat -t                 CRS-0184: Cannot communicate with the CRS daemon.            
心跳长时间missing，导致2节点被驱逐
az65381973的博客
02-15 945
当发现一个集群其中一个节点宕机，比如+ASM2在集群里offline，immediate，要重新拉起来所要做的步骤 一、停干净当前宕机节点集群/oragrid/app/11.2.0/grid/bin/crsctl stop crs -f  二、ps -ef |grep d.bin ----查看集群是否停干净，grep d.bin是因为有很多集群资源都是这个结尾的，例如 ohasd.bin,mdns...
[Oracle 11g r2(11.2.0.4.0)]案例分析4-由gipc 进程导致的节点无法启动
a743044559的专栏
10-17 3125
环境：RHEL 5.5 + 11.2.0.3 GI， 双节点。 问题描述： OBA 发现节点2 被重新启动， 之后无法加入到集群。 分析过程： 实际上这是两个不同的问题， 首先节点2被重新启动， 之后节点2无法加入集群。这需要从集群的告警日志开始分析。 1 . 节点1的集群alert.log2014-08-26 17:23:09.846 [cssd(l3527)]CRS-1612:Netwo
Openwrt路由搭建pppoe-server操作说明
wgl307293845的博客
10-22 2376
ssh登录到路由后台 使用putty或者SercureCRT都可以 安装软件包 opkg update opkg install rp-pppoe-server 修改/etc/ppp/pppoe-server-options # PPP options for the PPPoE server # LIC: GPL require-pap require-chap auth lcp-echo-interval 10 lcp-echo-failure 10 mru 1492 m...
oracle RAC集群日志,rac心跳-日志分析
weixin_30382147的博客
04-08 865
3 ocssd进程日志Node rac102的ocssd进程CSSD]2013-02-02 05:30:04.937 [1197242688] >TRACE: clssnmPollingThread: node rac101 (1) missed(3) checkin(s)[ CSSD]2013-02-02 05:30:05.938 [1197242688] >TRACE:...
oracle rac 重启一个节点,RAC一节点重启问题
weixin_29920697的博客
04-08 1741
OS：HP-UXB.11.31ORALCE：10.2.0.4.0 - 64bit两个节点，RAC+ASM心跳线通过同一个交换机连通。DB02自动重启，从DB01的CSSD日志看是DB01发现DB02不通了，而让DB02重启的。但不知具体原因，不知怎么查找。以前试过心跳线断了，会导致DB02重启，但当时会从两节点的CRS日志看到，两节点都会抢仲裁盘的资源，而想让对它重启的日志。而现在只有DB01...
asm_diskstring设置错误集群无法启动问题处理
关光磊的博客
09-24 1813
aix 6.1，grid 11.2.0.3+asm11.2.0.3+rac 建数据库的时候使用的是aix自带的多路径软件mpio，建了diskgroup 现在改造成veritas dmp多路径，已经修改了asm的disk_strings＝/dev/vx/rdmp/*，crs/asm启动的时候已经可以识别到磁盘/dev/vx/rdmp/开头的磁盘，但是读取不回原来的diskgroup信息。 crs启...
redis中master节点宕机从节点没有成为master
最新发布
07-14
在Redis中，当主节点宕机后，从节点默认不会自动晋升为新的主节点。这是因为Redis遵循主从复制的原则，从节点的主要目的是作为主节点的备份，并在主节点宕机时提供数据的冗余备份。 如果希望从节点能够自动晋升为新的主节点，可以通过以下步骤手动进行操作： 1. 确认主节点宕机：可以通过ping命令或者查看日志文件确认主节点是否宕机。 2. 登录从节点：使用Redis命令行客户端登录从节点。 3. 执行SLAVEOF命令：在从节点的命令行中执行SLAVEOF NO ONE命令，将从节点转变为主节点。该命令告诉从节点不再依赖于任何主节点。 4. 修改配置文件：在从节点的配置文件中（redis.conf），将replicaof配置项设置为replicaof no one。这样可以确保即使从节点重启，它也不会再次成为从节点。 5. 重新启动从节点：重新启动从节点，使其以新的配置生效。 请注意，手动晋升从节点为新的主节点可能会导致数据丢失或冲突，因此在执行此操作之前，请确保已经备份了重要的数据。此外，建议在进行此操作之前查看Redis官方文档或咨询Redis专家以获取更详细的指导。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值