CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal

 

ecsrac01$vi alertecsrac01.log  

2011-03-31 12:30:37.566
[crsd(10997)]CRS-1005:The OCR upgrade was completed. Version has changed from 169870336 to 169870336. Details in /u01/product/crs/log/ecsrac01/crsd/crsd.log.
2011-03-31 12:30:37.566
[crsd(10997)]CRS-1012:The OCR service started on node ecsrac01.
2011-03-31 12:30:37.577
[evmd(10953)]CRS-1401:EVMD started on node ecsrac01.
2011-03-31 12:30:39.252
[crsd(10997)]CRS-1201:CRSD started on node ecsrac01.
2011-03-31 12:30:39.424
[crsd(10997)]CRS-1205:Auto-start failed for the CRS resource . Details in ecsrac01.
2011-03-31 12:31:01.572
[crsd(10997)]CRS-1205:Auto-start failed for the CRS resource . Details in ecsrac01.
2011-03-31 12:31:01.576
[crsd(10997)]CRS-1205:Auto-start failed for the CRS resource . Details in ecsrac01.
2011-03-31 12:31:01.582
[crsd(10997)]CRS-1205:Auto-start failed for the CRS resource . Details in ecsrac01.
[cssd(11568)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ecsrac01 ecsrac02 .
2011-05-11 14:16:14.968
[cssd(11568)]CRS-1612:node ecsrac02 (2) at 50% heartbeat fatal, eviction in 29.210 seconds
2011-05-11 14:16:29.967
[cssd(11568)]CRS-1611:node ecsrac02 (2) at 75% heartbeat fatal, eviction in 14.210 seconds
2011-05-11 14:16:38.975
[cssd(11568)]CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal, eviction in 5.200 seconds
2011-05-11 14:16:39.968
[cssd(11568)]CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal, eviction in 4.210 seconds
2011-05-11 14:16:40.970
[cssd(11568)]CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal, eviction in 3.210 seconds
2011-05-11 14:16:41.972
[cssd(11568)]CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal, eviction in 2.210 seconds
2011-05-11 14:16:42.974
[cssd(11568)]CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal, eviction in 1.200 seconds
2011-05-11 14:16:43.965
[cssd(11568)]CRS-1610:node ecsrac02 (2) at 90% heartbeat fatal, eviction in 0.210 seconds
2011-05-11 14:17:12.232
[cssd(11568)]CRS-1607:CSSD evicting node ecsrac02. Details in /u01/product/crs/log/ecsrac01/cssd/ocssd.log.
[cssd(11568)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ecsrac01 .
2011-05-11 14:17:16.302
[crsd(10997)]CRS-1204:Recovering CRS resources for node ecsrac02.
[cssd(11568)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ecsrac01 ecsrac02 .
~

ecsrac01vi   /u01/product/crs/log/ecsrac01/cssd/ocssd.log  

[    CSSD]2011-03-31 12:32:13.246 [1189128544] >TRACE:   clssgmInitialRecv: conns done (2/2)
[    CSSD]2011-03-31 12:32:13.247 [1231087968] >TRACE:   clssgmEstablishMasterNode: MASTER for 2 is node(1) birth(1)
[    CSSD]2011-03-31 12:32:13.247 [1231087968] >TRACE:   clssgmMasterCMSync: Synchronizing group/lock status
[    CSSD]2011-03-31 12:32:13.248 [1231087968] >TRACE:   clssgmMasterSendDBDone: group/lock status synchronization complete
[    CSSD]CLSS-3000: reconfiguration successful, incarnation 2 with 2 nodes

[    CSSD]CLSS-3001: local node number 1, master node number 1

[    CSSD]2011-03-31 12:32:13.288 [1231087968] >TRACE:   clssgmReconfigThread:  completed for reconfig(2), with status(1)
[    CSSD]2011-05-11 14:16:14.968 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 50 3.118329e-317artbeat fatal, eviction in 29.210 seconds
[    CSSD]2011-05-11 14:16:14.968 [1199618400] >TRACE:   clssnmPollingThread: node ecsrac02 (2) is impending reconfig, flag 1037, misstime 30790
[    CSSD]2011-05-11 14:16:14.968 [1199618400] >TRACE:   clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[    CSSD]2011-05-11 14:16:29.967 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 75 3.118353e-317artbeat fatal, eviction in 14.210 seconds
[    CSSD]2011-05-11 14:16:38.975 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 90 3.118376e-317artbeat fatal, eviction in 5.200 seconds
[    CSSD]2011-05-11 14:16:39.968 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 90 3.118400e-317artbeat fatal, eviction in 4.210 seconds
[    CSSD]2011-05-11 14:16:40.970 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 90 3.118424e-317artbeat fatal, eviction in 3.210 seconds
[    CSSD]2011-05-11 14:16:41.972 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 90 3.118447e-317artbeat fatal, eviction in 2.210 seconds
[    CSSD]2011-05-11 14:16:42.974 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 90 3.118471e-317artbeat fatal, eviction in 1.200 seconds
[    CSSD]2011-05-11 14:16:43.965 [1199618400] >WARNING: clssnmPollingThread: node ecsrac02 (2) at 90 3.118495e-317artbeat fatal, eviction in 0.210 seconds
[    CSSD]2011-05-11 14:16:44.178 [1199618400] >TRACE:   clssnmPollingThread: Eviction started for node ecsrac02 (2), flags 0x040d, state 3, wt4c 0
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmDoSyncUpdate: Initiating sync 3
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmDoSyncUpdate: diskTimeout set to (57000)ms
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmSetupAckWait: Ack message type (11)
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmSetupAckWait: node(1) is ALIVE
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmSendSync: syncSeqNo(3)
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmWaitForAcks: Ack message type(11), ackCount(1)
[    CSSD]2011-05-11 14:16:44.178 [1147169120] >TRACE:   clssnmHandleSync: diskTimeout set to (57000)ms
[    CSSD]2011-05-11 14:16:44.178 [1147169120] >TRACE:   clssnmHandleSync: Acknowledging sync: src[1] srcName[ecsrac01] seq[9] sync[3]
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmWaitForAcks: done, msg type(11)
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmDoSyncUpdate: Terminating node 2, ecsrac02, misstime(60000) state(5)
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmSetupAckWait: Ack message type (13)
[    CSSD]2011-05-11 14:16:44.178 [1220598112] >TRACE:   clssnmSetupAckWait: node(1) is ACTIVE
[    CSSD]2011-05-11 14:16:44.178 [2538650368] >USER:    NMEVENT_SUSPEND [00][00][00][06]
[    CSSD]2011-05-11 14:16:44.179 [1220598112] >TRACE:   clssnmWaitForAcks: Ack message type(13), ackCount(1)
[    CSSD]2011-05-11 14:16:44.179 [1147169120] >TRACE:   clssnmSendVoteInfo: node(1) syncSeqNo(3)
[    CSSD]2011-05-11 14:16:44.179 [1220598112] >TRACE:   clssnmWaitForAcks: done, msg type(13)
[    CSSD]2011-05-11 14:16:44.179 [1220598112] >TRACE:   clssnmCheckDskInfo: Checking disk info...
[    CSSD]2011-05-11 14:16:44.179 [1220598112] >TRACE:   clssnmCheckDskInfo: Node 2, ecsrac02, potentially alive, min sleep for node is 4130, current sleep is 52870
[    CSSD]2011-05-11 14:16:45.180 [1220598112] >TRACE:   clssnmCheckDskInfo: Node 2, ecsrac02, potentially alive, min sleep for node is 4130, current sleep is 52870
[    CSSD]2011-05-11 14:16:46.182 [1220598112] >TRACE:   clssnmCheckDskInfo: Node 2, ecsrac02, potentially alive, min sleep for node is 4130, current sleep is 52870
[    CSSD]2011-05-11 14:17:09.226 [1220598112] >TRACE:   clssnmCheckDskInfo: Node 2, ecsrac02, potentially alive, min sleep for node is 4130, current sleep is 52870
[    CSSD]2011-05-11 14:17:10.228 [1220598112] >TRACE:   clssnmCheckDskInfo: Node 2, ecsrac02, potentially alive, min sleep for node is 4130, current sleep is 52870
[    CSSD]2011-05-11 14:17:11.229 [1220598112] >TRACE:   clssnmCheckDskInfo: Node 2, ecsrac02, potentially alive, min sleep for node is 4130, current sleep is 52870
[    CSSD]2011-05-11 14:17:12.231 [1220598112] >TRACE:   clssnmEvict: Start
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmEvict: Evicting node 2, ecsrac02, birth 2, death 3, impendingrcfg 1, stateflags 0x40d
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmSendShutdown: req to node 2, kill time 3547458654
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmDiscHelper: ecsrac02, node(2) connection failed, con (0x2a9786d7f0), probe((nil))
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmWaitOnEvictions: Start
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmWaitOnEvictions: node 2, ecsrac02, undead 0
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmSetupAckWait: Ack message type (15)
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmSetupAckWait: node(1) is ACTIVE
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmSendUpdate: syncSeqNo(3)
[    CSSD]2011-05-11 14:17:12.232 [1220598112] >TRACE:   clssnmWaitForAcks: Ack message type(15), ackCount(1)
[    CSSD]2011-05-11 14:17:12.232 [1147169120] >TRACE:   clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[    CSSD]2011-05-11 14:17:12.232 [1147169120] >TRACE:   clssnmUpdateNodeState: node 1, state (3/3) unique (1301545634/1301545634) prevConuni(0) birth (1/1) (old/new)
[    CSSD]2011-05-11 14:17:12.232 [1147169120] >TRACE:   clssnmUpdateNodeState: node 2, state (5/0) unique (1301545930/1301545930) prevConuni(1301545930) birth (2/2) (old/new)
[    CSSD]2011-05-11 14:17:12.232 [1147169120] >TRACE:   clssnmDeactivateNode: node 2 (ecsrac02) left cluster

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/35489/viewspace-695448/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/35489/viewspace-695448/