[ CSSD]2012-11-30 22:06:19.558 [2829] >WARNING: clssnmeventhndlr: Receive failure with node 2 (xjaccdb2), state 3, con(111f1a410), probe(0), rc=11
[ CSSD]2012-11-30 22:06:29.797 [3857] >TRACE: clssgmPeerDeactivate: node 2 (xjaccdb2), death 0, state 0x1 connstate 0xf
[ CSSD]2012-11-30 22:06:29.797 [2829] >TRACE: clssnmDiscHelper: xjaccdb2, node(2) connection failed, con (111f1a410), probe(0)
[ CSSD]2012-11-30 22:06:29.796 [4371] >TRACE: clscsendx: (111f1a410) Physical connection (111f18eb0) not active
[ CSSD]2012-11-30 22:06:29.801 [4371] >WARNING: clssnmsendmsg: send failed, node 2, type 3, rc 11
[ CSSD]2012-11-30 22:06:33.812 [4114] >WARNING: clssnmPollingThread: node xjaccdb2 (2) at 50 2.481040e-265artbeat fatal, eviction in 14.820 seconds seedhbimpd 0
[ CSSD]2012-11-30 22:06:33.812 [4114] >TRACE: clssnmPollingThread: node xjaccdb2 (2) is impending reconfig, flag 1, misstime 15180
[ CSSD]2012-11-30 22:06:33.812 [4114] >TRACE: clssnmPollingThread: diskTimeout set to (27000)ms impending reconfig status(1)
[ CSSD]2012-11-30 22:06:34.814 [4114] >WARNING: clssnmPollingThread: node xjaccdb2 (2) at 50 2.481040e-265artbeat fatal, eviction in 13.818 seconds seedhbimpd 1
[ CSSD]2012-11-30 22:06:41.819 [4114] >WARNING: clssnmPollingThread: node xjaccdb2 (2) at 75 2.481040e-265artbeat fatal, eviction in 6.813 seconds seedhbimpd 1
[ CSSD]2012-11-30 22:06:45.819 [4114] >WARNING: clssnmPollingThread: node xjaccdb2 (2) at 90 2.481040e-265artbeat fatal, eviction in 2.813 seconds seedhbimpd 1
[ CSSD]2012-11-30 22:06:46.827 [4114] >WARNING: clssnmPollingThread: node xjaccdb2 (2) at 90 2.481040e-265artbeat fatal, eviction in 1.805 seconds seedhbimpd 1
[ CSSD]2012-11-30 22:06:47.834 [4114] >WARNING: clssnmPollingThread: node xjaccdb2 (2) at 90 2.481040e-265artbeat fatal, eviction in 0.797 seconds seedhbimpd 1
[ CSSD]2012-11-30 22:06:48.635 [4114] >TRACE: clssnmPollingThread: Eviction started for node xjaccdb2 (2), flags 0x0001, state 3, wt4c 0 seedhbimpd 1
[ CSSD]2012-11-30 22:06:48.635 [4628] >TRACE: clssnmDoSyncUpdate: Initiating sync 3
[ CSSD]2012-11-30 22:06:48.635 [4628] >TRACE: clssnmDoSyncUpdate: diskTimeout set to (27000)ms
[ CSSD]2012-11-30 22:06:48.635 [4628] >TRACE: clssnmSetupAckWait: Ack message type (11)
[ CSSD]2012-11-30 22:06:48.635 [4628] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
[ CSSD]2012-11-30 22:06:48.635 [4628] >TRACE: clssnmSendSync: syncSeqNo(3)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmWaitForAcks: Ack message type(11), ackCount(1)
[ CSSD]2012-11-30 22:06:48.636 [2829] >TRACE: clssnmHandleSync: diskTimeout set to (27000)ms
[ CSSD]2012-11-30 22:06:48.636 [2829] >TRACE: clssnmHandleSync: Acknowledging sync: src[1] srcName[xjaccdb1] seq[9] sync[3]
[ CSSD]2012-11-30 22:06:48.636 [1] >USER: NMEVENT_SUSPEND [00][00][00][06]
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmWaitForAcks: done, msg type(11)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmDoSyncUpdate: Terminating node 2, xjaccdb2, misstime(30004) state(5), seedhbimpd(1)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmSetupAckWait: Ack message type (13)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmWaitForAcks: Ack message type(13), ackCount(1)
[ CSSD]2012-11-30 22:06:48.636 [2829] >TRACE: clssnmSendVoteInfo: node(1) syncSeqNo(3)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmWaitForAcks: done, msg type(13)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmCheckDskInfo: Checking disk info...
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmEvict: Start
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmEvict: Evicting node 2, xjaccdb2, birth 2, death 3, impendingrcfg 1, stateflags 0x1
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmWaitOnEvictions: Start
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmWaitOnEvictions: node 2, xjaccdb2, is not dead, seedhbimpd 1
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmCheckKillStatus: Node 2, xjaccdb2, down, LATS(1290002121),timeout(14822)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmSetupAckWait: Ack message type (15)
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2012-11-30 22:06:48.636 [4628] >TRACE: clssnmSendUpdate: syncSeqNo(3)
[ CSSD]2012-11-30 22:06:59.438 [4628] >WARNING: CLSSNMCTX_NODEDB_UNLOCK: lock held for 10802 ms
[ CSSD]2012-11-30 22:06:59.438 [4628] >TRACE: clssnmWaitForAcks: Ack message type(15), ackCount(1)
[ CSSD]2012-11-30 22:06:59.441 [2829] >TRACE: clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[ CSSD]2012-11-30 22:06:59.441 [2829] >TRACE: clssnmUpdateNodeState: node 1, state (3/3) unique (1354218233/1354218233) prevConuni(0) birth (1/1) (old/new)
[ CSSD]2012-11-30 22:06:59.441 [2829] >TRACE: clssnmUpdateNodeState: node 2, state (5/0) unique (1354219454/1354219454) prevConuni(1354219454) birth (2/2) (old/new)
[ CSSD]2012-11-30 22:06:59.441 [2829] >TRACE: clssnmDeactivateNode: node 2 (xjaccdb2) left cluster
[ CSSD]2012-11-30 22:06:59.441 [2829] >USER: clssnmHandleUpdate: SYNC(3) from node(1) completed
[ CSSD]2012-11-30 22:06:59.441 [2829] >USER: clssnmHandleUpdate: NODE 1 (xjaccdb1) IS ACTIVE MEMBER OF CLUSTER
[ CSSD]2012-11-30 22:06:59.441 [2829] >TRACE: clssnmHandleUpdate: diskTimeout set to (200000)ms
[ CSSD]2012-11-30 22:06:59.442 [4628] >TRACE: clssnmWaitForAcks: done, msg type(15)
[ CSSD]2012-11-30 22:06:59.442 [4628] >TRACE: clssnmDoSyncUpdate: Sync 3 complete!
[ CSSD]2012-11-30 22:06:59.450 [4887] >TRACE: clssgmReconfigThread: started for reconfig (3)
[ CSSD]2012-11-30 22:06:59.450 [4887] >USER: NMEVENT_RECONFIG [00][00][00][02]
[ CSSD]2012-11-30 22:06:59.450 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock crs_version type 2
[ CSSD]2012-11-30 22:06:59.450 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(crs_version) birth(2/0)
[ CSSD]2012-11-30 22:06:59.453 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_1_accdb type 2
[ CSSD]2012-11-30 22:06:59.453 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_1_accdb type 3
[ CSSD]2012-11-30 22:06:59.453 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_2_accdb type 2
[ CSSD]2012-11-30 22:07:00.190 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(ORA_CLSRD_2_accdb) birth(2/0)
[ CSSD]2012-11-30 22:07:00.190 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_2_accdb type 3
[ CSSD]2012-11-30 22:07:00.190 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(ORA_CLSRD_2_accdb) birth(2/0)
[ CSSD]2012-11-30 22:07:00.190 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock DBACCDB type 2
[ CSSD]2012-11-30 22:07:00.190 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DBACCDB) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock DGACCDB type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DGACCDB) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock DAALL_DB type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DAALL_DB) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock CRSDMAIN type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(CRSDMAIN) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock EVMDMAIN type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(EVMDMAIN) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock IGACCDBALL type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(IGACCDBALL) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock ocr_crs type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(ocr_crs) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock IGACCDBACCDB1 type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock IGACCDBACCDB2 type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(IGACCDBACCDB2) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock #CSS_CLSSOMON type 2
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(#CSS_CLSSOMON) birth(2/0)
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_xjaccdb1 type 3
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_xjaccdb2 type 3
[ CSSD]2012-11-30 22:07:00.191 [4887] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(_ORA_CRS_MEMBER_xjaccdb2) birth(2/0)
[ CSSD]2012-11-30 22:07:00.192 [4887] >TRACE: clssgmEstablishConnections: 1 nodes in cluster incarn 3
[ CSSD]2012-11-30 22:07:00.192 [3857] >TRACE: clssgmPeerListener: connects done (1/1)
[ CSSD]2012-11-30 22:07:00.192 [4887] >TRACE: clssgmEstablishMasterNode: MASTER for 3 is node(1) birth(1)
[ CSSD]2012-11-30 22:07:00.192 [4887] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[ CSSD]2012-11-30 22:07:00.241 [4887] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[ CSSD]CLSS-3000: reconfiguration successful, incarnation 3 with 1 nodes
[ CSSD]CLSS-3001: local node number 1, master node number 1