---hisdb01.ocssd.log
[ CSSD]2009-11-09 14:09:52.030 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(112721450) proc(11270fed0) pid() proto(10:2:1:1)
[ CSSD]2009-11-09 14:10:28.713 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(112721450) proc(11270fed0) pid() proto(10:2:1:1)
[ CSSD]2009-11-09 14:10:52.466 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(112721450) proc(11270fed0) pid() proto(10:2:1:1)
[ CSSD]2009-11-09 14:11:41.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(2) checkin(s)
[ CSSD]2009-11-09 14:11:42.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(3) checkin(s)
[ CSSD]2009-11-09 14:11:43.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(4) checkin(s)
[ CSSD]2009-11-09 14:11:44.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(5) checkin(s)
[ CSSD]2009-11-09 14:11:45.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(6) checkin(s)
[ CSSD]2009-11-09 14:11:46.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(7) checkin(s)
[ CSSD]2009-11-09 14:11:47.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(8) checkin(s)
[ CSSD]2009-11-09 14:11:48.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(9) checkin(s)
[ CSSD]2009-11-09 14:11:49.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(10) checkin(s)
[ CSSD]2009-11-09 14:11:50.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(11) checkin(s)
[ CSSD]2009-11-09 14:11:51.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(12) checkin(s)
[ CSSD]2009-11-09 14:11:52.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(13) checkin(s)
[ CSSD]2009-11-09 14:11:53.967 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(14) checkin(s)
[ CSSD]2009-11-09 14:11:54.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(15) checkin(s)
[ CSSD]2009-11-09 14:11:55.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(16) checkin(s)
[ CSSD]2009-11-09 14:11:56.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(17) checkin(s)
[ CSSD]2009-11-09 14:11:57.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(18) checkin(s)
[ CSSD]2009-11-09 14:11:58.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(19) checkin(s)
[ CSSD]2009-11-09 14:11:59.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(20) checkin(s)
[ CSSD]2009-11-09 14:12:00.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(21) checkin(s)
[ CSSD]2009-11-09 14:12:01.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(22) checkin(s)
[ CSSD]2009-11-09 14:12:02.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(23) checkin(s)
[ CSSD]2009-11-09 14:12:03.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(24) checkin(s)
[ CSSD]2009-11-09 14:12:04.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(25) checkin(s)
[ CSSD]2009-11-09 14:12:05.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(26) checkin(s)
[ CSSD]2009-11-09 14:12:06.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(27) checkin(s)
[ CSSD]2009-11-09 14:12:07.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(28) checkin(s)
[ CSSD]2009-11-09 14:12:08.968 [3343] >TRACE: clssnmPollingThread: node hisdb02 (2) missed(29) checkin(s)
[ CSSD]2009-11-09 14:12:09.968 [3343] >TRACE: clssnmPollingThread: Eviction started for node hisdb02 (2), flags 0x000d, state 3, wt4c 0
[ CSSD]2009-11-09 14:12:09.968 [3857] >TRACE: clssnmDoSyncUpdate: Initiating sync 3
[ CSSD]2009-11-09 14:12:09.968 [3857] >TRACE: clssnmSetupAckWait: Ack message type (11)
[ CSSD]2009-11-09 14:12:09.968 [3857] >TRACE: clssnmSetupAckWait: node(1) is ALIVE
[ CSSD]2009-11-09 14:12:09.968 [3857] >TRACE: clssnmSetupAckWait: node(2) is ALIVE
[ CSSD]2009-11-09 14:12:09.968 [3857] >TRACE: clssnmSendSync: syncSeqNo(3)
[ CSSD]2009-11-09 14:12:09.968 [2058] >TRACE: clssnmHandleSync: Acknowledging sync: src[1] srcName[hisdb01] seq[9] sync[3]
[ CSSD]2009-11-09 14:12:09.968 [3857] >TRACE: clssnmWaitForAcks: Ack message type(11), ackCount(2)
[ CSSD]2009-11-09 14:12:09.978 [1] >USER: NMEVENT_SUSPEND [00][00][00][06]
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmWaitForAcks: node(2) is expiring, msg type(11)
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmWaitForAcks: done, msg type(11)
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmDoSyncUpdate: node(0) missCount(1567298) state(0)
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmDoSyncUpdate: node(2) missCount(30) state(3)
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmSetupAckWait: Ack message type (13)
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmSendVote: syncSeqNo(3)
[ CSSD]2009-11-09 14:12:10.968 [3857] >TRACE: clssnmWaitForAcks: Ack message type(13), ackCount(1)
[ CSSD]2009-11-09 14:12:10.968 [2058] >TRACE: clssnmSendVoteInfo: node(1) syncSeqNo(3)
[ CSSD]2009-11-09 14:12:11.968 [3857] >TRACE: clssnmWaitForAcks: done, msg type(13)
[ CSSD]2009-11-09 14:12:11.968 [3857] >TRACE: clssnmCheckDskInfo: Checking disk info...
[ CSSD]2009-11-09 14:12:12.997 [3857] >TRACE: clssnmEvict: Start
[ CSSD]2009-11-09 14:12:12.997 [3857] >TRACE: clssnmEvict: Evicting node 2, birth 2, death 3, killme 1
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmEvict: Evicting Node(2), timeout(33280)
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmSendShutdown: req to node 2, kill time 3892238601
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmDiscHelper: node hisdb02 (2) connection failed
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmWaitOnEvictions: Start
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmWaitOnEvictions: Node(0) down, LATS(0),timeout(-402728637)
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmWaitOnEvictions: Node(2) down, LATS(3892205321),timeout(33338)
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmSetupAckWait: Ack message type (15)
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmSetupAckWait: node(1) is ACTIVE
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmSendUpdate: syncSeqNo(3)
[ CSSD]2009-11-09 14:12:13.055 [3857] >TRACE: clssnmWaitForAcks: Ack message type(15), ackCount(1)
[ CSSD]2009-11-09 14:12:13.055 [2058] >TRACE: clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[ CSSD]2009-11-09 14:12:13.055 [2058] >TRACE: clssnmDeactivateNode: node 0 () left cluster
[ CSSD]2009-11-09 14:12:13.055 [2058] >TRACE: clssnmUpdateNodeState: node 1, state (3/3) unique (1256179885/1256179885) prevConuni(0) birth (1/1) (old/new)
[ CSSD]2009-11-09 14:12:13.055 [2058] >TRACE: clssnmUpdateNodeState: node 2, state (0/0) unique (1256179892/1256179892) prevConuni(1256179892) birth (2/0) (old/new)
[ CSSD]2009-11-09 14:12:13.055 [2058] >TRACE: clssnmDeactivateNode: node 2 (hisdb02) left cluster
[ CSSD]2009-11-09 14:12:13.055 [2058] >USER: clssnmHandleUpdate: SYNC(3) from node(1) completed
[ CSSD]2009-11-09 14:12:13.055 [2058] >USER: clssnmHandleUpdate: NODE 1 (hisdb01) IS ACTIVE MEMBER OF CLUSTER
[ CSSD]2009-11-09 14:12:13.084 [4116] >TRACE: clssgmReconfigThread: started for reconfig (3)
[ CSSD]2009-11-09 14:12:13.084 [4116] >USER: NMEVENT_RECONFIG [00][00][00][02]
[ CSSD]2009-11-09 14:12:13.084 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock crs_version type 2
[ CSSD]2009-11-09 14:12:13.084 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(crs_version) birth(2/2)
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_1_orcl type 3
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_2_orcl type 3
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(ORA_CLSRD_2_orcl) birth(2/2)
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DB+ASM type 2
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DB+ASM) birth(2/2)
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DG+ASM type 2
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DG+ASM) birth(2/2)
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DBORCL type 2
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DBORCL) birth(2/2)
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DGORCL type 2
[ CSSD]2009-11-09 14:12:13.085 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DGORCL) birth(2/2)
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock OSM_ALL type 2
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(OSM_ALL) birth(2/2)
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DAALL_DB type 2
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(DAALL_DB) birth(2/2)
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(5) grock(DAALL_DB) birth(2/2)
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(3) grock(DAALL_DB) birth(2/2)
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock CRSDMAIN type 2
[ CSSD]2009-11-09 14:12:13.096 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(CRSDMAIN) birth(2/2)
[ CSSD]2009-11-09 14:12:13.106 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock EVMDMAIN type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(EVMDMAIN) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock IGORCLALL type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(IGORCLALL) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DBSTANDBY type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DBSTANDBY) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DGSTANDBY type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DGSTANDBY) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock ocr_crs type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(ocr_crs) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DG_ASM_DATA type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(2) grock(DG_ASM_DATA) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(3) grock(DG_ASM_DATA) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_hisdb01 type 3
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_hisdb02 type 3
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(_ORA_CRS_MEMBER_hisdb02) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupGrocks: cleaning up grock DG_ASM_DATADG type 2
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(DG_ASM_DATADG) birth(2/2)
[ CSSD]2009-11-09 14:12:13.107 [4116] >TRACE: clssgmCleanupOrphanMembers: cleaning up remote mbr(3) grock(DG_ASM_DATADG) birth(2/2)
[ CSSD]2009-11-09 14:12:13.108 [4116] >TRACE: clssgmEstablishConnections: 1 nodes in cluster incarn 3
[ CSSD]2009-11-09 14:12:13.108 [3086] >TRACE: clssgmPeerDeactivate: node 2 (hisdb02), death 3, state 0x0 connstate 0xf
[ CSSD]2009-11-09 14:12:13.108 [3086] >TRACE: clssgmPeerListener: connects done (1/1)
[ CSSD]2009-11-09 14:12:13.108 [4116] >TRACE: clssgmEstablishMasterNode: MASTER for 3 is node(1) birth(1)
[ CSSD]2009-11-09 14:12:13.108 [4116] >TRACE: clssgmMasterCMSync: Synchronizing group/lock status
[ CSSD]2009-11-09 14:12:13.108 [4116] >TRACE: clssgmMasterSendDBDone: group/lock status synchronization complete
[ CSSD]CLSS-3000: reconfiguration successful, incarnation 3 with 1 nodes
[ CSSD]CLSS-3001: local node number 1, master node number 1
[ CSSD]2009-11-09 14:12:13.129 [4116] >TRACE: clssgmReconfigThread: completed for reconfig(3), with status(1)
[ CSSD]2009-11-09 14:12:13.960 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(11272bbb0) proc(11270d310) pid() proto(10:2:1:1)
[ CSSD]2009-11-09 14:12:14.055 [3857] >TRACE: clssnmWaitForAcks: done, msg type(15)
[ CSSD]2009-11-09 14:12:14.055 [3857] >TRACE: clssnmDoSyncUpdate: Sync Complete!
[ CSSD]2009-11-09 14:12:14.120 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(11272c430) proc(112706150) pid() proto(10:2:1:1)
[ CSSD]2009-11-09 14:12:14.792 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(11272bbb0) proc(11270d310) pid() proto(10:2:1:1)
[ CSSD]2009-11-09 14:12:23.023 [2315] >TRACE: clssgmClientConnectMsg: Connect from con(112709770) proc(1126f37f0) pid(3379236) proto(10:2:1:1)