本帖最后由 gongwenbo7 于 2013-3-4 09:37 编辑
操作系统为SUSE,数据库版本为11.2.0.3
使用CRS命令启动里报错如下:
grid@zxdb02:~> sudo /Oracle/app/crs/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
cssd一直是STARING状态
grid@zxdb02:/Oracle/app/crs/log/zxdb02> sudo /Oracle/app/crs/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE OFFLINE
ora.cluster_interconnect.haip
1 ONLINE OFFLINE
ora.crf
1 ONLINE ONLINE zxdb02
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE OFFLINE STARTING
ora.cssdmonitor
1 ONLINE ONLINE zxdb02
ora.ctssd
1 ONLINE OFFLINE
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE OFFLINE
ora.evmd
1 ONLINE OFFLINE
ora.gipcd
1 ONLINE ONLINE zxdb02
ora.gpnpd
1 ONLINE ONLINE zxdb02
ora.mdnsd
1 ONLINE ONLINE zxdb02
具体的ocssd日志如下:
grid@zxdb02:/Oracle/app/crs/log/zxdb02> tail -100f /Oracle/app/crs/log/zxdb02/cssd/ocssd.log
2013-03-03 22:40:00.921: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222779, LATS 1758696, lastSeqNo 11222778, uniqueness 1361347261, timestamp 1362321604/975363106
2013-03-03 22:40:01.316: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:01.925: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222780, LATS 1759706, lastSeqNo 11222779, uniqueness 1361347261, timestamp 1362321605/975364106
2013-03-03 22:40:02.316: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:02.816: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:02.816: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:02.930: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222781, LATS 1760706, lastSeqNo 11222780, uniqueness 1361347261, timestamp 1362321606/975365106
2013-03-03 22:40:03.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:03.941: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222782, LATS 1761716, lastSeqNo 11222781, uniqueness 1361347261, timestamp 1362321607/975366106
2013-03-03 22:40:04.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:04.946: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222783, LATS 1762726, lastSeqNo 11222782, uniqueness 1361347261, timestamp 1362321608/975367106
2013-03-03 22:40:05.059: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:05.059: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b0360520) allocated
2013-03-03 22:40:05.059: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b0360520 - 1,2,3,4,5
2013-03-03 22:40:05.059: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x2e74) proc(0x7f70b0360520) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:05.059: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:05.061: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b0360520
2013-03-03 22:40:05.061: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b0360520
2013-03-03 22:40:05.061: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b0360520), client(1/0x7f70b0156a60)
2013-03-03 22:40:05.061: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b0156a60 with con 0x7f7000002ea3, requested num -1, flags 0x4000e00
2013-03-03 22:40:05.061: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:05.061: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2ea3
2013-03-03 22:40:05.062: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b0360520
2013-03-03 22:40:05.062: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b0360520) con(0x2e74) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:05.062: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2e74
2013-03-03 22:40:05.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:05.951: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222784, LATS 1763726, lastSeqNo 11222783, uniqueness 1361347261, timestamp 1362321609/975368106
2013-03-03 22:40:06.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:06.811: [ CSSD][2941732624]clssnmRcfgMgrThread: Local Join
2013-03-03 22:40:06.811: [ CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2013-03-03 22:40:06.811: [ CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1764586) for my node
2013-03-03 22:40:06.811: [ CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes
2013-03-03 22:40:06.811: [ CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3
2013-03-03 22:40:06.812: [ CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2013-03-03 22:40:06.962: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222785, LATS 1764736, lastSeqNo 11222784, uniqueness 1361347261, timestamp 1362321610/975369106
2013-03-03 22:40:07.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:07.816: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:07.816: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:07.966: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222786, LATS 1765746, lastSeqNo 11222785, uniqueness 1361347261, timestamp 1362321611/975370106
2013-03-03 22:40:08.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:08.971: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222787, LATS 1766746, lastSeqNo 11222786, uniqueness 1361347261, timestamp 1362321612/975371106
2013-03-03 22:40:09.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:09.982: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222788, LATS 1767756, lastSeqNo 11222787, uniqueness 1361347261, timestamp 1362321613/975372106
2013-03-03 22:40:10.065: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:10.065: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b022db90) allocated
2013-03-03 22:40:10.066: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b022db90 - 1,2,3,4,5
2013-03-03 22:40:10.066: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x2f3c) proc(0x7f70b022db90) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:10.066: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:10.067: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b022db90
2013-03-03 22:40:10.067: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b022db90
2013-03-03 22:40:10.067: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b022db90), client(1/0x7f70b02de960)
2013-03-03 22:40:10.067: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b02de960 with con 0x7f7000002f6b, requested num -1, flags 0x4000e00
2013-03-03 22:40:10.067: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:10.067: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2f6b
2013-03-03 22:40:10.068: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b022db90
2013-03-03 22:40:10.068: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b022db90) con(0x2f3c) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:10.068: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2f3c
2013-03-03 22:40:10.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:10.987: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222789, LATS 1768766, lastSeqNo 11222788, uniqueness 1361347261, timestamp 1362321614/975373106
2013-03-03 22:40:11.317: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:11.992: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222790, LATS 1769776, lastSeqNo 11222789, uniqueness 1361347261, timestamp 1362321615/975374106
2013-03-03 22:40:12.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:12.817: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:12.817: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:13.002: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222791, LATS 1770776, lastSeqNo 11222790, uniqueness 1361347261, timestamp 1362321616/975375106
2013-03-03 22:40:13.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:13.812: [ CSSD][2941732624]clssnmRcfgMgrThread: Local Join
2013-03-03 22:40:13.812: [ CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2013-03-03 22:40:13.812: [ CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1771596) for my node
2013-03-03 22:40:13.812: [ CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes
2013-03-03 22:40:13.812: [ CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3
2013-03-03 22:40:13.812: [ CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2013-03-03 22:40:14.007: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222792, LATS 1771786, lastSeqNo 11222791, uniqueness 1361347261, timestamp 1362321617/975376106
2013-03-03 22:40:14.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:15.012: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222793, LATS 1772796, lastSeqNo 11222792, uniqueness 1361347261, timestamp 1362321618/975377106
2013-03-03 22:40:15.072: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:15.072: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated
2013-03-03 22:40:15.072: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5
2013-03-03 22:40:15.072: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x3012) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:15.072: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:15.074: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460
2013-03-03 22:40:15.074: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460
2013-03-03 22:40:15.074: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b011f5d0)
2013-03-03 22:40:15.074: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b011f5d0 with con 0x7f7000003041, requested num -1, flags 0x4000e00
2013-03-03 22:40:15.074: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:15.074: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3041
2013-03-03 22:40:15.075: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460
2013-03-03 22:40:15.075: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x3012) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:15.075: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3012
2013-03-03 22:40:15.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:16.023: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222794, LATS 1773796, lastSeqNo 11222793, uniqueness 1361347261, timestamp 1362321619/975378116
2013-03-03 22:40:16.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:17.034: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222795, LATS 1774816, lastSeqNo 11222794, uniqueness 1361347261, timestamp 1362321620/975379116
2013-03-03 22:40:17.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:17.817: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:17.817: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:18.039: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222796, LATS 1775816, lastSeqNo 11222795, uniqueness 1361347261, timestamp 1362321621/975380116
2013-03-03 22:40:18.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:19.049: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222797, LATS 1776826, lastSeqNo 11222796, uniqueness 1361347261, timestamp 1362321622/975381116
2013-03-03 22:40:19.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:20.054: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222798, LATS 1777836, lastSeqNo 11222797, uniqueness 1361347261, timestamp 1362321623/975382116
2013-03-03 22:40:20.078: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:20.078: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b036f030) allocated
2013-03-03 22:40:20.079: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b036f030 - 1,2,3,4,5
2013-03-03 22:40:20.079: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x30da) proc(0x7f70b036f030) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:20.079: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:20.080: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b036f030
2013-03-03 22:40:20.080: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b036f030
2013-03-03 22:40:20.080: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b036f030), client(1/0x7f70b0360fd0)
2013-03-03 22:40:20.080: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b0360fd0 with con 0x7f7000003109, requested num -1, flags 0x4000e00
2013-03-03 22:40:20.080: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:20.081: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3109
2013-03-03 22:40:20.081: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b036f030
2013-03-03 22:40:20.081: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b036f030) con(0x30da) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:20.081: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x30da
2013-03-03 22:40:20.318: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:20.812: [ CSSD][2941732624]clssnmRcfgMgrThread: Local Join
2013-03-03 22:40:20.812: [ CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2013-03-03 22:40:20.812: [ CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1778596) for my node
2013-03-03 22:40:20.812: [ CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes
2013-03-03 22:40:20.812: [ CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3
2013-03-03 22:40:20.813: [ CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2013-03-03 22:40:21.059: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222799, LATS 1778836, lastSeqNo 11222798, uniqueness 1361347261, timestamp 1362321624/975383116
2013-03-03 22:40:21.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:22.070: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222800, LATS 1779846, lastSeqNo 11222799, uniqueness 1361347261, timestamp 1362321625/975384116
2013-03-03 22:40:22.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:22.818: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:22.818: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:23.075: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222801, LATS 1780856, lastSeqNo 11222800, uniqueness 1361347261, timestamp 1362321626/975385116
2013-03-03 22:40:23.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:24.080: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222802, LATS 1781866, lastSeqNo 11222801, uniqueness 1361347261, timestamp 1362321627/975386116
2013-03-03 22:40:24.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:25.085: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:25.085: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated
2013-03-03 22:40:25.085: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5
2013-03-03 22:40:25.085: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x31b3) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:25.085: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:25.087: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460
2013-03-03 22:40:25.087: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460
2013-03-03 22:40:25.087: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b022df30)
2013-03-03 22:40:25.087: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b022df30 with con 0x7f70000031e2, requested num -1, flags 0x4000e00
2013-03-03 22:40:25.087: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:25.087: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x31e2
2013-03-03 22:40:25.088: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460
2013-03-03 22:40:25.088: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x31b3) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:25.088: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x31b3
2013-03-03 22:40:25.090: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222803, LATS 1782866, lastSeqNo 11222802, uniqueness 1361347261, timestamp 1362321628/975387116
2013-03-03 22:40:25.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:26.095: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222804, LATS 1783876, lastSeqNo 11222803, uniqueness 1361347261, timestamp 1362321629/975388116
2013-03-03 22:40:26.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:27.100: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222805, LATS 1784886, lastSeqNo 11222804, uniqueness 1361347261, timestamp 1362321630/975389116
2013-03-03 22:40:27.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:27.813: [ CSSD][2941732624]clssnmRcfgMgrThread: Local Join
2013-03-03 22:40:27.813: [ CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2013-03-03 22:40:27.813: [ CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1785596) for my node
2013-03-03 22:40:27.813: [ CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes
2013-03-03 22:40:27.813: [ CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3
2013-03-03 22:40:27.814: [ CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2013-03-03 22:40:27.818: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:27.818: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:28.111: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222806, LATS 1785886, lastSeqNo 11222805, uniqueness 1361347261, timestamp 1362321631/975390116
2013-03-03 22:40:28.319: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:29.116: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222807, LATS 1786896, lastSeqNo 11222806, uniqueness 1361347261, timestamp 1362321632/975391116
2013-03-03 22:40:29.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:30.091: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:30.091: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated
2013-03-03 22:40:30.092: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5
2013-03-03 22:40:30.092: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x327b) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:30.092: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:30.093: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460
2013-03-03 22:40:30.093: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460
2013-03-03 22:40:30.093: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b022df30)
2013-03-03 22:40:30.093: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b022df30 with con 0x7f70000032aa, requested num -1, flags 0x4000e00
2013-03-03 22:40:30.093: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:30.094: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x32aa
2013-03-03 22:40:30.094: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460
2013-03-03 22:40:30.094: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x327b) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:30.094: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x327b
2013-03-03 22:40:30.121: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222808, LATS 1787906, lastSeqNo 11222807, uniqueness 1361347261, timestamp 1362321633/975392116
2013-03-03 22:40:30.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:31.125: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222809, LATS 1788906, lastSeqNo 11222808, uniqueness 1361347261, timestamp 1362321634/975393116
2013-03-03 22:40:31.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:32.130: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222810, LATS 1789906, lastSeqNo 11222809, uniqueness 1361347261, timestamp 1362321635/975394116
2013-03-03 22:40:32.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:32.819: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:32.819: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:33.135: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222811, LATS 1790916, lastSeqNo 11222810, uniqueness 1361347261, timestamp 1362321636/975395116
2013-03-03 22:40:33.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:34.146: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222812, LATS 1791926, lastSeqNo 11222811, uniqueness 1361347261, timestamp 1362321637/975396116
2013-03-03 22:40:34.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:34.813: [ CSSD][2941732624]clssnmRcfgMgrThread: Local Join
2013-03-03 22:40:34.813: [ CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2013-03-03 22:40:34.813: [ CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1792596) for my node
2013-03-03 22:40:34.813: [ CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes
2013-03-03 22:40:34.814: [ CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3
2013-03-03 22:40:34.814: [ CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2013-03-03 22:40:35.098: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:35.098: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated
2013-03-03 22:40:35.098: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5
2013-03-03 22:40:35.098: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x3351) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:35.098: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:35.100: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460
2013-03-03 22:40:35.100: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460
2013-03-03 22:40:35.100: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b036bf50)
2013-03-03 22:40:35.100: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b036bf50 with con 0x7f7000003380, requested num -1, flags 0x4000e00
2013-03-03 22:40:35.100: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:35.100: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3380
2013-03-03 22:40:35.101: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460
2013-03-03 22:40:35.101: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x3351) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:35.101: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3351
2013-03-03 22:40:35.151: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222813, LATS 1792926, lastSeqNo 11222812, uniqueness 1361347261, timestamp 1362321638/975397116
2013-03-03 22:40:35.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:36.155: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222814, LATS 1793936, lastSeqNo 11222813, uniqueness 1361347261, timestamp 1362321639/975398116
2013-03-03 22:40:36.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:37.160: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222815, LATS 1794946, lastSeqNo 11222814, uniqueness 1361347261, timestamp 1362321640/975399116
2013-03-03 22:40:37.320: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:37.819: [ CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes
2013-03-03 22:40:37.819: [ CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes
2013-03-03 22:40:38.165: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222816, LATS 1795946, lastSeqNo 11222815, uniqueness 1361347261, timestamp 1362321641/975400116
2013-03-03 22:40:38.321: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:39.170: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222817, LATS 1796946, lastSeqNo 11222816, uniqueness 1361347261, timestamp 1362321642/975401126
2013-03-03 22:40:39.321: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:40.104: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0
2013-03-03 22:40:40.104: [ CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated
2013-03-03 22:40:40.105: [ CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5
2013-03-03 22:40:40.105: [ CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x3419) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5
2013-03-03 22:40:40.105: [ CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000
2013-03-03 22:40:40.106: [ CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460
2013-03-03 22:40:40.106: [ CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460
2013-03-03 22:40:40.106: [ CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b00f6300)
2013-03-03 22:40:40.106: [ CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b00f6300 with con 0x7f7000003448, requested num -1, flags 0x4000e00
2013-03-03 22:40:40.106: [ CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00
2013-03-03 22:40:40.106: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3448
2013-03-03 22:40:40.107: [ CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460
2013-03-03 22:40:40.107: [ CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x3419) skgpid ospid 13642 with 0 clients, refcount 0
2013-03-03 22:40:40.107: [ CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3419
2013-03-03 22:40:40.181: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222818, LATS 1797966, lastSeqNo 11222817, uniqueness 1361347261, timestamp 1362321643/975402126
2013-03-03 22:40:40.321: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:41.186: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222819, LATS 1798966, lastSeqNo 11222818, uniqueness 1361347261, timestamp 1362321644/975403126
2013-03-03 22:40:41.321: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
2013-03-03 22:40:41.814: [ CSSD][2941732624]clssnmRcfgMgrThread: Local Join
2013-03-03 22:40:41.814: [ CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000
2013-03-03 22:40:41.814: [ CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1799596) for my node
2013-03-03 22:40:41.814: [ CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes
2013-03-03 22:40:41.814: [ CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3
2013-03-03 22:40:41.815: [ CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk
2013-03-03 22:40:42.190: [ CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222820, LATS 1799966, lastSeqNo 11222819, uniqueness 1361347261, timestamp 1362321645/975404126
2013-03-03 22:40:42.321: [ CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0