rac 启动其中一个node_启动RAC一个节点正常另一节点cssd一直是starting

本帖最后由 gongwenbo7 于 2013-3-4 09:37 编辑

操作系统为SUSE，数据库版本为11.2.0.3

使用CRS命令启动里报错如下：

grid@zxdb02:~> sudo /Oracle/app/crs/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

cssd一直是STARING状态

grid@zxdb02:/Oracle/app/crs/log/zxdb02> sudo /Oracle/app/crs/bin/crsctl stat res -t -init

--------------------------------------------------------------------------------

NAME           TARGET  STATE        SERVER                   STATE_DETAILS

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

1        ONLINE  OFFLINE

ora.cluster_interconnect.haip

1        ONLINE  OFFLINE

ora.crf

1        ONLINE  ONLINE       zxdb02

ora.crsd

1        ONLINE  OFFLINE

ora.cssd

1        ONLINE  OFFLINE                               STARTING

ora.cssdmonitor

1        ONLINE  ONLINE       zxdb02

ora.ctssd

1        ONLINE  OFFLINE

ora.diskmon

1        OFFLINE OFFLINE

ora.drivers.acfs

1        ONLINE  OFFLINE

ora.evmd

1        ONLINE  OFFLINE

ora.gipcd

1        ONLINE  ONLINE       zxdb02

ora.gpnpd

1        ONLINE  ONLINE       zxdb02

ora.mdnsd

1        ONLINE  ONLINE       zxdb02

具体的ocssd日志如下：

grid@zxdb02:/Oracle/app/crs/log/zxdb02> tail -100f /Oracle/app/crs/log/zxdb02/cssd/ocssd.log

2013-03-03 22:40:00.921: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222779, LATS 1758696, lastSeqNo 11222778, uniqueness 1361347261, timestamp 1362321604/975363106

2013-03-03 22:40:01.316: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:01.925: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222780, LATS 1759706, lastSeqNo 11222779, uniqueness 1361347261, timestamp 1362321605/975364106

2013-03-03 22:40:02.316: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:02.816: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:02.816: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:02.930: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222781, LATS 1760706, lastSeqNo 11222780, uniqueness 1361347261, timestamp 1362321606/975365106

2013-03-03 22:40:03.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:03.941: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222782, LATS 1761716, lastSeqNo 11222781, uniqueness 1361347261, timestamp 1362321607/975366106

2013-03-03 22:40:04.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:04.946: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222783, LATS 1762726, lastSeqNo 11222782, uniqueness 1361347261, timestamp 1362321608/975367106

2013-03-03 22:40:05.059: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:05.059: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b0360520) allocated

2013-03-03 22:40:05.059: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b0360520 - 1,2,3,4,5

2013-03-03 22:40:05.059: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x2e74) proc(0x7f70b0360520) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:05.059: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:05.061: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b0360520

2013-03-03 22:40:05.061: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b0360520

2013-03-03 22:40:05.061: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b0360520), client(1/0x7f70b0156a60)

2013-03-03 22:40:05.061: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b0156a60 with con 0x7f7000002ea3, requested num -1, flags 0x4000e00

2013-03-03 22:40:05.061: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:05.061: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2ea3

2013-03-03 22:40:05.062: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b0360520

2013-03-03 22:40:05.062: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b0360520) con(0x2e74) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:05.062: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2e74

2013-03-03 22:40:05.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:05.951: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222784, LATS 1763726, lastSeqNo 11222783, uniqueness 1361347261, timestamp 1362321609/975368106

2013-03-03 22:40:06.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:06.811: [    CSSD][2941732624]clssnmRcfgMgrThread: Local Join

2013-03-03 22:40:06.811: [    CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000

2013-03-03 22:40:06.811: [    CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1764586) for my node

2013-03-03 22:40:06.811: [    CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes

2013-03-03 22:40:06.811: [    CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3

2013-03-03 22:40:06.812: [    CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk

2013-03-03 22:40:06.962: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222785, LATS 1764736, lastSeqNo 11222784, uniqueness 1361347261, timestamp 1362321610/975369106

2013-03-03 22:40:07.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:07.816: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:07.816: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:07.966: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222786, LATS 1765746, lastSeqNo 11222785, uniqueness 1361347261, timestamp 1362321611/975370106

2013-03-03 22:40:08.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:08.971: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222787, LATS 1766746, lastSeqNo 11222786, uniqueness 1361347261, timestamp 1362321612/975371106

2013-03-03 22:40:09.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:09.982: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222788, LATS 1767756, lastSeqNo 11222787, uniqueness 1361347261, timestamp 1362321613/975372106

2013-03-03 22:40:10.065: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:10.065: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b022db90) allocated

2013-03-03 22:40:10.066: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b022db90 - 1,2,3,4,5

2013-03-03 22:40:10.066: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x2f3c) proc(0x7f70b022db90) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:10.066: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:10.067: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b022db90

2013-03-03 22:40:10.067: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b022db90

2013-03-03 22:40:10.067: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b022db90), client(1/0x7f70b02de960)

2013-03-03 22:40:10.067: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b02de960 with con 0x7f7000002f6b, requested num -1, flags 0x4000e00

2013-03-03 22:40:10.067: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:10.067: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2f6b

2013-03-03 22:40:10.068: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b022db90

2013-03-03 22:40:10.068: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b022db90) con(0x2f3c) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:10.068: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x2f3c

2013-03-03 22:40:10.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:10.987: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222789, LATS 1768766, lastSeqNo 11222788, uniqueness 1361347261, timestamp 1362321614/975373106

2013-03-03 22:40:11.317: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:11.992: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222790, LATS 1769776, lastSeqNo 11222789, uniqueness 1361347261, timestamp 1362321615/975374106

2013-03-03 22:40:12.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:12.817: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:12.817: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:13.002: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222791, LATS 1770776, lastSeqNo 11222790, uniqueness 1361347261, timestamp 1362321616/975375106

2013-03-03 22:40:13.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:13.812: [    CSSD][2941732624]clssnmRcfgMgrThread: Local Join

2013-03-03 22:40:13.812: [    CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000

2013-03-03 22:40:13.812: [    CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1771596) for my node

2013-03-03 22:40:13.812: [    CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes

2013-03-03 22:40:13.812: [    CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3

2013-03-03 22:40:13.812: [    CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk

2013-03-03 22:40:14.007: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222792, LATS 1771786, lastSeqNo 11222791, uniqueness 1361347261, timestamp 1362321617/975376106

2013-03-03 22:40:14.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:15.012: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222793, LATS 1772796, lastSeqNo 11222792, uniqueness 1361347261, timestamp 1362321618/975377106

2013-03-03 22:40:15.072: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:15.072: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated

2013-03-03 22:40:15.072: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5

2013-03-03 22:40:15.072: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x3012) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:15.072: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:15.074: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460

2013-03-03 22:40:15.074: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460

2013-03-03 22:40:15.074: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b011f5d0)

2013-03-03 22:40:15.074: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b011f5d0 with con 0x7f7000003041, requested num -1, flags 0x4000e00

2013-03-03 22:40:15.074: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:15.074: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3041

2013-03-03 22:40:15.075: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460

2013-03-03 22:40:15.075: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x3012) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:15.075: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3012

2013-03-03 22:40:15.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:16.023: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222794, LATS 1773796, lastSeqNo 11222793, uniqueness 1361347261, timestamp 1362321619/975378116

2013-03-03 22:40:16.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:17.034: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222795, LATS 1774816, lastSeqNo 11222794, uniqueness 1361347261, timestamp 1362321620/975379116

2013-03-03 22:40:17.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:17.817: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:17.817: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:18.039: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222796, LATS 1775816, lastSeqNo 11222795, uniqueness 1361347261, timestamp 1362321621/975380116

2013-03-03 22:40:18.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:19.049: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222797, LATS 1776826, lastSeqNo 11222796, uniqueness 1361347261, timestamp 1362321622/975381116

2013-03-03 22:40:19.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:20.054: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222798, LATS 1777836, lastSeqNo 11222797, uniqueness 1361347261, timestamp 1362321623/975382116

2013-03-03 22:40:20.078: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:20.078: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b036f030) allocated

2013-03-03 22:40:20.079: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b036f030 - 1,2,3,4,5

2013-03-03 22:40:20.079: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x30da) proc(0x7f70b036f030) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:20.079: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:20.080: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b036f030

2013-03-03 22:40:20.080: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b036f030

2013-03-03 22:40:20.080: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b036f030), client(1/0x7f70b0360fd0)

2013-03-03 22:40:20.080: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b0360fd0 with con 0x7f7000003109, requested num -1, flags 0x4000e00

2013-03-03 22:40:20.080: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:20.081: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3109

2013-03-03 22:40:20.081: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b036f030

2013-03-03 22:40:20.081: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b036f030) con(0x30da) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:20.081: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x30da

2013-03-03 22:40:20.318: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:20.812: [    CSSD][2941732624]clssnmRcfgMgrThread: Local Join

2013-03-03 22:40:20.812: [    CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000

2013-03-03 22:40:20.812: [    CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1778596) for my node

2013-03-03 22:40:20.812: [    CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes

2013-03-03 22:40:20.812: [    CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3

2013-03-03 22:40:20.813: [    CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk

2013-03-03 22:40:21.059: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222799, LATS 1778836, lastSeqNo 11222798, uniqueness 1361347261, timestamp 1362321624/975383116

2013-03-03 22:40:21.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:22.070: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222800, LATS 1779846, lastSeqNo 11222799, uniqueness 1361347261, timestamp 1362321625/975384116

2013-03-03 22:40:22.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:22.818: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:22.818: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:23.075: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222801, LATS 1780856, lastSeqNo 11222800, uniqueness 1361347261, timestamp 1362321626/975385116

2013-03-03 22:40:23.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:24.080: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222802, LATS 1781866, lastSeqNo 11222801, uniqueness 1361347261, timestamp 1362321627/975386116

2013-03-03 22:40:24.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:25.085: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:25.085: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated

2013-03-03 22:40:25.085: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5

2013-03-03 22:40:25.085: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x31b3) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:25.085: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:25.087: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460

2013-03-03 22:40:25.087: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460

2013-03-03 22:40:25.087: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b022df30)

2013-03-03 22:40:25.087: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b022df30 with con 0x7f70000031e2, requested num -1, flags 0x4000e00

2013-03-03 22:40:25.087: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:25.087: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x31e2

2013-03-03 22:40:25.088: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460

2013-03-03 22:40:25.088: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x31b3) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:25.088: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x31b3

2013-03-03 22:40:25.090: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222803, LATS 1782866, lastSeqNo 11222802, uniqueness 1361347261, timestamp 1362321628/975387116

2013-03-03 22:40:25.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:26.095: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222804, LATS 1783876, lastSeqNo 11222803, uniqueness 1361347261, timestamp 1362321629/975388116

2013-03-03 22:40:26.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:27.100: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222805, LATS 1784886, lastSeqNo 11222804, uniqueness 1361347261, timestamp 1362321630/975389116

2013-03-03 22:40:27.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:27.813: [    CSSD][2941732624]clssnmRcfgMgrThread: Local Join

2013-03-03 22:40:27.813: [    CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000

2013-03-03 22:40:27.813: [    CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1785596) for my node

2013-03-03 22:40:27.813: [    CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes

2013-03-03 22:40:27.813: [    CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3

2013-03-03 22:40:27.814: [    CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk

2013-03-03 22:40:27.818: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:27.818: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:28.111: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222806, LATS 1785886, lastSeqNo 11222805, uniqueness 1361347261, timestamp 1362321631/975390116

2013-03-03 22:40:28.319: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:29.116: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222807, LATS 1786896, lastSeqNo 11222806, uniqueness 1361347261, timestamp 1362321632/975391116

2013-03-03 22:40:29.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:30.091: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:30.091: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated

2013-03-03 22:40:30.092: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5

2013-03-03 22:40:30.092: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x327b) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:30.092: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:30.093: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460

2013-03-03 22:40:30.093: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460

2013-03-03 22:40:30.093: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b022df30)

2013-03-03 22:40:30.093: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b022df30 with con 0x7f70000032aa, requested num -1, flags 0x4000e00

2013-03-03 22:40:30.093: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:30.094: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x32aa

2013-03-03 22:40:30.094: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460

2013-03-03 22:40:30.094: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x327b) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:30.094: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x327b

2013-03-03 22:40:30.121: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222808, LATS 1787906, lastSeqNo 11222807, uniqueness 1361347261, timestamp 1362321633/975392116

2013-03-03 22:40:30.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:31.125: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222809, LATS 1788906, lastSeqNo 11222808, uniqueness 1361347261, timestamp 1362321634/975393116

2013-03-03 22:40:31.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:32.130: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222810, LATS 1789906, lastSeqNo 11222809, uniqueness 1361347261, timestamp 1362321635/975394116

2013-03-03 22:40:32.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:32.819: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:32.819: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:33.135: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222811, LATS 1790916, lastSeqNo 11222810, uniqueness 1361347261, timestamp 1362321636/975395116

2013-03-03 22:40:33.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:34.146: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222812, LATS 1791926, lastSeqNo 11222811, uniqueness 1361347261, timestamp 1362321637/975396116

2013-03-03 22:40:34.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:34.813: [    CSSD][2941732624]clssnmRcfgMgrThread: Local Join

2013-03-03 22:40:34.813: [    CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000

2013-03-03 22:40:34.813: [    CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1792596) for my node

2013-03-03 22:40:34.813: [    CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes

2013-03-03 22:40:34.814: [    CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3

2013-03-03 22:40:34.814: [    CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk

2013-03-03 22:40:35.098: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:35.098: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated

2013-03-03 22:40:35.098: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5

2013-03-03 22:40:35.098: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x3351) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:35.098: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:35.100: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460

2013-03-03 22:40:35.100: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460

2013-03-03 22:40:35.100: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b036bf50)

2013-03-03 22:40:35.100: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b036bf50 with con 0x7f7000003380, requested num -1, flags 0x4000e00

2013-03-03 22:40:35.100: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:35.100: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3380

2013-03-03 22:40:35.101: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460

2013-03-03 22:40:35.101: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x3351) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:35.101: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3351

2013-03-03 22:40:35.151: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222813, LATS 1792926, lastSeqNo 11222812, uniqueness 1361347261, timestamp 1362321638/975397116

2013-03-03 22:40:35.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:36.155: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222814, LATS 1793936, lastSeqNo 11222813, uniqueness 1361347261, timestamp 1362321639/975398116

2013-03-03 22:40:36.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:37.160: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222815, LATS 1794946, lastSeqNo 11222814, uniqueness 1361347261, timestamp 1362321640/975399116

2013-03-03 22:40:37.320: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:37.819: [    CSSD][2943309584]clssnmSendingThread: sending join msg to all nodes

2013-03-03 22:40:37.819: [    CSSD][2943309584]clssnmSendingThread: sent 5 join msgs to all nodes

2013-03-03 22:40:38.165: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222816, LATS 1795946, lastSeqNo 11222815, uniqueness 1361347261, timestamp 1362321641/975400116

2013-03-03 22:40:38.321: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:39.170: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222817, LATS 1796946, lastSeqNo 11222816, uniqueness 1361347261, timestamp 1362321642/975401126

2013-03-03 22:40:39.321: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:40.104: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b002a2c0

2013-03-03 22:40:40.104: [    CSSD][3031762704]clssgmAllocProc: (0x7f70b011f460) allocated

2013-03-03 22:40:40.105: [    CSSD][3031762704]clssgmClientConnectMsg: properties of cmProc 0x7f70b011f460 - 1,2,3,4,5

2013-03-03 22:40:40.105: [    CSSD][3031762704]clssgmClientConnectMsg: Connect from con(0x3419) proc(0x7f70b011f460) pid(13642) version 11:2:1:4, properties: 1,2,3,4,5

2013-03-03 22:40:40.105: [    CSSD][3031762704]clssgmClientConnectMsg: msg flags 0x0000

2013-03-03 22:40:40.106: [    CSSD][3031762704]clssscSelect: cookie accept request 0x7f70b011f460

2013-03-03 22:40:40.106: [    CSSD][3031762704]clssscevtypSHRCON: getting client with cmproc 0x7f70b011f460

2013-03-03 22:40:40.106: [    CSSD][3031762704]clssgmRegisterClient: proc(4/0x7f70b011f460), client(1/0x7f70b00f6300)

2013-03-03 22:40:40.106: [    CSSD][3031762704]clssgmJoinGrock: global grock CRF- new client 0x7f70b00f6300 with con 0x7f7000003448, requested num -1, flags 0x4000e00

2013-03-03 22:40:40.106: [    CSSD][3031762704]clssgmJoinGrock: ignoring grock join for client not requiring fencing until group information has been received from the master; group name CRF-, member number -1, flags 0x4000e00

2013-03-03 22:40:40.106: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3448

2013-03-03 22:40:40.107: [    CSSD][3031762704]clssgmDeadProc: proc 0x7f70b011f460

2013-03-03 22:40:40.107: [    CSSD][3031762704]clssgmDestroyProc: cleaning up proc(0x7f70b011f460) con(0x3419) skgpid  ospid 13642 with 0 clients, refcount 0

2013-03-03 22:40:40.107: [    CSSD][3031762704]clssgmDiscEndpcl: gipcDestroy 0x3419

2013-03-03 22:40:40.181: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222818, LATS 1797966, lastSeqNo 11222817, uniqueness 1361347261, timestamp 1362321643/975402126

2013-03-03 22:40:40.321: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:41.186: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222819, LATS 1798966, lastSeqNo 11222818, uniqueness 1361347261, timestamp 1362321644/975403126

2013-03-03 22:40:41.321: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0

2013-03-03 22:40:41.814: [    CSSD][2941732624]clssnmRcfgMgrThread: Local Join

2013-03-03 22:40:41.814: [    CSSD][2941732624]clssnmLocalJoinEvent: begin on node(2), waittime 193000

2013-03-03 22:40:41.814: [    CSSD][2941732624]clssnmLocalJoinEvent: set curtime (1799596) for my node

2013-03-03 22:40:41.814: [    CSSD][2941732624]clssnmLocalJoinEvent: scanning 32 nodes

2013-03-03 22:40:41.814: [    CSSD][2941732624]clssnmLocalJoinEvent: Node zxdb01, number 1, is in an existing cluster with disk state 3

2013-03-03 22:40:41.815: [    CSSD][2941732624]clssnmLocalJoinEvent: takeover aborted due to cluster member node found on disk

2013-03-03 22:40:42.190: [    CSSD][2951210768]clssnmvDHBValidateNCopy: node 1, zxdb01, has a disk HB, but no network HB, DHB has rcfg 246562024, wrtcnt, 11222820, LATS 1799966, lastSeqNo 11222819, uniqueness 1361347261, timestamp 1362321645/975404126

2013-03-03 22:40:42.321: [    CSSD][2946463504]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0