首次遭遇ORA-15064

整理一下前段时间遇到的ORA-15064错误。

有一套测试环境是11gR2的rac使用ASM，应用在测试时发现一个节点宕机了，检查了一下alert日志如下错误：

Mon Jun 04 23:27:32 2012

NOTE: ASMB terminating
Errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
Errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
ASMB (ospid: 13828564): terminating the instance due to error 15064
Termination issued to instance processes. Waiting for the processes to exit
Mon Jun 04 23:27:43 2012
Instance termination failed to kill one or more processes

Instance terminated by ASMB, pid = 13828564

通过错误可知，改错无与ASM有关，应该是ASM意外关闭才导致的Instance关闭。

错误的trace文件内容如下：

[pt1rac2@root]# cat /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc

Trace file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /soft/oracle/product/11.2.0.2/dbh
System name:    AIX
Node name:      pt1rac2
Release:        1
Version:        6
Machine:        00F770E14C00
Instance name: pt1db2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 27
Unix process pid: 13828564, image: oracle@pt1rac2 (ASMB)




*** 2012-06-04 16:19:14.483
*** SESSION ID:(2998.1) 2012-06-04 16:19:14.483
*** CLIENT ID:() 2012-06-04 16:19:14.483
*** SERVICE NAME:() 2012-06-04 16:19:14.483
*** MODULE NAME:() 2012-06-04 16:19:14.483
*** ACTION NAME:() 2012-06-04 16:19:14.483
 
NOTE: initiating MARK startup 


*** 2012-06-04 16:19:18.290
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7


*** 2012-06-04 23:07:12.286
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7


*** 2012-06-04 23:27:32.492
NOTE:  ASMB terminating
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 23 Serial number: 1161
error 15064 detected in background process
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 23 Serial number: 1161
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+644<-kjzdssdmp()+444<-kjzduptcctx()+272<-kjzdicrshnfy()+96<-ksuitm()+1284<-ksbrdp()+4508<-opirip()+1216<-opidrv()+1156<-sou2o()+192<-opimai_real()+308<-ssthrdmain()+340<-main()+216<-__start()+112 
----- End of Abridged Call Stack Trace -----


*** 2012-06-04 23:27:32.854
ASMB (ospid: 13828564): terminating the instance due to error 15064


*** 2012-06-04 23:27:43.387
Instance termination failed to kill one or more processes

可知的确是由于ASM意外关闭才导致的instance关闭，ASM意外关闭的原因继续查。

准备生产一下awr报告看一下，结果发现由于数据关闭了，没有awr可看。

一下数据说明数据库被重启过几次，且经过与应用人员确认，这些重启的时间点他们都在做压力测试，因此初步怀疑是和性能有关，继续查。

                                                        Snap
Instance     DB Name        Snap Id    Snap Started    Level
------------ ------------ --------- ------------------ -----
pt1db2       PT1DB              503 02 Jun 2012 16:00      1
                                504 02 Jun 2012 17:00      1


                                505 04 Jun 2012 09:43      1


                                506 04 Jun 2012 11:00      1
                                507 04 Jun 2012 12:00      1
                                508 04 Jun 2012 13:00      1
                                509 04 Jun 2012 14:00      1
                                510 04 Jun 2012 15:00      1
                                511 04 Jun 2012 16:00      1
                                512 04 Jun 2012 17:00      1
                                513 04 Jun 2012 18:00      1
                                514 04 Jun 2012 19:00      1
                                515 04 Jun 2012 20:00      1
                                516 04 Jun 2012 21:00      1
                                517 04 Jun 2012 22:00      1
                                518 04 Jun 2012 23:00      1


                                522 05 Jun 2012 01:20      1
                                523 05 Jun 2012 01:30      1
                                524 05 Jun 2012 01:45      1

先查看ocssd.log日志

2012-06-04 23:26:02.274: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 113598 msecs
2012-06-04 23:26:03.126: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.126: [    CSSD][1029]clssgmAllocProc: (11328bcd0) allocated
2012-06-04 23:26:03.127: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11328bcd0 - 0,1,2,3,4
2012-06-04 23:26:03.127: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea97f) proc(11328bcd0) pid(15335558) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.127: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.274: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:03.274: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:03.309: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.309: [    CSSD][1029]clssgmAllocProc: (1132d30d0) allocated
2012-06-04 23:26:03.310: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132d30d0 - 0,1,2,3,4
2012-06-04 23:26:03.310: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea9fa) proc(1132d30d0) pid(9306434/9306434) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.310: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.315: [    CSSD][1029]clssscSelect: cookie accept request 1132d30d0
2012-06-04 23:26:03.315: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132d30d0
2012-06-04 23:26:03.316: [    CSSD][1029]clssgmRegisterClient: proc(57/1132d30d0), client(1/11313ef70)
2012-06-04 23:26:03.316: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

2012-06-04 23:26:29.658: [    CSSD][1029]clssgmDeadProc: proc 112f32d30
2012-06-04 23:26:29.658: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(112f32d30) con(2ea049) skgpid  ospid 35520704 with 0 clients, refcount 0
2012-06-04 23:26:29.658: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea049
2012-06-04 23:26:29.675: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:29.675: [    CSSD][1029]clssgmAllocProc: (113242130) allocated
2012-06-04 23:26:29.676: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113242130 - 0,1,2,3,4
2012-06-04 23:26:29.676: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eadef) proc(113242130) pid(10879472) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:29.676: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.226: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.226: [    CSSD][1029]clssgmAllocProc: (1132a3390) allocated
2012-06-04 23:26:30.227: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132a3390 - 0,1,2,3,4
2012-06-04 23:26:30.227: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eae64) proc(1132a3390) pid(35520706) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.228: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.542: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.542: [    CSSD][1029]clssgmAllocProc: (1132ad030) allocated
2012-06-04 23:26:30.544: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132ad030 - 0,1,2,3,4
2012-06-04 23:26:30.544: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaede) proc(1132ad030) pid(8585490/8585490) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.544: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.553: [    CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.553: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.553: [    CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(1/1132ad370)
2012-06-04 23:26:30.554: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.554: [    CSSD][1029]clssgmQueueShare: (1132ad510) target global grock DG_DATA member 1 type 1 queued from client (1132ad370), global grock DG_DATA, refcount 15
2012-06-04 23:26:30.554: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 15
2012-06-04 23:26:30.557: [    CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.557: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.557: [    CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(2/1132a3d90)
2012-06-04 23:26:30.558: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.558: [    CSSD][1029]clssgmQueueShare: (1132ad910) target global grock DB+ASM member 1 type 1 queued from client (1132a3d90), global grock DB+ASM, refcount 17
2012-06-04 23:26:30.558: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 17
2012-06-04 23:26:30.636: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.637: [    CSSD][1029]clssgmAllocProc: (11333ba70) allocated
2012-06-04 23:26:30.638: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11333ba70 - 0,1,2,3,4
2012-06-04 23:26:30.638: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaf6f) proc(11333ba70) pid(8913188/8913188) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.638: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.650: [    CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.650: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.650: [    CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(1/112e8e550)
2012-06-04 23:26:30.652: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.652: [    CSSD][1029]clssgmQueueShare: (11328ed90) target global grock DG_DATA member 1 type 1 queued from client (112e8e550), global grock DG_DATA, refcount 16
2012-06-04 23:26:30.652: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 16
2012-06-04 23:26:30.766: [    CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.766: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.766: [    CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(2/11328eff0)
2012-06-04 23:26:30.772: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.772: [    CSSD][1029]clssgmQueueShare: (11328f190) target global grock DB+ASM member 1 type 1 queued from client (11328eff0), global grock DB+ASM, refcount 18
2012-06-04 23:26:30.773: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 18
2012-06-04 23:26:31.382: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:31.382: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:34.394: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 145718 msecs
2012-06-04 23:26:35.392: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:35.392: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:36.396: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 47678 msecs
2012-06-04 23:26:38.571: [    CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 75% maximum time, 200000 ms, will be considered unusable in 49104 ms
2012-06-04 23:26:39.408: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:39.409: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:42.413: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 153738 msecs
2012-06-04 23:26:43.413: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:43.413: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:44.414: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 55696 msecs
2012-06-04 23:26:47.421: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:47.421: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:50.433: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 161758 msecs
2012-06-04 23:26:51.431: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:51.431: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:52.439: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 63721 msecs
2012-06-04 23:26:55.442: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:55.442: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:58.445: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 169771 msecs
2012-06-04 23:26:59.455: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:59.455: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:00.493: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 71776 msecs
2012-06-04 23:27:03.470: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:03.470: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:06.473: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 177799 msecs
2012-06-04 23:27:07.475: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:07.476: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:08.192: [    CSSD][1029]clssgmDeadProc: proc 112f35f50
2012-06-04 23:27:08.192: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(112f35f50) con(2ea340) skgpid  ospid 10551654 with 0 clients, refcount 0
2012-06-04 23:27:08.192: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea340
2012-06-04 23:27:08.192: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 79475 msecs
2012-06-04 23:27:08.498: [    CSSD][1029]clssgmDeadProc: proc 1131b4b10
2012-06-04 23:27:08.498: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1131b4b10) con(2ea3c4) skgpid  ospid 10355074 with 0 clients, refcount 0
2012-06-04 23:27:08.498: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea3c4
2012-06-04 23:27:08.569: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.569: [    CSSD][1029]clssgmAllocProc: (1131fe670) allocated
2012-06-04 23:27:08.571: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1131fe670 - 0,1,2,3,4
2012-06-04 23:27:08.571: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb1f8) proc(1131fe670) pid(10551656) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.571: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.637: [    CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 90% maximum time, 200000 ms, will be considered unusable in 19038 ms
2012-06-04 23:27:08.809: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.810: [    CSSD][1029]clssgmAllocProc: (1132593f0) allocated
2012-06-04 23:27:08.811: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132593f0 - 0,1,2,3,4
2012-06-04 23:27:08.811: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb278) proc(1132593f0) pid(10355076) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.811: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.855: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.855: [    CSSD][1029]clssgmAllocProc: (113259730) allocated
2012-06-04 23:27:08.857: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113259730 - 0,1,2,3,4
2012-06-04 23:27:08.858: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb2ec) proc(113259730) pid(8192508/8192508) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.858: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.867: [    CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.867: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.867: [    CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(1/112f35f50)
2012-06-04 23:27:08.868: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:08.868: [    CSSD][1029]clssgmQueueShare: (11334b430) target global grock DG_DATA member 1 type 1 queued from client (112f35f50), global grock DG_DATA, refcount 17
2012-06-04 23:27:08.868: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 17
2012-06-04 23:27:08.872: [    CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.872: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.872: [    CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(2/113215ed0)
2012-06-04 23:27:08.874: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:08.874: [    CSSD][1029]clssgmQueueShare: (11334b690) target global grock DB+ASM member 1 type 1 queued from client (113215ed0), global grock DB+ASM, refcount 19
2012-06-04 23:27:08.874: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 19
2012-06-04 23:27:09.000: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:09.000: [    CSSD][1029]clssgmAllocProc: (11334b8f0) allocated
2012-06-04 23:27:09.001: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11334b8f0 - 0,1,2,3,4
2012-06-04 23:27:09.001: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb37d) proc(11334b8f0) pid(8061184/8061184) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:09.001: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:09.010: [    CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.010: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.010: [    CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(1/1131b5310)
2012-06-04 23:27:09.011: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:09.011: [    CSSD][1029]clssgmQueueShare: (11334bc30) target global grock DG_DATA member 1 type 1 queued from client (1131b5310), global grock DG_DATA, refcount 18
2012-06-04 23:27:09.011: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 18
2012-06-04 23:27:09.014: [    CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.014: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.014: [    CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(2/11334be90)
2012-06-04 23:27:09.015: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:09.015: [    CSSD][1029]clssgmQueueShare: (113350230) target global grock DB+ASM member 1 type 1 queued from client (11334be90), global grock DB+ASM, refcount 20
2012-06-04 23:27:09.015: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 20
2012-06-04 23:27:11.480: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:11.480: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:14.494: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 185820 msecs
2012-06-04 23:27:15.500: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:15.500: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:16.504: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 87787 msecs
2012-06-04 23:27:19.519: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:19.519: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:20.285: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.285: [    CSSD][1029]clssgmAllocProc: (1132b9290) allocated
2012-06-04 23:27:20.294: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b9290 - 0,1,2,3,4
2012-06-04 23:27:20.294: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb49d) proc(1132b9290) pid(8257826) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:20.294: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:20.299: [    CSSD][1029]clssnmGetNodeNumber: pt1rac2 
2012-06-04 23:27:20.457: [    CSSD][1029]clssgmDeadProc: proc 1132b9290
2012-06-04 23:27:20.457: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b9290) con(2eb49d) skgpid  ospid 8257826 with 0 clients, refcount 0
2012-06-04 23:27:20.457: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb49d
2012-06-04 23:27:20.999: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.999: [    CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:21.000: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 0,1,2,3,4
2012-06-04 23:27:21.000: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb51a) proc(1132b2650) pid(7733742) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:21.000: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:21.005: [    CSSD][1029]clssnmGetNodeNumber: pt1rac1 
2012-06-04 23:27:21.289: [    CSSD][1029]clssgmDeadProc: proc 1132b2650
2012-06-04 23:27:21.290: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b2650) con(2eb51a) skgpid  ospid 7733742 with 0 clients, refcount 0
2012-06-04 23:27:21.290: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb51a
2012-06-04 23:27:22.001: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 193327 msecs
2012-06-04 23:27:23.533: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:23.534: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:24.945: [    CSSD][1029]clssgmDeadProc: proc 113248030
2012-06-04 23:27:24.945: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(113248030) con(2ea626) skgpid  ospid 9699686 with 0 clients, refcount 0
2012-06-04 23:27:24.945: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea626
2012-06-04 23:27:25.142: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.143: [    CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:25.143: [    CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 96425 msecs
2012-06-04 23:27:25.144: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 1,2,3,4
2012-06-04 23:27:25.144: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb5c3) proc(1132b2650) pid(9699688) version 11:2:1:4, properties: 1,2,3,4
2012-06-04 23:27:25.144: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.323: [    CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.324: [    CSSD][1029]clssgmAllocProc: (1132b7e50) allocated
2012-06-04 23:27:25.326: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b7e50 - 0,1,2,3,4
2012-06-04 23:27:25.326: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb63a) proc(1132b7e50) pid(7733746/7733746) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:25.326: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.485: [    CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.485: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.485: [    CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(1/113153d90)
2012-06-04 23:27:25.487: [    CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:25.487: [    CSSD][1029]clssgmQueueShare: (113338450) target global grock DG_DATA member 1 type 1 queued from client (113153d90), global grock DG_DATA, refcount 19
2012-06-04 23:27:25.487: [    CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 19
2012-06-04 23:27:25.495: [    CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.495: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.495: [    CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(2/1132b42b0)
2012-06-04 23:27:25.497: [    CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:25.497: [    CSSD][1029]clssgmQueueShare: (11334c0f0) target global grock DB+ASM member 1 type 1 queued from client (1132b42b0), global grock DB+ASM, refcount 21
2012-06-04 23:27:25.497: [    CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 21
2012-06-04 23:27:27.542: [    CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:27.542: [    CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:27.680: [    CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200006 ms for voting file /dev/rhdisk1)
2012-06-04 23:27:27.693: [    CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/rhdisk1 now offline
2012-06-04 23:27:27.693: [    CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2012-06-04 23:27:27.694: [    CSSD][2834]###################################
2012-06-04 23:27:27.694: [    CSSD][2834] clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.694: [    CSSD][2834]###################################
2012-06-04 23:27:27.694: [    CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2012-06-04 23:27:27.695: [    CSSD][2834]


----- Call Stack Trace -----
2012-06-04 23:27:27.723: [    CSSD][2834]calling              call     entry                argument values in hex      
2012-06-04 23:27:27.723: [    CSSD][2834]location             type     point                (? means dubious value)     
2012-06-04 23:27:27.723: [    CSSD][2834]-------------------- -------- -------------------- ----------------------------
2012-06-04 23:27:27.724: [    CSSD][2834]clssscExit()+692     bl       100067114            67800000678 ? 1000EB40C ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   110091190 ? 900000000BA4D14 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   000000000 ? 013960031 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [    CSSD][2834]clssnmvDiskCheck()+  call     clssscExit()         111E38B90 ? 200000002 ?
2012-06-04 23:27:27.724: [    CSSD][2834]2576                                               000000001 ? 1000F52A0 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   110091190 ? 013960031 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [    CSSD][2834]clssnmvDiskPingMoni  call     clssnmvDiskCheck()   111E38B90 ? 110BF7FB0 ?
2012-06-04 23:27:27.724: [    CSSD][2834]torThread()+392                                    112468680 ? 000000000 ?
2012-06-04 23:27:27.724: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   013960031 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]clssscthrdmain()+20  call     clssnmvDiskPingMoni  111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]8                             torThread()          000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]_pthread_body()+240  call     clssscthrdmain()     111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]fffffffffffffffc     call     _pthread_body()      000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [    CSSD][2834]                                                   000000000 ? 000000000 ?
2012-06-04 23:27:27.726: [    CSSD][2834] 
2012-06-04 23:27:27.726: [    CSSD][2834]--------------------- Binary Stack Dump ---------------------
2012-06-04 23:27:27.726: [    CSSD][2834] 
2012-06-04 23:27:27.726: [    CSSD][2834]========== FRAME [1] (clssscExit()+692 -> 100067114) ==========
2012-06-04 23:27:27.726: [    CSSD][2834]defined by frame pointers 0x1124683f0  and 0x112468260
2012-06-04 23:27:27.726: [    CSSD][2834]CALL TYPE: bl   ERROR SIGNALED: no   CALLER: clssscExit
2012-06-04 23:27:27.726: [    CSSD][2834]
Dump of memory from 0x112468260 to 0x1124683f0
2012-06-04 23:27:27.726: [    CSSD][2834]112468260 00000001 124683F0 44000224 000003D7  [.....F..D..$....]
2012-06-04 23:27:27.726: [    CSSD][2834]112468270 00000001 000031D8 00000000 00000083  [......1.........]
2012-06-04 23:27:27.726: [    CSSD][2834]112468280 00000000 00000029 00000001 100182B0  [.......)........]
2012-06-04 23:27:27.726: [    CSSD][2834]112468290 00000678 00000678 00000001 000EB40C  [...x...x........]
2012-06-04 23:27:27.726: [    CSSD][2834]1124682A0 00000001 10091190 09000000 00BA4D14  [..............M.]
2012-06-04 23:27:27.726: [    CSSD][2834]1124682B0 00000000 00000000 00000000 13960031  [...............1]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682C0 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682D0 00000001 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682E0 00000000 00000001 43525300 12468368  [........CRS..F.h]
2012-06-04 23:27:27.727: [    CSSD][2834]1124682F0 00000001 00000000 00000001 100181F0  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468300 00000001 10018208 00000001 100181F0  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468310 00000001 10018208 00000001 11E38B90  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468320 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.727: [    CSSD][2834]112468330 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.728: [    CSSD][2834]        Repeat 3 times
2012-06-04 23:27:27.728: [    CSSD][2834]112468370 00000000 00000000 00000000 13960031  [...............1]
2012-06-04 23:27:27.728: [    CSSD][2834]112468380 09000000 00C27160 00000000 00000000  [......q`........]
2012-06-04 23:27:27.728: [    CSSD][2834]112468390 00000001 1002BD70 00000001 117B6590  [.......p.....{e.]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683A0 00000001 00000001 00000000 00000000  [................]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683B0 00000000 00000000 00000646 00000646  [...........F...F]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683C0 00000001 12468428 00000001 117C6C84  [.....F.(.....|l.]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683D0 00000001 117C6C84 00000001 1008FF50  [.....|l........P]
2012-06-04 23:27:27.728: [    CSSD][2834]1124683E0 00000000 00000000 00000001 000F24E8  [..............$.]
2012-06-04 23:27:27.728: [    CSSD][2834]========== FRAME [2] (clssnmvDiskCheck()+2576 -> clssscExit()) ==========
2012-06-04 23:27:27.728: [    CSSD][2834]defined by frame pointers 0x112468600  and 0x1124683f0
2012-06-04 23:27:27.729: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: clssnmvDiskCheck
2012-06-04 23:27:27.729: [    CSSD][2834]
Dump of memory from 0x1124683f0 to 0x112468600
2012-06-04 23:27:27.729: [    CSSD][2834]1124683F0 00000001 12468600 09001000 A0688640  [.....F.......h.@]
2012-06-04 23:27:27.729: [    CSSD][2834]112468400 00000001 0002CD94 4400B22B 00000000  [........D..+....]
2012-06-04 23:27:27.729: [    CSSD][2834]112468410 00000001 001058B8 00000001 100182B0  [......X.........]
2012-06-04 23:27:27.729: [    CSSD][2834]112468420 00000001 11E38B90 00000002 00000002  [................]
2012-06-04 23:27:27.729: [    CSSD][2834]112468430 00000000 00000001 00000001 000F52A0  [..............R.]
2012-06-04 23:27:27.729: [    CSSD][2834]112468440 00000001 10091190 00000000 13960031  [...............1]
2012-06-04 23:27:27.729: [    CSSD][2834]112468450 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.730: [    CSSD][2834]112468460 00000001 1008FF50 00000001 118508F0  [.......P........]
2012-06-04 23:27:27.730: [    CSSD][2834]112468470 00000001 117B6590 8C91440C 00030D46  [.....{e...D....F]
2012-06-04 23:27:27.730: [    CSSD][2834]112468480 00000001 12C508B0 00000000 00030D40  [...............@]
2012-06-04 23:27:27.730: [    CSSD][2834]112468490 00010001 9DD81012 D0E54F97 BFEE0F2C  [..........O....,]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684A0 AF8AA88C 00000000 2200088B 00000000  [........".......]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684B0 09000000 014F0814 09001000 A0688640  [.....O.......h.@]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684C0 00030D40 1045F530 00000004 00000004  [...@.E.0........]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684D0 00000001 00105DE8 00000001 000F2BD4  [......].......+.]
2012-06-04 23:27:27.730: [    CSSD][2834]1124684E0 00000001 000F2BD4 00030D40 000F24E8  [......+....@..$.]
2012-06-04 23:27:27.731: [    CSSD][2834]1124684F0 00000001 00000001 2200022B 00000000  [........"..+....]
2012-06-04 23:27:27.731: [    CSSD][2834]112468500 00000000 00000000 FFFFFFF5 000F24E8  [..............$.]
2012-06-04 23:27:27.731: [    CSSD][2834]112468510 00000001 12468600 09001000 A0688640  [.....F.......h.@]
2012-06-04 23:27:27.731: [    CSSD][2834]112468520 00000001 00070DFC 00000001 117C6C84  [.............|l.]
2012-06-04 23:27:27.731: [    CSSD][2834]112468530 00000001 117C6C84 00000001 100182B0  [.....|l.........]
2012-06-04 23:27:27.731: [    CSSD][2834]112468540 00000001 1045F530 00000004 00000004  [.....E.0........]
2012-06-04 23:27:27.731: [    CSSD][2834]112468550 00000001 00105DE8 00000000 00000009  [......].........]
2012-06-04 23:27:27.731: [    CSSD][2834]112468560 00000000 000003E8 00000000 000003E8  [................]
2012-06-04 23:27:27.731: [    CSSD][2834]112468570 00000000 13960031 00000000 00000040  [.......1.......@]
2012-06-04 23:27:27.732: [    CSSD][2834]112468580 00000001 1008FF50 00000001 117B6590  [.......P.....{e.]
2012-06-04 23:27:27.732: [    CSSD][2834]112468590 00000001 1002BD70 00000001 117B65A0  [.......p.....{e.]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685A0 00000001 117B66A0 FFFFFFB2 000003D4  [.....{f.........]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685B0 00000001 117C6C84 00000001 117C6C84  [.....|l......|l.]
2012-06-04 23:27:27.732: [    CSSD][2834]        Repeat 1 times
2012-06-04 23:27:27.732: [    CSSD][2834]1124685D0 00000001 000F2BD4 00000001 000F2BD4  [......+.......+.]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685E0 00000000 00000000 00000001 12C508B0  [................]
2012-06-04 23:27:27.732: [    CSSD][2834]1124685F0 00000000 00000000 00000001 000F24E8  [..............$.]
2012-06-04 23:27:27.732: [    CSSD][2834]========== FRAME [3] (clssnmvDiskPingMonitorThread()+392 -> clssnmvDiskCheck()) ==========
2012-06-04 23:27:27.732: [    CSSD][2834]defined by frame pointers 0x1124686f0  and 0x112468600
2012-06-04 23:27:27.732: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.732: [    CSSD][2834]
Dump of memory from 0x112468600 to 0x1124686f0
2012-06-04 23:27:27.733: [    CSSD][2834]112468600 00000001 124686F0 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468610 00000001 0002CFAC 09001000 A0688640  [.............h.@]
2012-06-04 23:27:27.733: [    CSSD][2834]112468620 00000000 00000000 00000001 100182B0  [................]
2012-06-04 23:27:27.733: [    CSSD][2834]112468630 00000001 11E38B90 00000001 10BF7FB0  [................]
2012-06-04 23:27:27.733: [    CSSD][2834]112468640 00000001 12468680 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468650 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.733: [    CSSD][2834]112468660 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590  [.............{e.]
2012-06-04 23:27:27.733: [    CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0  [...@..@$........]
2012-06-04 23:27:27.733: [    CSSD][2834]112468690 09000000 014F0814 00000000 00000000  [.....O..........]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686A0 00000000 00000000 00000000 00000001  [................]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8  [................]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.734: [    CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78  [.............F.x]
2012-06-04 23:27:27.734: [    CSSD][2834]========== FRAME [4] (clssscthrdmain()+208 -> clssnmvDiskPingMonitorThread()) ==========
2012-06-04 23:27:27.734: [    CSSD][2834]defined by frame pointers 0x112468790  and 0x1124686f0
2012-06-04 23:27:27.734: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: clssscthrdmain
2012-06-04 23:27:27.734: [    CSSD][2834]
Dump of memory from 0x1124686f0 to 0x112468790
2012-06-04 23:27:27.734: [    CSSD][2834]1124686F0 00000001 12468790 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.734: [    CSSD][2834]112468700 00000001 0000A294 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468710 00000000 00000000 00000001 100182B0  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468720 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468730 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]        Repeat 2 times
2012-06-04 23:27:27.735: [    CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.735: [    CSSD][2834]112468770 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.735: [    CSSD][2834]112468780 00000000 00000000 00000001 1008FF50  [...............P]
2012-06-04 23:27:27.735: [    CSSD][2834]========== FRAME [5] (_pthread_body()+240 -> clssscthrdmain()) ==========
2012-06-04 23:27:27.735: [    CSSD][2834]defined by frame pointers 0x112468810  and 0x112468790
2012-06-04 23:27:27.735: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: _pthread_body
2012-06-04 23:27:27.735: [    CSSD][2834]
Dump of memory from 0x112468790 to 0x112468890
2012-06-04 23:27:27.736: [    CSSD][2834]112468790 00000001 12468810 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687A0 09000000 00B7BD54 00000000 00000000  [.......T........]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687B0 00000000 00000000 09001000 A0288350  [.............(.P]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687C0 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.736: [    CSSD][2834]1124687D0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.737: [    CSSD][2834]  Repeat 11 times
2012-06-04 23:27:27.737: [    CSSD][2834]========== FRAME [6] (fffffffffffffffc -> _pthread_body()) ==========
2012-06-04 23:27:27.737: [    CSSD][2834]defined by frame pointers 0x0  and 0x112468810
2012-06-04 23:27:27.737: [    CSSD][2834]CALL TYPE: call   ERROR SIGNALED: no   CALLER: fffffffffffffffc
2012-06-04 23:27:27.737: [    CSSD][2834]
----- Argument/Register Address Dump -----


2012-06-04 23:27:27.737: [    CSSD][2834]Argument/Register addr=0x1000eb40c.
Dump of memory from 0x1000eb3cc to 0x1000eb50c
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3C0                            6C206572              [l er]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3D0 726F7220 6F636375 72726564 20616E64  [ror occurred and]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3E0 20746865 20435353 20646165 6D6F6E20  [ the CSS daemon ]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB3F0 69732074 65726D69 6E617469 6E672061  [is terminating a]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB400 626E6F72 6D616C6C 79004942 283A4353  [bnormally.IB(:CS]
2012-06-04 23:27:27.737: [    CSSD][2834]1000EB410 53534330 30303132 3A290049 636C7373  [SSC00012:).Iclss]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB420 73634578 69743A20 53746172 74696E67  [scExit: Starting]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB430 20435253 4420636C 65616E75 70004942  [ CRSD cleanup.IB]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB440 636C7373 73634578 69743A20 43525344  [clssscExit: CRSD]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB450 20636C65 616E7570 20737461 74757320  [ cleanup status ]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB460 25640049 636C7373 73634578 69743A20  [%d.IclssscExit: ]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB470 43525344 20636C65 616E7570 20666169  [CRSD cleanup fai]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB480 6C656420 77697468 20256400 636C7373  [led with %d.clss]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB490 73634578 69743A20 43525344 20636C65  [scExit: CRSD cle]
2012-06-04 23:27:27.738: [    CSSD][2834]1000EB4A0 616E7570 20737563 63657373 66756C6C  [anup successfull]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4B0 7920636F 6D706C65 74656400 636C7373  [y completed.clss]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4C0 73634578 69743A20 61626F72 7420616C  [scExit: abort al]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4D0 72656164 79207365 74202564 0049424D  [ready set %d.IBM]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4E0 41535345 52542025 73202564 0049424D  [ASSERT %s %d.IBM]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB4F0 636C7373 73632E63 0049424D 283A4353  [clsssc.c.IBM(:CS]
2012-06-04 23:27:27.739: [    CSSD][2834]1000EB500 53534330 30303133 3A29636C           [SSC00013:)cl]    
2012-06-04 23:27:27.739: [    CSSD][2834]Argument/Register addr=0x110091190.
Dump of memory from 0x110091150 to 0x110091290
2012-06-04 23:27:27.739: [    CSSD][2834]110091150 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.739: [    CSSD][2834]        Repeat 2 times
2012-06-04 23:27:27.739: [    CSSD][2834]110091180 00000000 00000000 00000001 00000001  [................]
2012-06-04 23:27:27.740: [    CSSD][2834]110091190 2F736F66 742F6772 69642F70 726F6475  [/soft/grid/produ]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911A0 63742F31 312E322E 302E322F 6769682F  [ct/11.2.0.2/gih/]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911B0 6C6F672F 70743172 6163322F 63737364  [log/pt1rac2/cssd]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911C0 2F6F6373 73642E6C 6F670000 00000000  [/ocssd.log......]
2012-06-04 23:27:27.740: [    CSSD][2834]1100911D0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.741: [    CSSD][2834]  Repeat 11 times
2012-06-04 23:27:27.741: [    CSSD][2834]Argument/Register addr=0x900000000ba4d14.
Dump of memory from 0x900000000ba4cd4 to 0x900000000ba4e14
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4CD0          38600000 4BFD5608 00000000      [8`..K.V.....]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4CE0 000C0000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4CF0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D00 348D8278 41810008 38800000 E8630000  [4..xA...8....c..]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D10 38A00008 81840230 7C0004AC 398CFFFF  [8......0|...9...]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D20 E9040110 38E00000 65081000 7D251828  [....8...e...}%.(]
2012-06-04 23:27:27.741: [    CSSD][2834]900000000BA4D30 7C894000 40860014 7CE5192D 40A2FFF0  [|.@.@...|..-@...]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D40 91840230 4E800020 7C632A14 4BFD4694  [...0N.. |c*.K.F.]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D50 00000000 000C0000 00000000 00000000  [................]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D60 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.742: [    CSSD][2834]        Repeat 1 times
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D80 7D251828 64871000 2C890000 40860014  [}%.(d...,...@...]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4D90 7CE5192D 4C00012C 4D820020 4BFFFFE4  [|..-L..,M.. K...]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4DA0 7C862378 7C851A14 7CC53378 4BFD5534  [|.#x|...|.3xK.U4]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4DB0 00000000 000C0000 00000000 00000000  [................]
2012-06-04 23:27:27.742: [    CSSD][2834]900000000BA4DC0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.743: [    CSSD][2834]        Repeat 3 times
2012-06-04 23:27:27.743: [    CSSD][2834]900000000BA4E00 7D251828 64871000 2C890000 40860014  [}%.(d...,...@...]
2012-06-04 23:27:27.743: [    CSSD][2834]900000000BA4E10 7CE5192D                             [|..-]            
2012-06-04 23:27:27.743: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.743: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.744: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.744: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.744: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.745: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.745: [    CSSD][2834]Argument/Register addr=0x111e38b90.
Dump of memory from 0x111e38b50 to 0x111e38c90
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B50 00000001 00000001 8C8FC179 000003E8  [...........y....]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B60 8C914038 8C913A22 0000000B 00000000  [..@8..:"........]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B70 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B80 5B5B5B5B 00000000 00000000 000001E0  [[[[[............]
2012-06-04 23:27:27.745: [    CSSD][2834]111E38B90 00000001 1008FF50 00000001 10BF7FB0  [.......P........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BA0 00000001 11E3A320 00000001 11E389A0  [....... ........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BB0 00000B12 00000B12 636C7373 6E6D7644  [........clssnmvD]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BC0 69736B50 696E674D 6F6E6974 6F725468  [iskPingMonitorTh]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BD0 72656164 00000000 00000000 00000000  [read............]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BE0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38BF0 00000000 00000000 00020000 00000000  [................]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38C00 00000001 1045F530 00000001 11E38B90  [.....E.0........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38C10 00000001 100179B0 00000000 00000000  [......y.........]
2012-06-04 23:27:27.746: [    CSSD][2834]111E38C20 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.747: [    CSSD][2834]        Repeat 4 times
2012-06-04 23:27:27.747: [    CSSD][2834]111E38C70 00000000 00000001 00000000 00000005  [................]
2012-06-04 23:27:27.747: [    CSSD][2834]111E38C80 00000000 00001677 00000000 00000000  [.......w........]
2012-06-04 23:27:27.747: [    CSSD][2834]Argument/Register addr=0x1.
Dump of memory from 0x0 to 0x101
2012-06-04 23:27:27.747: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.748: [    CSSD][2834]        Repeat 15 times
2012-06-04 23:27:27.748: [    CSSD][2834]000000100 7C7243A6                             [|rC.]            
2012-06-04 23:27:27.748: [    CSSD][2834]Argument/Register addr=0x1000f52a0.
Dump of memory from 0x1000f5260 to 0x1000f53a0
2012-06-04 23:27:27.748: [    CSSD][2834]1000F5260 6B436865 636B3A20 41626F72 74696E67  [kCheck: Aborting]
2012-06-04 23:27:27.748: [    CSSD][2834]1000F5270 2C202564 206F6620 25642025 7320766F  [, %d of %d %s vo]
2012-06-04 23:27:27.748: [    CSSD][2834]1000F5280 74696E67 20646973 6B732061 7661696C  [ting disks avail]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5290 61626C65 2C206E65 65642025 75004942  [able, need %u.IB]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52A0 283A4353 534E4D30 30303138 3A290049  [(:CSSNM00018:).I]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52B0 636C7373 6E6D7644 69736B50 696E674D  [clssnmvDiskPingM]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52C0 6F6E6974 6F725468 72656164 3A20206E  [onitorThread:  n]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52D0 6F20766F 74696E67 20646973 6B2C0049  [o voting disk,.I]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52E0 636C7373 6E6D7644 69736B50 4D543A20  [clssnmvDiskPMT: ]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F52F0 736C6565 70696E67 20666F72 20256420  [sleeping for %d ]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5300 6D730049 636C7373 6E6D7653 63616E43  [ms.IclssnmvScanC]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5310 6F6D706C 6574696F 6E733A20 4661696C  [ompletions: Fail]
2012-06-04 23:27:27.749: [    CSSD][2834]1000F5320 75726520 2564206E 6F746966 79696E67  [ure %d notifying]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5330 20636C69 656E7473 206F6620 776F726B  [ clients of work]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5340 20636F6D 706C6574 696F6E2E 0049424D  [ completion..IBM]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5350 636C7373 6E6D7653 63616E43 6F6D706C  [clssnmvScanCompl]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5360 6574696F 6E733A20 636F6D70 6C657465  [etions: complete]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5370 64202564 20697465 6D730049 636C7373  [d %d items.Iclss]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5380 6E6D7652 6561644C 65617365 3A20656E  [nmvReadLease: en]
2012-06-04 23:27:27.750: [    CSSD][2834]1000F5390 74657200 636C7373 6E6D7652 6561644C  [ter.clssnmvReadL]
2012-06-04 23:27:27.750: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.750: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.751: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.751: [    CSSD][2834]Argument/Register addr=0x110bf7fb0.
Dump of memory from 0x110bf7f70 to 0x110bf80b0
2012-06-04 23:27:27.751: [    CSSD][2834]110BF7F70 00000001 10BD5F90 00000001 10BF7F70  [......_........p]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7F80 00000000 00000000 00000000 00022000  [.............. .]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7F90 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7FA0 5B5B5B5B 00000000 00000000 00002938  [[[[[..........)8]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7FB0 4E4D6378 00000000 00000001 1008FF50  [NMcx...........P]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7FC0 00000001 00000100 00000000 00000000  [................]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7FD0 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.752: [    CSSD][2834]        Repeat 1 times
2012-06-04 23:27:27.752: [    CSSD][2834]110BF7FF0 00000000 00000000 00000000 00000001  [................]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF8000 00000000 00000005 00000000 00000A14  [................]
2012-06-04 23:27:27.752: [    CSSD][2834]110BF8010 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.753: [    CSSD][2834]  Repeat 9 times
2012-06-04 23:27:27.753: [    CSSD][2834]Argument/Register addr=0x112468680.
Dump of memory from 0x112468640 to 0x112468780
2012-06-04 23:27:27.753: [    CSSD][2834]112468640 00000001 12468680 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.753: [    CSSD][2834]112468650 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.753: [    CSSD][2834]112468660 00000000 13960031 00000000 00000000  [.......1........]
2012-06-04 23:27:27.754: [    CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590  [.............{e.]
2012-06-04 23:27:27.754: [    CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0  [...@..@$........]
2012-06-04 23:27:27.754: [    CSSD][2834]112468690 09000000 014F0814 00000000 00000000  [.....O..........]
2012-06-04 23:27:27.754: [    CSSD][2834]1124686A0 00000000 00000000 00000000 00000001  [................]
2012-06-04 23:27:27.754: [    CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.754: [    CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8  [................]
2012-06-04 23:27:27.754: [    CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.754: [    CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78  [.............F.x]
2012-06-04 23:27:27.754: [    CSSD][2834]1124686F0 00000001 12468790 00000000 00000000  [.....F..........]
2012-06-04 23:27:27.754: [    CSSD][2834]112468700 00000001 0000A294 00000000 00000000  [................]
2012-06-04 23:27:27.755: [    CSSD][2834]112468710 00000000 00000000 00000001 100182B0  [................]
2012-06-04 23:27:27.755: [    CSSD][2834]112468720 00000001 11E38B90 00000000 00000000  [................]
2012-06-04 23:27:27.755: [    CSSD][2834]112468730 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.755: [    CSSD][2834]        Repeat 2 times
2012-06-04 23:27:27.755: [    CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70  [.......P.......p]
2012-06-04 23:27:27.755: [    CSSD][2834]112468770 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.755: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.755: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.756: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.756: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.756: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.757: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.757: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.757: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.758: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.758: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.759: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.759: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.759: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.760: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.760: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.761: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.761: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.762: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.762: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.762: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.763: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.763: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.763: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.764: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.764: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.764: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.765: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.765: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.765: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.766: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.766: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.766: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.767: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.767: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.767: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.768: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.768: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.768: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.769: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.769: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.769: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.770: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.770: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.770: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.771: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.771: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.771: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.772: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.772: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.772: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.773: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.773: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.773: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.774: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.774: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.775: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.776: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.776: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.776: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.777: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.777: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.777: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.778: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.778: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.778: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.779: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.779: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.779: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.780: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.780: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.780: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.781: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.781: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.781: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.782: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.782: [    CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.782: [    CSSD][2834]000000000 00000000 00000000 00000000 00000000  [................]
2012-06-04 23:27:27.783: [    CSSD][2834]  Repeat 15 times
2012-06-04 23:27:27.783: [    CSSD][2834]


call stack performance statistics:
2012-06-04 23:27:27.783: [    CSSD][2834]total                  : 0.088000 sec
2012-06-04 23:27:27.783: [    CSSD][2834]setup                  : 0.028000 sec
2012-06-04 23:27:27.783: [    CSSD][2834]stack unwind           : 0.000000 sec
2012-06-04 23:27:27.783: [    CSSD][2834]symbol translation     : 0.000000 sec
2012-06-04 23:27:27.783: [    CSSD][2834]printing the call stack: 0.002000 sec
2012-06-04 23:27:27.784: [    CSSD][2834]printing frame data    : 0.012000 sec
2012-06-04 23:27:27.784: [    CSSD][2834]printing argument data : 0.046000 sec
2012-06-04 23:27:27.784: [    CSSD][2834]


----- End of Call Stack Trace -----


2012-06-04 23:27:27.784: [    CSSD][2834]clssnmSendMeltdownStatus: node pt1rac2, number 2, has experienced a failure in thread number 9 and is shutting down
2012-06-04 23:27:27.784: [    CSSD][2834]clssscExit: Starting CRSD cleanup
2012-06-04 23:27:28.284: [   SKGFD][2320]Lib :UFS:: closing handle 112115390 for disk :/dev/rhdisk1:


2012-06-04 23:27:30.363: [    CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 201689 msecs
2012-06-04 23:27:31.194: [    CSSD][2834]clssscExit: CRSD cleanup status 0
2012-06-04 23:27:31.194: [    CSSD][2834]clssscExit: CRSD cleanup successfully completed
2012-06-04 23:27:31.198: [ default][2834]kgzf_gen_node_reid2: generated reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100


2012-06-04 23:27:31.198: [ default][2834]kgzf_send_fence4: queued fence for reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100, handle 2


2012-06-04 23:27:31.198: [    CSSD][2834]clssnmrFenceSage: Fenced node pt1rac2, number 2, with EXADATA, handle 2
2012-06-04 23:27:31.198: [    CSSD][2834]clssgmUpdateEventValue: CmInfo State  val 0, changes 60
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: Aborting client (11188d7d0) proc (112c52170), iocapables 1.
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c52170), pid (56951010), iocapables 1, client (11188d7d0)
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e154f0) proc (112fd5c10), iocapables 2.
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fd5c10), pid (52691020), iocapables 2, client (111e154f0)
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: Aborting client (11189b310) proc (112c524b0), iocapables 3.
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c524b0), pid (50200582), iocapables 3, client (11189b310)
2012-06-04 23:27:31.200: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e16130) proc (112f90870), iocapables 4.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 4, client (111e16130)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (111863110) proc (112f90870), iocapables 5.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 5, client (111863110)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (111862ad0) proc (112f90870), iocapables 6.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 6, client (111862ad0)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (111869530) proc (112f90870), iocapables 7.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 7, client (111869530)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e1ddf0) proc (112c65490), iocapables 8.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 8, client (111e1ddf0)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (11186a570) proc (112c65490), iocapables 9.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 9, client (11186a570)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (1118644b0) proc (112fa6d10), iocapables 10.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 10, client (1118644b0)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (111ef2570) proc (112fa6d10), iocapables 11.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 11, client (111ef2570)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (11186d210) proc (112fa6d10), iocapables 12.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 12, client (11186d210)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 13.
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 13, client (112f97d30)
2012-06-04 23:27:31.201: [    CSSD][1029]clssgmSendShutdown: Aborting client (1118a1c90) proc (113039a10), iocapables 14.
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 14, client (1118a1c90)
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e694d0) proc (11305be70), iocapables 15.
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (11305be70), pid (46268552), iocapables 15, client (111e694d0)
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: Aborting client (112c4f470) proc (112ea2b90), iocapables 16.
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 16, client (112c4f470)
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: Aborting client (112c4cef0) proc (112ea2b90), iocapables 17.
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 17, client (112c4cef0)
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: Aborting client (111867490) proc (112c77ff0), iocapables 18.
2012-06-04 23:27:31.204: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c77ff0), pid (15925342), iocapables 18, client (111867490)
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e204f0) proc (112ea1150), iocapables 19.
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea1150), pid (60031284), iocapables 19, client (111e204f0)
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e4eb70) proc (112ebec30), iocapables 20.
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ebec30), pid (34472120), iocapables 20, client (111e4eb70)
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e62d30) proc (112edbcb0), iocapables 21.
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112edbcb0), pid (41812046), iocapables 21, client (111e62d30)
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e56010) proc (112efd370), iocapables 22.
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 22, client (111e56010)
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 23.
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 23, client (111e5fd90)
2012-06-04 23:27:31.205: [    CSSD][1029]clssgmSendShutdown: Aborting client (112ed4510) proc (112f05c70), iocapables 24.
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 24, client (112ed4510)
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: Aborting client (112c485b0) proc (112f05c70), iocapables 25.
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 25, client (112c485b0)
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: Aborting client (112eb7310) proc (112f05c70), iocapables 26.
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 26, client (112eb7310)
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: Aborting client (112e95a70) proc (112f05c70), iocapables 27.
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 27, client (112e95a70)
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: Aborting client (112c68030) proc (112f05c70), iocapables 28.
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 28, client (112c68030)
2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: Aborting client (112c71a90) proc (112ef8f10), iocapables 29.

2012-06-04 23:27:31.206: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ef8f10), pid (40370574), iocapables 29, client (112c71a90)

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

2012-06-04 23:27:31.273: [    CSSD][1029]clssgmDeadProc: proc 112c524b0
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmFenceClient: fencing client (11189b310), member 0 in group GR+GCR1, no share, death fence 1, SAGE fence 0
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmUnreferenceMember: global grock GR+GCR1 member 0 refcount is 1
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmFenceProcessDeath: client (11189b310) pid 50200582 undead
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmQueueFenceForCheck: (1117e0630) Death check for object type 3, pid 50200582
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 16f1
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(112c524b0) con(16c2) skgpid 50200582 ospid 50200582 with 0 clients, refcount 0
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 16c2
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c52170), pid (56951010), iocapables 1, client (11188d7d0)
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 2, client (111e1ddf0)
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 3, client (11186a570)
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: Aborting client (1118644b0) proc (112fa6d10), iocapables 4.
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 4, client (1118644b0)
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: Aborting client (111ef2570) proc (112fa6d10), iocapables 5.
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 5, client (111ef2570)
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: Aborting client (11186d210) proc (112fa6d10), iocapables 6.
2012-06-04 23:27:31.273: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 6, client (11186d210)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 7.
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 7, client (112f97d30)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 8, client (1118a1c90)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 9, client (112c4f470)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 10, client (112c4cef0)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112c77ff0), pid (15925342), iocapables 11, client (111867490)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea1150), pid (60031284), iocapables 12, client (111e204f0)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ebec30), pid (34472120), iocapables 13, client (111e4eb70)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e62d30) proc (112edbcb0), iocapables 14.
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112edbcb0), pid (41812046), iocapables 14, client (111e62d30)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 15, client (111e56010)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 16.
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 16, client (111e5fd90)
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: Aborting client (112c71a90) proc (112ef8f10), iocapables 17.
2012-06-04 23:27:31.274: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112ef8f10), pid (40370574), iocapables 17, client (112c71a90)

2012-06-04 23:27:31.274: [    CSSD][1029]clssgmClientShutdown: total iocapables 17

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

2012-06-04 23:27:31.508: [    CSSD][3091]clssgmAllocateRPCIndex: allocated rpc 772 (111259170)
2012-06-04 23:27:31.508: [    CSSD][3091]clssgmRPC: RPC(#2) to node(772) not sent due to impending local GM shutdown
2012-06-04 23:27:31.508: [    CSSD][3091]clssgmRPC: failed to send RPC#772 to node(2), rpcret(10), master(2), DBInfo(1), masterRPC(1),unsentRPC(1), queuing RPC to unsent queue
2012-06-04 23:27:31.508: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [    CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.694: [ default][4890]kgzf_recv_main4: got response for request 111efc710 (handle 2)


2012-06-04 23:27:31.694: [ default][4890]kgzf_decode_msg2: ENTITY_FENCED for req handle 2, error = 0


2012-06-04 23:27:35.036: [    CSSD][1029]clssgmClientShutdown: total iocapables 4
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 1.
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 1, client (112f97d30)
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 2, client (1118a1c90)
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 3, client (111e56010)
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 4.
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 4, client (111e5fd90)
2012-06-04 23:27:35.037: [    CSSD][1029]clssgmClientShutdown: total iocapables 4
2012-06-04 23:27:35.038: [    CSSD][1029]clssnmCompleteGMReq: Completed request type 18 with status 1
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmDoneQEle: re-queueing req 1115ded30 status 1
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmCheckReqNMCompletion: Completing request type 18 for proc (1132b2650), operation status 1, client status 0
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 1.
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 1, client (112f97d30)
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 2, client (1118a1c90)
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 3, client (111e56010)
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 4.
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 4, client (111e5fd90)
2012-06-04 23:27:35.038: [    CSSD][1029]clssgmClientShutdown: total iocapables 4
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmFenceClient: fencing client (113057150), member 0 in group DG_LOCAL_DATA, same group share, death fence 1, SAGE fence 0
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmFenceProcessDeath: client (113057150) pid 62390694 undead
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmQueueFenceForCheck: (1115ded30) Death check for object type 2, pid 62390694
2012-06-04 23:27:35.039: [    CSSD][3091]clssgmFenceCompletion: (1115ded30) process death fence completed for process 62390694, object type 2
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2b3c7e
2012-06-04 23:27:35.039: [    CSSD][3091]clssgmTermShare: (113247350) local grock DG_LOCAL_DATA member 0 type 1
2012-06-04 23:27:35.039: [    CSSD][3091]clssgmUnreferenceMember: local grock DG_LOCAL_DATA member 0 refcount is 8
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 1.
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 1, client (112f97d30)
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 2, client (1118a1c90)
2012-06-04 23:27:35.039: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 3, client (111e56010)
2012-06-04 23:27:35.040: [    CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 4.
2012-06-04 23:27:35.040: [    CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 4, client (111e5fd90)

。。。。。。。。。。。。。。。。。。。。。。。

2012-06-04 23:28:01.619: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1118a2610) client(0)
2012-06-04 23:28:01.622: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1115db470) client(0)
2012-06-04 23:28:01.634: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1115db7b0) client(0)
2012-06-04 23:28:01.634: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1115dbaf0) client(0)
2012-06-04 23:28:01.651: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1118a2610) client(0)
2012-06-04 23:28:01.669: [    CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.669: [    CSSD][1029]clssgmAllocProc: (1118a6cd0) allocated
2012-06-04 23:28:01.669: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1118a6cd0 - 0,1,2,3,4
2012-06-04 23:28:01.669: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(5ee) proc(1118a6cd0) pid(56951112/56951112) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:28:01.669: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.670: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1118a6cd0) client(0)
2012-06-04 23:28:01.671: [    CSSD][1029]clssgmDeadProc: proc 1118a6cd0
2012-06-04 23:28:01.671: [    CSSD][1029]clssgmDestroyProc: cleaning up proc(1118a6cd0) con(5ee) skgpid 56951112 ospid 56951112 with 0 clients, refcount 0
2012-06-04 23:28:01.671: [    CSSD][1029]clssgmDiscEndpcl: gipcDestroy 5ee
2012-06-04 23:28:01.709: [    CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.709: [    CSSD][1029]clssgmAllocProc: (111e1a130) allocated
2012-06-04 23:28:01.709: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 111e1a130 - 0,1,2,3,4
2012-06-04 23:28:01.709: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(64f) proc(111e1a130) pid(50724892) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:28:01.709: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.712: [    CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.713: [    CSSD][1029]clssgmAllocProc: (111e51c90) allocated
2012-06-04 23:28:01.713: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 111e51c90 - 1,2,3,4,5
2012-06-04 23:28:01.713: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(6a1) proc(111e51c90) pid(262412) version 11:2:1:4, properties: 1,2,3,4,5
2012-06-04 23:28:01.713: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.717: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e1a130) client(0)
2012-06-04 23:28:01.726: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e51c90) client(0)
2012-06-04 23:28:01.742: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e1a130) client(0)
2012-06-04 23:28:01.746: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e51c90) client(0)
2012-06-04 23:28:01.753: [    CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.753: [    CSSD][1029]clssgmAllocProc: (112044a30) allocated
2012-06-04 23:28:01.754: [    CSSD][1029]clssgmClientConnectMsg: properties of cmProc 112044a30 - 1,2,3,4,5
2012-06-04 23:28:01.754: [    CSSD][1029]clssgmClientConnectMsg: Connect from con(718) proc(112044a30) pid(196930) version 11:2:1:4, properties: 1,2,3,4,5
2012-06-04 23:28:01.754: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.762: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(112044a30) client(0)
2012-06-04 23:28:01.772: [    CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(112044a30) client(0)
2012-06-04 23:28:02.150: [   SKGFD][2061]NOTE: No asm libraries found in the system


2012-06-04 23:28:02.150: [    CLSF][2061]Allocated CLSF context
2012-06-04 23:28:02.150: [   SKGFD][2061]Handle 112062170 from lib :UFS:: for disk :/dev/rhdisk1:


2012-06-04 23:28:02.151: [   SKGFD][2061]Lib :UFS:: closing handle 112062170 for disk :/dev/rhdisk1:


2012-06-04 23:28:03.151: [    CSSD][1]clssnmlfmtlease: uniqueness 1338823677, gipc addr gipcha://pt1rac2:nm2_pt1rac-cluster
2012-06-04 23:28:03.177: [    CSSD][1]clssnmvStatusBlkInit: myinfo nodename pt1rac2, uniqueness 1338823677
2012-06-04 23:28:03.206: [    CSSD][1]clssnmlgetslot:lease acquisition for node pt1rac2/slot 2 completed in 5041 msecs
2012-06-04 23:28:03.339: [    CSSD][1]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 232297358, wrtcnt, 2006767, LATS 2358366040, lastSeqNo 0, uniqueness 1338794747, timestamp 1338822439/2180206706
2012-06-04 23:28:03.339: [    CSSD][1]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 232297359, wrtcnt, 1995087, LATS 2358366040, lastSeqNo 0, uniqueness 1338823677, timestamp 1338823448/2358131374
2012-06-04 23:28:03.354: [   SKGFD][1]Lib :UFS:: closing handle 111e2d670 for disk :/dev/rhdisk1:


2012-06-04 23:28:03.354: [    CSSD][1]clssnmInitNodeDB: Initializing with OCR id 0
2012-06-04 23:28:04.356: [    CSSD][2062]clssnm_skgxnmon: Compatible vendor clusterware not in use
2012-06-04 23:28:04.357: [    CSSD][1]clssnmNotifyReq: type (20)
2012-06-04 23:28:04.357: [    CSSD][1]clssnmReadNodeInfo: dynamic endp HA gipcha://pt1rac1:nm2_pt1rac-cluster len 35 ver 11.2.0.2.0
2012-06-04 23:28:04.357: [    CSSD][1]clssnmReadNodeInfo: NM endp for node 1 (pt1rac1) - gipcha://pt1rac1:nm2_pt1rac-cluster
2012-06-04 23:28:04.357: [    CSSD][1]clssnmReadNodeInfo: added node 1 (pt1rac1) to cluster, lease uniqueness 1338794747
2012-06-04 23:28:04.357: [    CSSD][1]clssnmReadNodeInfo: dynamic endp HA gipcha://pt1rac2:nm2_pt1rac-cluster len 35 ver 11.2.0.2.0
2012-06-04 23:28:04.360: [    GPNP][1] clsgpnp_Init: [at clsgpnp0.c:583 clsgpnp_Init] '/soft/grid/product/11.2.0.2/gih' in effect as GPnP home base.
2012-06-04 23:28:04.360: [    GPNP][1] clsgpnp_Init: [at clsgpnp0.c:629 clsgpnp_Init] GPnP pid=10879068, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
2012-06-04 23:28:04.389: [    GPNP][1] clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399 clsgpnpkwf_initwfloc] Using FS Wallet Location : /soft/grid/product/11.2.0.2/gih/gpnp/pt1rac2/wallets/peer/


[   CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2012-06-04 23:28:04.473: [    GPNP][1] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2108 clsgpnp_profileCallUrlInt] get-profile call to url "ipc://GPNPD_pt1rac2" disco "" [f=3 claimed- host: cname: seq: auth:] 
2012-06-04 23:28:04.490: [    GPNP][1] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2236 clsgpnp_profileCallUrlInt] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_pt1rac2" disco ""
2012-06-04 23:28:04.694: [    CSSD][1]clssnmReadNodeInfo: NM endp for node 2 (pt1rac2) - gipcha://pt1rac2:nm2_pt1rac-cluster
2012-06-04 23:28:04.694: [    CSSD][1]clssnmReadNodeInfo: added node 2 (pt1rac2) to cluster, lease uniqueness 1338823677
2012-06-04 23:28:04.694: [    CSSD][1][INFO]clssnmInitNodeDB: no vendor clusterware, IPMI allowed
[   CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2012-06-04 23:28:04.755: [    CSSD][1]clssnmReadWallet: OpenWallet: CLSW-9: The cluster wallet to be operated on does not exist., returned 9
2012-06-04 23:28:04.756: [    CSSD][1]clssnmSetupCookie: clssnmReadWallet fails - Node not configured for NodeKill
2012-06-04 23:28:04.756: [    CSSD][1]clssnmInitNodeDB: failure 1 reading node kill 
2012-06-04 23:28:04.756: [   SKGFD][1]NOTE: No asm libraries found in the system

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:29:59.973: [    CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:29:59.985: [    CSSD][1029]clssscSelect: cookie accept request 113034870
2012-06-04 23:29:59.985: [    CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113034870
2012-06-04 23:29:59.986: [    CSSD][1029]clssgmRegisterClient: proc(23/113034870), client(1/1117fd310)
2012-06-04 23:29:59.986: [    CSSD][1029]clssgmJoinGrock: global grock DGPT1DB0 new client 1117fd310 with con 2e67, requested num 1, flags 0x0
2012-06-04 23:29:59.986: [    CSSD][1029]clssgmAddGrockMember: adding member to grock DGPT1DB0
2012-06-04 23:29:59.986: [    CSSD][1029]clssgmAllocateRPCIndex: allocated rpc 160 (11121cc30)
2012-06-04 23:29:59.986: [    CSSD][1029]clssgmRPC: rpc 11121cc30 (RPC#160) tag(a0002a) sent to node 2
2012-06-04 23:29:59.986: [    CSSD][3348]clssgmHandleMasterMemberAdd: [s(2) d(2)]
2012-06-04 23:29:59.986: [    CSSD][3348]clssgmAddMember: new master 1 for group(DGPT1DB0)
2012-06-04 23:29:59.986: [    CSSD][3348]clssgmAddMember: Adding fencing for member 1, group DGPT1DB0, death 1, SAGE 0
2012-06-04 23:29:59.986: [    CSSD][3348]clssgmAddMember: member (1/113044f50) added. pbsz(11) prsz(124) flags 0x0 to grock (113044b30/DGPT1DB0)
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmQueueGrockEvent: groupName(DGPT1DB0) count(1) master(1) event(1), incarn 1, mbrc 1, to member 1, events 0x0, state 0x0
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmCommonAddMember: global group grock DGPT1DB0 member(1/Local) node(2) flags 0x0 0x13044b30 
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmRPCDone: rpc 11121cc30 (RPC#160) state 6, flags 0x100
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmAddGrockMemCmpl: rpc 11121cc30, ret 0, client 1117fd310 member 113044f50
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmAddGrockMemCmpl: sending type 6, size 540 to 1117fd310
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmFreeRPCIndex: freeing rpc 160
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmAllocateRPCIndex: allocated rpc 161 (11121cdc0)
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmRPCBroadcast: rpc(0xa1002a), status(0), sendcount(0), filtered by specific properties: 
2012-06-04 23:29:59.987: [    CSSD][3348]clssgmFreeRPCIndex: freeing rpc 161

有以上日志可知，最开始出现了ASM的DiskPing检测超时的情况，然后crsd服务进程出现coredump，最终导致实例Shutdown。

继续检查crsd.log日志

2012-06-04 23:26:13.427: [UiServer][11823] CS(117435bd0)set Properties ( grid,117435e90)
2012-06-04 23:26:13.451: [UiServer][11566] {2:53227:2141} Sending message to PE. ctx= 116bce710
2012-06-04 23:26:13.451: [   CRSPE][11309] {2:53227:2141} Processing PE command id=2140. Description: [Stat Resource : 1174374b0]
2012-06-04 23:26:13.456: [UiServer][11566] {2:53227:2141} Done for ctx=116bce710
2012-06-04 23:27:13.615: [UiServer][11823] CS(116c26750)set Properties ( grid,116c230b0)
2012-06-04 23:27:13.640: [UiServer][11566] {2:53227:2142} Sending message to PE. ctx= 116be97d0
2012-06-04 23:27:13.642: [   CRSPE][11309] {2:53227:2142} Processing PE command id=2141. Description: [Stat Resource : 116c7ca10]
2012-06-04 23:27:13.648: [UiServer][11566] {2:53227:2142} Done for ctx=116be97d0
2012-06-04 23:27:18.354: [UiServer][11823] CS(1160f2910)set Properties ( root,1160f2150)
2012-06-04 23:27:18.372: [UiServer][11566] {2:53227:2143} Sending message to PE. ctx= 116bce330
2012-06-04 23:27:18.373: [   CRSPE][11309] {2:53227:2143} Processing PE command id=2142. Description: [Stat Resource : 116c23170]
2012-06-04 23:27:18.407: [UiServer][11566] {2:53227:2143} Done for ctx=116bce330
2012-06-04 23:27:27.841: [UiServer][11823] CS(116c28710)set Properties ( grid,116c23490)
2012-06-04 23:27:27.862: [UiServer][11566] {2:53227:2144} Container [ Name: FENCESERVER
CLIENT: 
TextMessage[]
CLIENT_PRIMARY_GROUP: 
TextMessage[oinstall]
LOCALE: 
TextMessage[AMERICAN_AMERICA.WE8ISO8859P1]
]
2012-06-04 23:27:27.863: [UiServer][11566] {2:53227:2144} Sending message to AGFW. ctx= 116b07a30
2012-06-04 23:27:27.863: [UiServer][11566] {2:53227:2144} Force-disconnecting [0]  existing PE clients...
2012-06-04 23:27:27.863: [UiServer][11566] {2:53227:2144} Sending message: 9978 to AGFW proxy server.
2012-06-04 23:27:27.910: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server received the message: UnKnown[Proxy] ID 20489:9978
2012-06-04 23:27:27.978: [   CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.DATA.dg
2012-06-04 23:27:28.035: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER.lsnr pt1rac2 1] ID 4100:9979 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.043: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:9980 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.044: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER_SCAN2.lsnr 1 1] ID 4100:9981 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.045: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER_SCAN3.lsnr 1 1] ID 4100:9982 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.045: [   CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.asm
2012-06-04 23:27:28.048: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.cvu 1 1] ID 4100:9983 to the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.055: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.net1.network pt1rac2 1] ID 4100:9984 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.055: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:9985 to the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.057: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.ons pt1rac2 1] ID 4100:9986 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.076: [   CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.pt1db.db
2012-06-04 23:27:28.077: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.pt1rac1.vip 1 1] ID 4100:9987 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.077: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.pt1rac2.vip 1 1] ID 4100:9988 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.086: [   CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.registry.acfs
2012-06-04 23:27:28.094: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.scan1.vip 1 1] ID 4100:9989 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.105: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.scan2.vip 1 1] ID 4100:9990 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.106: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.scan3.vip 1 1] ID 4100:9991 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.319: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:9985 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.320: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:9985 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.343: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.oc4j]
2012-06-04 23:27:28.344: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.cvu 1 1] ID 4100:9983 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.344: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.cvu 1 1] ID 4100:9983 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.360: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.cvu]
2012-06-04 23:27:28.369: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server received the message: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:142
2012-06-04 23:27:28.369: [    AGFW][10024] {2:53227:2144} Shutdown request received from /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.369: [    AGFW][10024] {2:53227:2144} Agfw Proxy Server replying to the message: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:142
2012-06-04 23:27:28.479: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.net1.network pt1rac2 1] ID 4100:9984 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.645: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER.lsnr pt1rac2 1] ID 4100:9979 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.730: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN2.lsnr 1 1] ID 4100:9981 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.871: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.net1.network pt1rac2 1] ID 4100:9984 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.871: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.net1.network]
2012-06-04 23:27:28.879: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan2.vip 1 1] ID 4100:9990 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.880: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan2.vip 1 1] ID 4100:9990 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.890: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.scan2.vip]
2012-06-04 23:27:28.891: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac2.vip 1 1] ID 4100:9988 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.894: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac2.vip 1 1] ID 4100:9988 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.895: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.pt1rac2.vip]
2012-06-04 23:27:28.899: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac1.vip 1 1] ID 4100:9987 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.907: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac1.vip 1 1] ID 4100:9987 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.925: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.pt1rac1.vip]
2012-06-04 23:27:28.925: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan3.vip 1 1] ID 4100:9991 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.926: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan3.vip 1 1] ID 4100:9991 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.938: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.scan3.vip]
2012-06-04 23:27:28.944: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:9980 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.946: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN3.lsnr 1 1] ID 4100:9982 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.973: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan1.vip 1 1] ID 4100:9989 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.975: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan1.vip 1 1] ID 4100:9989 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.976: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.scan1.vip]
2012-06-04 23:27:29.734: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.ons pt1rac2 1] ID 4100:9986 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:29.933: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.ons pt1rac2 1] ID 4100:9986 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:29.933: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.ons]
2012-06-04 23:27:30.618: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER.lsnr pt1rac2 1] ID 4100:9979 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:30.633: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER.lsnr]
2012-06-04 23:27:30.637: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN2.lsnr 1 1] ID 4100:9981 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:30.654: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER_SCAN2.lsnr]
2012-06-04 23:27:30.718: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN3.lsnr 1 1] ID 4100:9982 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:30.719: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER_SCAN3.lsnr]
2012-06-04 23:27:31.191: [    AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:9980 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:31.191: [    AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER_SCAN1.lsnr]
2012-06-04 23:27:31.192: [   CRSPE][10024] {2:53227:2144} Fence command status: Container [ Name: UI_DATA
RESULT: 
TextMessage[0]
]
2012-06-04 23:27:31.192: [    AGFW][10024] {2:53227:2144} Fence command completed!
2012-06-04 23:27:31.192: [UiServer][11566] {2:53227:2144} UI server recvd reply from Agfw Proxy Server: 2042
2012-06-04 23:27:31.192: [UiServer][11566] {2:53227:2144} Container [ Name: UI_DATA
RESULT: 
TextMessage[0]
]
2012-06-04 23:27:42.055: [ default][1] CSS is not ready. Received status 3
2012-06-04 23:27:42.055: [ default][1] First attempt: init CSS context failed. Error = 3
[  clsdmt][515]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=pt1rac2DBG_CRSD))
2012-06-04 23:27:42.067: [  clsdmt][515]PID for the Process [56951100], connkey 1 
2012-06-04 23:27:42.068: [  clsdmt][515]Creating PID [56951100] file for home /soft/grid/product/11.2.0.2/gih host pt1rac2 bin crs to /soft/grid/product/11.2.0.2/gih/crs/init/
2012-06-04 23:27:42.068: [  clsdmt][515]Writing PID [56951100] to the file [/soft/grid/product/11.2.0.2/gih/crs/init/pt1rac2.pid] 
2012-06-04 23:27:44.350: [ default][1] CRS Daemon Starting
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: AGENT  1
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: AGFW  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CLSFRAME  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CLSVER  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CLUCLS  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSAPP  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSCCL  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSCEVT  0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSCOMM  0
2012-06-04 23:27:44.361: [    CRSD][1] ENV Logging level for Module: CRSD  0
2012-06-04 23:27:44.361: [    CRSD][1] ENV Logging level for Module: CRSEVT  0
2012-06-04 23:27:44.361: [    CRSD][1] ENV Logging level for Module: CRSMAIN  0
2012-06-04 23:27:44.361: [    CRSD][1] ENV Logging level for Module: CRSOCR  0
2012-06-04 23:27:44.361: [    CRSD][1] ENV Logging level for Module: CRSPE  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSPLACE  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSRES  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSRPT  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSRTI  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSSE  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSSEC  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSTIMER  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: CRSUI  0
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCRAPI  1
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCRASM  1
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCRCAC  1
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCRCLI  1
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCRMAS  1
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCRMSG  1
2012-06-04 23:27:44.362: [    CRSD][1] ENV Logging level for Module: OCROSD  1
2012-06-04 23:27:44.363: [    CRSD][1] ENV Logging level for Module: OCRRAW  1
2012-06-04 23:27:44.363: [    CRSD][1] ENV Logging level for Module: OCRSRV  1
2012-06-04 23:27:44.363: [    CRSD][1] ENV Logging level for Module: OCRUTL  1
2012-06-04 23:27:44.363: [    CRSD][1] ENV Logging level for Module: SuiteTes  1
2012-06-04 23:27:44.363: [    CRSD][1] ENV Logging level for Module: UiServer  0
2012-06-04 23:27:44.363: [ CRSMAIN][1] Checking the OCR device
2012-06-04 23:27:44.363: [ CRSMAIN][1] Sync-up with OCR
2012-06-04 23:27:44.363: [ CRSMAIN][1] Connecting to the CSS Daemon
2012-06-04 23:27:44.376: [  CRSRTI][1] CSS is not ready. Received status 3
2012-06-04 23:27:44.400: [ CRSMAIN][1] Created alert : (:CRSD00109:) :  Could not init the CSS context, error: 3
2012-06-04 23:27:44.400: [    CRSD][1][PANIC] CRSD exiting: Could not init the CSS context, error: 3
2012-06-04 23:27:44.400: [    CRSD][1] Done.


相同的时间点crsd也出现了错误

下面是diskmon.log的日志

2012-06-04 23:27:00.240: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:03.235: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:06.236: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:09.268: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:12.239: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:15.268: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:18.272: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:21.267: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:24.292: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:27.298: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:30.307: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:31.544: [ DISKMON][8781838:1286] dskm_process_msg5: received msg type KGZM_FENCE_ENTITY (0x0007)
2012-06-04 23:27:31.662: [ DISKMON][8781838:1286] dskm_proc_fence_entity13: fenced reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100 (size 93, msgno 2, event 2)
2012-06-04 23:27:31.694: [ DISKMON][8781838:1286] dskm_send_entity_fenced1:
2012-06-04 23:27:31.694: [ DISKMON][8781838:1286] dskm_send_entity_fenced4: done
2012-06-04 23:27:31.731: [ DISKMON][8781838:1286] dskm_process_msg7: processed msg 2 type KGZM_FENCE_ENTITY (0x0007), retcode 0
2012-06-04 23:27:34.483: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:35.180: [ DISKMON][8781838:1286] dskm_slave_thrd_main3: peer disconnected
2012-06-04 23:27:35.180: [ DISKMON][8781838:1286] dskm_slave_thrd_main5: client kgzf/8126478 disconnected, reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2012-06-04 23:27:35.531: [ DISKMON][8781838:1] dskm_cleanup_thrds: cleaning up slave 1 thread tid 1286
2012-06-04 23:27:36.945: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:39.681: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:40.442: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1
2012-06-04 23:27:40.464: [ CSSCLNT]clssgsGroupGetStatus:  communications failed (0/3/-1)


2012-06-04 23:27:40.464: [ CSSCLNT]clssgsGroupGetStatus: returning 8


2012-06-04 23:27:40.464: [ DISKMON][8781838:515] dskm_rac_thrd_main6: cssd ABORT event
2012-06-04 23:27:40.465: [ DISKMON][8781838:515] SHUTDOWN FORCE due to CSSD ABORT
2012-06-04 23:27:40.466: [ DISKMON][8781838:515] dskm_rac_thrd_main: exiting
2012-06-04 23:27:40.681: [ DISKMON][8781838:1029] dskm_slave_thrd_main5: client orarootagent/7798854 disconnected, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
2012-06-04 23:27:41.955: [ DISKMON][8781838:1] dskm_cleanup_thrds: cleaning up the rac event handling thread tid 515
2012-06-04 23:27:41.962: [ DISKMON][8781838:1] dskm_cleanup_thrds: cleaning up slave 0 thread tid 1029
2012-06-04 23:27:41.978: [ DISKMON][8781838:772] dskm_oss_thrd_main2: posted
2012-06-04 23:27:42.000: [ DISKMON][8781838:772] dskm_oss_thrd_main: exiting
2012-06-04 23:27:42.016: [ DISKMON][8781838:1] Diskmon Aborting
[ DISKMON][8781838] 
Process 8781838 exiting on 2012-06-04 at 23:27:42.035


[ DISKMON][8781842] 


I/O Fencing and SKGXP HA monitoring daemon -- Version 1.1.0.0
Process 8781842 started on 2012-06-04 at 23:27:44.409


2012-06-04 23:27:44.425: [ DISKMON][8781842] dskm main: starting up
2012-06-04 23:27:44.425: [ DISKMON][8781842:515] dskm_rac_thrd_main: running
2012-06-04 23:27:44.426: [ DISKMON][8781842:515] dskm_clss_ini1: calling clssscbinit
2012-06-04 23:27:44.426: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:44.426: [ DISKMON][8781842:1] dskm_rac_thrd_creat2: got the post from the css event handling thread
2012-06-04 23:27:44.427: [ DISKMON][8781842:772] dskm_oss_thrd_main: running
2012-06-04 23:27:44.427: [ DISKMON][8781842:1] dskm_oss_thrd_creat2: got the post from the oss check status thread
2012-06-04 23:27:44.437: [ DISKMON][8781842:1] dskm main: startup complete
2012-06-04 23:27:44.437: [ DISKMON][8781842:1]            listening on -> /tmp/.oracle/master_diskmon
2012-06-04 23:27:44.934: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:45.314: [ DISKMON][8781842:1] dskm_slave_thrd_creat: thread created
2012-06-04 23:27:45.314: [ DISKMON][8781842:1029] dskm_slave_thrd_main1: slave 0 running
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_proc_identify8: client orarootagent/7798854, diskmon kgzm 2.1, slave 0, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_send_version1:
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_send_version4: done
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2012-06-04 23:27:45.325: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:45.437: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:45.940: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:46.443: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:46.946: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:47.449: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:47.952: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:48.375: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:48.457: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:48.959: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:49.462: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:49.965: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:50.469: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:51.113: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:51.371: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:51.629: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:52.152: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:52.661: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:53.589: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:54.096: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:54.392: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:54.609: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:55.115: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:55.618: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:56.125: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:56.633: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:57.136: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:57.381: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:57.640: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:57.648: [ DISKMON][8781842:515] dskm_clss_ini5: successful clsssinit(), clssvers 2.1
2012-06-04 23:27:57.648: [ DISKMON][8781842:515] dskm_clss_ini6: calling clssnsqlnum
2012-06-04 23:28:00.379: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:03.386: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:04.848: [ DISKMON][8781842:515] dskm_clss_ini8: calling clsssattrib
2012-06-04 23:28:04.848: [ DISKMON][8781842:515] dskm_clss_ini11: calling clssnsqname
2012-06-04 23:28:04.850: [ DISKMON][8781842:515] dskm_clss_ini13: calling clsssattrib
2012-06-04 23:28:04.850: [ DISKMON][8781842:515] dskm_clss_ini15: calling clssgsregnodegrp
2012-06-04 23:28:04.854: [ DISKMON][8781842:515] dskm_clss_ini18: calling clssgsgrpstat
2012-06-04 23:28:04.854: [ DISKMON][8781842:515] dskm_clss_ini21: calling clssgsgrpstat
2012-06-04 23:28:06.414: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:09.390: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_slave_thrd_main1: slave 1 running
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_proc_identify8: client kgzf/10879068, diskmon kgzm 2.1, slave 1, reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297360,nmn=2,lnid=232297360,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_send_version1:
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_send_version4: done
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2012-06-04 23:28:12.362: [ DISKMON][8781842:1] dskm_slave_thrd_creat: thread created
2012-06-04 23:28:12.363: [ DISKMON][8781842:1286] dskm_process_msg5: received msg type KGZM_KGZF_HANDSHAKE (0x0010)
2012-06-04 23:28:12.363: [ DISKMON][8781842:1286] dskm_proc_kgzf_handshake3: client kgzf/10879068, kgzf version 2.0, slave 1
2012-06-04 23:28:12.365: [ DISKMON][8781842:515] dskm_clss_ini20: node pt1rac2 (2) registered in cluster
2012-06-04 23:28:12.365: [ DISKMON][8781842:515] dskm_clss_ini23: successful dskm_clss_ini
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_check_cell_config: CELL storage configuration file /etc/oracle/cell/network-config/cellinit.ora not found
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini3: calling clsssattrib
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini7: calling clsssattrib
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini9: calling clsssattrib
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini13: calling clssgsqgrp
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_rac_ini80: called clssgsqgrp:
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_dump_group_priv: vers: 0 flags: 0x0 confInc: 0 My confInc: 0
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_dump_group_priv: CSS Msg Hdr: vers: 0 type: UNKNOWN (0) chunks: NO MORE CHUNKS (0) transport: UNKNOWN (0) mSize: 0
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_dump_group_priv: Group Private Data is not of type DSKM_MSG_SS_REQ. Not proceeding with msg dump
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_nfy_kgzf1: notified thread kgzf enabled
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_rac_ini30: calling clsslsmutexlock(MASTER#DISKMON#GROUP#MX)
2012-06-04 23:28:12.369: [ DISKMON][8781842:1286] dskm_proc_kgzf_handshake5: got the post from the hb thread
2012-06-04 23:28:12.369: [ CSSCLNT]clsslsNativeLock: CSS daemon has not entered ACTIVE state.Retrying.


2012-06-04 23:28:12.370: [ DISKMON][8781842:1286] dskm_proc_kgzf_handshake9: done, kgzf enabled
2012-06-04 23:28:12.370: [ DISKMON][8781842:1286] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
2012-06-04 23:28:12.429: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:13.373: [ DISKMON][8781842:515] dskm_rac_ini32: calling clssgsreg
2012-06-04 23:28:13.375: [ DISKMON][8781842:515] dskm_rac_ini34: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 2 (GSDGRPSZ 512)
2012-06-04 23:28:13.375: [ DISKMON][8781842:515] dskm_rac_ini63: calling clsslsunlock (proc retcode 0)
2012-06-04 23:28:15.443: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:18.440: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:21.431: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:24.435: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)


由于cssd服务出现了coredump，所以diskmon服务会把asm实例强制shutdown。

综合以上分析，可知故障是由于开发人员做压力测试导致磁盘IO过载，因此Grid在做VOTE磁盘检测时出现了超时，导致cssd服务coredump退出，进而引起diskmon服务关闭asm实例，最终数据库实例也被关闭。