Fri Dec 11 20:21:59 2015
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 1.
WARNING: Waited 16 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 16 secs for write IO to PST disk 1 in group 1.
WARNING: Waited 16 secs for write IO to PST disk 2 in group 1.
WARNING: Waited 16 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 16 secs for write IO to PST disk 0 in group 2.
Fri Dec 11 20:21:59 2015
NOTE: process _b000_+asm1 (13259) initiating offline of disk 0.3940456251 (CRSDG_0000) with mask 0x7e in group 1
NOTE: process _b000_+asm1 (13259) initiating offline of disk 1.3940456250 (CRSDG_0001) with mask 0x7e in group 1
NOTE: process _b000_+asm1 (13259) initiating offline of disk 2.3940456249 (CRSDG_0002) with mask 0x7e in group 1
NOTE: checking PST: grp = 1
GMON checking disk modes for group 1 at 7 for pid 32, osid 13259
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 1 done.
NOTE: initiating PST update: grp = 1, dsk = 0/0xeade973b, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 1, dsk = 1/0xeade973a, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 1, dsk = 2/0xeade9739, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 8 for pid 32, osid 13259
ERROR: no read quorum in group: required 2, found 0 disks
Fri Dec 11 20:21:59 2015
NOTE: cache dismounting (not clean) group 1/0x389E67C4 (CRSDG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 13261, image: oracle@rac01 (B001)
Fri Dec 11 20:21:59 2015
NOTE: halting all I/Os to diskgroup 1 (CRSDG)
Fri Dec 11 20:21:59 2015
NOTE: LGWR doing non-clean dismount of group 1 (CRSDG)
NOTE: LGWR sync ABA=12.66 last written ABA 12.66
Fri Dec 11 20:21:59 2015
kjbdomdet send to inst 2
detach from dom 1, sending detach message to inst 2
WARNING: Offline for disk CRSDG_0000 in mode 0x7f failed.
WARNING: Offline for disk CRSDG_0001 in mode 0x7f failed.
WARNING: Offline for disk CRSDG_0002 in mode 0x7f failed.
Fri Dec 11 20:21:59 2015
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 56)
Fri Dec 11 20:21:59 2015
NOTE: No asm libraries found in the system
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 1 invalid = TRUE
114 GCS resources traversed, 0 cancelled
ASM Health Checker found 1 new failures
Dirty Detach Reconfiguration complete
Fri Dec 11 20:21:59 2015
WARNING: dirty detached from domain 1
NOTE: cache dismounted group 1/0x389E67C4 (CRSDG)
SQL> alter diskgroup CRSDG dismount force /* ASM SERVER:949905348 */
Fri Dec 11 20:22:00 2015
NOTE: cache deleting context for group CRSDG 1/0x389e67c4
GMON dismounting group 1 at 9 for pid 36, osid 13261
NOTE: Disk CRSDG_0000 in mode 0x7f marked for de-assignment
NOTE: Disk CRSDG_0001 in mode 0x7f marked for de-assignment
NOTE: Disk CRSDG_0002 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 1
WARNING: Waited 19 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 19 secs for write IO to PST disk 0 in group 2.
Fri Dec 11 20:22:03 2015
SQL> ALTER DISKGROUP CRSDG MOUNT /* asm agent *//* {0:1:6} */
WARNING: Disk Group CRSDG containing spfile for this instance is not mounted
WARNING: Disk Group CRSDG containing configured OCR is not mounted
WARNING: Disk Group CRSDG containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "CRSDG" cannot be mounted
ORA-15013: diskgroup "CRSDG" is already mounted
ERROR: ALTER DISKGROUP CRSDG MOUNT /* asm agent *//* {0:1:6} */
Fri Dec 11 20:22:30 2015
SUCCESS: diskgroup CRSDG was dismounted
SUCCESS: alter diskgroup CRSDG dismount force /* ASM SERVER:949905348 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group CRSDG
Fri Dec 11 20:22:30 2015
NOTE: diskgroup resource ora.CRSDG.dg is offline
Fri Dec 11 20:44:49 2015
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
Fri Dec 11 21:15:58 2015
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_6434.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
WARNING: requested mirror side 1 of virtual extent 6 logical extent 0 offset 720896 is not allocated; I/O request failed
WARNING: requested mirror side 2 of virtual extent 6 logical extent 1 offset 720896 is not allocated; I/O request failed
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_6434.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Fri Dec 11 21:15:58 2015
SQL> alter diskgroup CRSDG check /* proxy */
ORA-15032: not all alterations performed
ORA-15001: diskgroup "CRSDG" does not exist or is not mounted
ERROR: alter diskgroup CRSDG check /* proxy */
NOTE: client exited [6419]
Fri Dec 11 21:16:08 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16827] opening OCR file
Fri Dec 11 21:16:10 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16840] opening OCR file
Fri Dec 11 21:16:12 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16853] opening OCR file
Fri Dec 11 21:16:14 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16867] opening OCR file
Fri Dec 11 21:16:16 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16880] opening OCR file
Fri Dec 11 21:16:18 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16893] opening OCR file
Fri Dec 11 21:16:20 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16906] opening OCR file
Fri Dec 11 21:16:22 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16919] opening OCR file
Fri Dec 11 21:16:24 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16932] opening OCR file
Fri Dec 11 21:16:26 2015
NOTE: [crsd.bin@rac01 (TNS V1-V3) 16947] opening OCR file