查看当前orc的备份情况
[04:46:45 root(grid)@rac1 ~]# ocrconfig -showbackup
rac1 2014/10/27 04:30:23 /tmp/backup00.ocr
rac1 2014/10/27 04:30:23 /tmp/day.ocr
rac1 2014/10/27 04:30:23 /tmp/week.ocr
rac1 2014/10/27 04:12:17 /tmp/backup_20141027_041217.ocr
查看当前crs的状况
[04:39:17 root(grid)@rac1 ~]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type OFFLINE OFFLINE
ora....N1.lsnr ora....er.type OFFLINE OFFLINE
ora.OCRVOTE.dg ora....up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type OFFLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.prod.db ora....se.type OFFLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application OFFLINE OFFLINE
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type OFFLINE OFFLINE
两个节点手动停止crs服务
[04:47:02 root(grid)@rac1 ~]# crsctl stop crs -f
[00:15:28 root(grid)@rac2 ~]# crsctl stop crs -f
做破坏性试验把ocrvotedisk格掉
[04:51:11 root(grid)@rac1 ~]# dd if=/dev/zero of=/dev/sdd1 bs=1024k count=200
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0172843 s, 60.7 MB/s
[04:51:36 root(grid)@rac1 ~]# oracleasm querydisk /dev/sdd1
Device "/dev/sdd1" is not marked as an ASM disk
我们发现此时原来的仲裁盘已经不属于任何asmdisk了
这个时候我们尝试启动crs
[04:52:56 root(grid)@rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
查看grid报警日志:
[ohasd(2248)]CRS-2767:Resource state recovery not attempted for 'ora.cssdmonitor' as its target state is OFFLINE
2014-10-27 04:49:40.221:
[ohasd(2248)]CRS-2769:Unable to failover resource 'ora.cssdmonitor'.
2014-10-27 04:49:42.525:
[gpnpd(2609)]CRS-2329:GPNPD on node rac1 shutdown.
2014-10-27 04:55:05.740:
[ohasd(39995)]CRS-2112:The OLR service started on node rac1.
2014-10-27 04:55:05.771:
[ohasd(39995)]CRS-1301:Oracle High Availability Service started on node rac1.
2014-10-27 04:55:05.777:
[ohasd(39995)]CRS-8017:location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2014-10-27 04:55:10.133:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(40038)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-10-27 04:55:14.687:
[gpnpd(40133)]CRS-2328:GPNPD started on node rac1.
2014-10-27 04:55:17.612:
[cssd(40207)]CRS-1713:CSSD daemon is started in clustered mode
2014-10-27 04:55:19.094:
[ohasd(39995)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2014-10-27 04:55:19.096:
[ohasd(39995)]CRS-2769:Unable to failover resource 'ora.diskmon'.
2014-10-27 04:55:21.823:
[cssd(40207)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /u01/app/11.2.0/grid/log/rac1/cssd/ocssd.log
[05:04:32 root(grid)@rac1 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
[05:15:03 oracle(grid)@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 27 05:15:08 2014
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
05:15:08 idle>
05:15:34 idle> create diskgroup OCRVOTE external redundancy
05:16:22 2 disk 'ORCL:OCRVOTE'
05:16:22 3 ATTRIBUTE 'compatible.rdbms' = '11.2', 'compatible.asm' = '11.2';
Diskgroup created.
[05:06:57 root(grid)@rac1 ~]# ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
rac1 2014/10/27 04:30:23 /tmp/backup00.ocr
rac1 2014/10/27 04:30:23 /tmp/day.ocr
rac1 2014/10/27 04:30:23 /tmp/week.ocr
rac1 2014/10/27 04:12:17 /tmp/backup_20141027_041217.ocr
[05:18:03 root(grid)@rac1 ~]# ocrconfig -restore /tmp/backup00.ocr
[05:18:34 root(grid)@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3112
Available space (kbytes) : 259008
ID : 1190837800
Device/File Name : +OCRVOTE
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
OCR已经恢复,我们现在来回复votedisk
[05:19:11 root(grid)@rac1 ~]# crsctl query css votedisk
Located 0 voting disk(s).
可以看到,现在还没有votedisk的任何的信息
[05:20:22 root(grid)@rac1 ~]# crsctl replace votedisk +OCRVOTE
Successful addition of voting disk 4d680831fca94f19bf393f63713e05bd.
Successfully replaced voting disk group with +OCRVOTE.
CRS-4266: Voting file(s) successfully replaced
[05:22:30 root(grid)@rac1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4d680831fca94f19bf393f63713e05bd (ORCL:OCRVOTE) [OCRVOTE]
重建spfile文件
05:24:12 idle> show parameter spfile;
NAME TYPE VALUE
------------------------------------ --------------------------------- ------------------------------
spfile string
05:24:21 idle> create pfile='/tmp/init.ora' from memory;
File created.
Elapsed: 00:00:00.06
05:24:56 idle> shutdown immediate;
ASM diskgroups volume disabled
ASM diskgroups dismounted
ASM instance shutdown
05:25:20 idle> startup mount pfile='/tmp/init.ora';
ASM instance started
Total System Global Area 1135747072 bytes
Fixed Size 2260728 bytes
Variable Size 1108320520 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
05:26:32 idle> create spfile ='+OCRVOTE' from pfile='/tmp/init.ora';
File created.
Elapsed: 00:00:03.20
05:26:55 idle> startup mount force;
ASM instance started
Total System Global Area 1135747072 bytes
Fixed Size 2260728 bytes
Variable Size 1108320520 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
05:27:18 idle> show parameter spfile;
NAME TYPE VALUE
------------------------------------ --------------------------------- ------------------------------
spfile string +OCRVOTE/rac-cluster/asmparameterfile/registry.253.862032413
完成以后重启crs服务
[05:22:40 root(grid)@rac1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[05:27:58 root(grid)@rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[05:13:09 root(grid)@rac2 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[05:30:57 root(grid)@rac1 ~]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type OFFLINE OFFLINE
ora....N1.lsnr ora....er.type OFFLINE OFFLINE
ora.OCRVOTE.dg ora....up.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type OFFLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora.prod.db ora....se.type OFFLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application OFFLINE OFFLINE
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application OFFLINE OFFLINE
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type OFFLINE OFFLINE
可以看到CRS服务顺利的起来了
我们再在两个节点启动数据库,发现数据库顺利的启动了,一切ok,恢复完成!