11g r2 模拟OCR和voting disk不可用，完整恢复过程，以及一些注意事项

环境：RHEL5.8 RAC 11.2.0.3.0

1：查看ORC和voting disk信息：

In 11g Release 2 your voting disk data is automatically backed up in the OCR whenever there is a configuration change.
所以恢复时恢复备份OCR即可，这里和10g是不同的，不需要备份voting disk，备份OCR即可

2：使用cluvfy 工具检查OCR完整性
[grid@rac1 ~]$ cluvfy comp ocr -n all
Verifying OCR integrity 
Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ASM Running check passed. ASM is running on all specified nodes
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
Disk group for ocr location "+CRSDATA" available on all the nodes

NOTE: 
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.
OCR integrity check passed
Verification of OCR integrity was successful.

3：使用ocrcheck检测OCR内容的完整性
[grid@rac1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          3
Total space (kbytes)     :     262120
Used space (kbytes)      :       3016
Available space (kbytes) :     259104
ID                       : 1236405787
Device/File Name         :   +CRSDATA
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user    --如果使用root用户执行ocrcheck时，会显示Logical corruption check succeeded

4：检测voting disk的信息
[grid@rac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2b1bd0c122584f5abf72033b2b2d26bd (/dev/asm-b_crs) [CRSDATA]
 2. ONLINE   2bc03776cdd94f5cbfb9165c473fdb0e (/dev/asm-c_crs) [CRSDATA]
 3. ONLINE   3b43c39513a64f2dbf7083a9510ada89 (/dev/asm-d_crs) [CRSDATA]
Located 3 voting disk(s).

从上面看出，OCR和voting disk都位于+CRSDATA磁盘组 ，注意+CRSDATA磁盘组还有ASM的启动参数文件，ASM启动是根据磁盘头的kfdhdb.spfile指向ASM上的此磁盘的UA NUMBER从而读取spfile文件

5：手动备份一份OCR信息：
[root@rac1 grid]# ocrconfig -export /tmp/ocr_20130717.dmp
[root@rac1 grid]# ll /tmp/ocr_20130717.dmp -h
-rw------- 1 root root 102K Jul 17 14:45 /tmp/ocr_20130717.dmp

6：查看OCR自动备份信息
[grid@rac1 ~]$ ocrconfig -showbackup

rac1     2013/07/16 15:45:24     /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr
rac2     2013/07/16 08:13:38     /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup01.ocr
rac2     2013/07/16 04:14:09     /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup02.ocr
rac2     2013/07/16 00:14:38     /u01/app/11.2.0.3/grid/cdata/ad-cluster/day.ocr
rac2     2013/07/07 04:40:11     /u01/app/11.2.0.3/grid/cdata/ad-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

7：保存一份ASM参数文件，如果提前没保存，可以到$CRS_HOME/dbs/init.ora获取一份，后面此启动参数的详细内容

[grid@rac1 dbs]$ sqlplus / as sysasm
 
SQL> create pfile='/tmp/asm_pfile_130717.txt' from spfile;

File created.

8：破坏保存OCR信息的磁盘组+CRSDATA
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-b_crs bs=1024 count=1000
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-c_crs bs=1024 count=1000

9：破坏了磁盘b和c后，都检测通过，没报错，在rac1和rac2停止crs
[root@rac1 dev]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
.....................
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 dev]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
.....................
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac1 dev]# ps -ef |grep ora_
root     16189 32265  0 16:26 pts/0    00:00:00 grep ora_
[root@rac1 dev]# ps -ef |grep asm_
root     16195 32265  0 16:26 pts/0    00:00:00 grep asm_

10:再启动CRS,报错
[root@rac1 dev]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@rac1 ~]# tail -50f /u01/app/11.2.0.3/grid/log/rac1/alertrac1.log 

[cssd(16559)]C