如何恢复损坏的表决磁盘（votedisk）和OCR

今天是2014-03-28，继续写RAC的内容，今天写的是如何恢复损坏的表决磁盘（votedisk）和OCR。

 首先查看目前状态：

[root@rac-two bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3472
         Available space (kbytes) :     258648
         ID                       :  152256349
         Device/File Name         :      +GIDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac-two bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   af2fed5d6ac54f1bbf19e4982e364c57 (/dev/asm-diske) [GIDG]
 2. ONLINE   34b65f2e3db14f23bf88e689b55058b7 (/dev/asm-diskd) [GIDG]
 3. ONLINE   07bf3ce9af7d4ff6bf38d160255c671c (/dev/asm-diskc) [GIDG]
Located 3 voting disk(s).
[root@rac-two bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.GIDG.dg
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.TESTDG.dg
               OFFLINE OFFLINE      rac-one                                      
               ONLINE  OFFLINE      rac-two                                      
ora.asm
               ONLINE  ONLINE       rac-one                  Started             
               ONLINE  ONLINE       rac-two                  Started             
ora.gsd
               OFFLINE OFFLINE      rac-one                                      
               OFFLINE OFFLINE      rac-two                                      
ora.net1.network
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.ons
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.registry.acfs
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac-two                                      
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac-one                                      
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac-one                                      
ora.cvu
      1        ONLINE  ONLINE       rac-one                                      
ora.oc4j
      1        ONLINE  ONLINE       rac-one                                      
ora.rac-one.vip
      1        ONLINE  ONLINE       rac-one                                      
ora.rac-two.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.rac.db
      1        ONLINE  ONLINE       rac-two                  Open                
      2        ONLINE  ONLINE       rac-one                  Open                
ora.scan1.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.scan2.vip
      1        ONLINE  ONLINE       rac-one                                      
ora.scan3.vip
      1        ONLINE  ONLINE       rac-one               

模拟votedisk和ocr损坏。

[root@rac-two bin]# dd if=/dev/zero of=/dev/asm-diske bs=1024 count=200;
dd if=/dev/zero of=/dev/asm-diskd bs=1024 count=200;
dd if=/dev/zero of=/dev/asm-diskc bs=1024 count=200;200+0 records in
200+0 records out
204800 bytes (205 kB) copied, 0.0884688 s, 2.3 MB/s
[root@rac-two bin]# dd if=/dev/zero of=/dev/asm-diskd bs=1024 count=200;
200+0 records in
200+0 records out
204800 bytes (205 kB) copied, 0.138348 s, 1.5 MB/s
[root@rac-two bin]# dd if=/dev/zero of=/dev/asm-diskc bs=1024 count=200;
200+0 records in
200+0 records out
204800 bytes (205 kB) copied, 0.103544 s, 2.0 MB/s

开始进行恢复：

停止crs资源。

[root@rac-two bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac-two'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac-two'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac-two'
CRS-2673: Attempting to stop 'ora.GIDG.dg' on 'rac-two'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac-two'
CRS-2673: Attempting to stop 'ora.rac.db' on 'rac-two'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.rac-two.vip' on 'rac-two'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac-two'
CRS-2677: Stop of 'ora.rac-two.vip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.rac-two.vip' on 'rac-one'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac-one'
CRS-2677: Stop of 'ora.rac.db' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.DATADG.dg' on 'rac-two'
CRS-2677: Stop of 'ora.DATADG.dg' on 'rac-two' succeeded
CRS-2676: Start of 'ora.rac-two.vip' on 'rac-one' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac-one'
CRS-2677: Stop of 'ora.GIDG.dg' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac-two'
CRS-2677: Stop of 'ora.ons' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac-two'
CRS-2677: Stop of 'ora.net1.network' on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-two' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac-two'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac-two'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-two'
CRS-2677: Stop of 'ora.crf' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-two'
CRS-2677: Stop of 'ora.gipcd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-two'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-two' has completed
CRS-4133: Oracle High Availability Services has been stopped.

启动到独占模式且不启动ora.crsd:

[root@rac-two bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac-two'
CRS-2676: Start of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac-two'
CRS-2676: Start of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac-two'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac-two'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac-two' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac-two'
CRS-2676: Start of 'ora.asm' on 'rac-two' succeeded

创建磁盘组：

[root@rac-two bin]# su - grid
[grid@rac-two ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Fri Mar 28 00:01:05 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup gidg normal redundancy
  2  disk '/dev/asm-diske','/dev/asm-diskd','/dev/asm-diskc' attribute 'au_size'='1M','compatible.asm'='11.2','compatible.rdbms'='11.2','compatible.advm'='11.2';

Diskgroup created.

SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATADG                         DISMOUNTED
TESTDG                         DISMOUNTED
GIDG                           MOUNTED

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@rac-two ~]$ exit

首先要恢复的是ocr

[root@rac-two bin]# ./ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-two-cluster/backup00.ocr
[root@rac-two bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3472
         Available space (kbytes) :     258648
         ID                       :  152256349
         Device/File Name         :      +GIDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

恢复votedisk：

[root@rac-two bin]# ./crsctl query css votedisk
Located 0 voting disk(s).
[root@rac-two bin]# ./crsctl replace votedisk +GIDG
CRS-4602: Failed 27 to add voting file 58c5df27b4884f70bfb38fc4c93b8f72.
CRS-4602: Failed 27 to add voting file 51caa3f834964f5abf6efcfb90476d85.
CRS-4602: Failed 27 to add voting file 92b70ca3ea5d4f80bf9df66a8e975ba9.
Failed to replace voting disk group with +GIDG.
CRS-4000: Command Replace failed, or completed with errors.

提示失败，需要重启crs资源。

[root@rac-two bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac-two'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-two'
CRS-2677: Stop of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-two'
CRS-2677: Stop of 'ora.gipcd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-two'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-two' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac-two bin]# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac-two'
CRS-2676: Start of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac-two'
CRS-2676: Start of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac-two'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac-two'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac-two' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac-two'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac-two'
CRS-2676: Start of 'ora.diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac-two'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac-two'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac-two' succeeded
CRS-2676: Start of 'ora.ctssd' on 'rac-two' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac-two'
CRS-2676: Start of 'ora.asm' on 'rac-two' succeeded
[root@rac-two bin]# su - grid
[grid@rac-two ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Fri Mar 28 00:13:30 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> col name for a30
SQL> col path for a50
SQL> set linesize 200
SQL> select name,path from v$asm_disk;

no rows selected

SQL> alter system set asm_diskstring='/dev/asm-disk*';

System altered.

SQL> select name,path from v$asm_disk;

NAME                           PATH
------------------------------ --------------------------------------------------
                               /dev/asm-diskg
                               /dev/asm-diskd
                               /dev/asm-diskc
                               /dev/asm-diskf
                               /dev/asm-diskh
                               /dev/asm-diskm
                               /dev/asm-diskk
                               /dev/asm-diskj
                               /dev/asm-diski
                               /dev/asm-diskb
                               /dev/asm-diske

NAME                           PATH
------------------------------ --------------------------------------------------
                               /dev/asm-diskl

12 rows selected.

SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATADG                         DISMOUNTED
GIDG                           DISMOUNTED
TESTDG                         DISMOUNTED

SQL> alter diskgroup gidg mount;

Diskgroup altered.

SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
DATADG                         DISMOUNTED
GIDG                           MOUNTED
TESTDG                         DISMOUNTED

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@rac-two ~]$ exit
logout
[root@rac-two bin]# ./ocrctl replace votedisk +GIDG
-bash: ./ocrctl: No such file or directory
[root@rac-two bin]# ./crsctl replace votedisk +GIDG
Successful addition of voting disk 4b55dbd068344fb6bfb89250b3586256.
Successful addition of voting disk 6b157dc173bd4f7ebf1846df01c23d5d.
Successful addition of voting disk 505d811a77d64f57bfa2797afba6a5f5.
Successfully replaced voting disk group with +GIDG.
CRS-4266: Voting file(s) successfully replaced

至此完成了ocr和votedisk的恢复。

重启crs，查看状况：

[root@rac-two bin]# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac-two'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac-two'
CRS-2673: Attempting to stop 'ora.asm' on 'rac-two'
CRS-2677: Stop of 'ora.mdnsd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac-two'
CRS-2677: Stop of 'ora.ctssd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac-two' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac-two'
CRS-2677: Stop of 'ora.cssd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac-two'
CRS-2677: Stop of 'ora.gipcd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac-two'
CRS-2677: Stop of 'ora.gpnpd' on 'rac-two' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-two' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac-two bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[grid@rac-two ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
               ONLINE  ONLINE       rac-two                                      
ora.GIDG.dg
               ONLINE  ONLINE       rac-two                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac-two                                      
ora.TESTDG.dg
               ONLINE  OFFLINE      rac-two                                      
ora.asm
               ONLINE  ONLINE       rac-two                  Started             
ora.gsd
               OFFLINE OFFLINE      rac-two                                      
ora.net1.network
               ONLINE  ONLINE       rac-two                                      
ora.ons
               ONLINE  ONLINE       rac-two                                      
ora.registry.acfs
               ONLINE  ONLINE       rac-two                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac-two                                      
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac-two                                      
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac-two                                      
ora.cvu
      1        ONLINE  ONLINE       rac-two                                      
ora.oc4j
      1        ONLINE  ONLINE       rac-two                                      
ora.rac-one.vip
      1        ONLINE  INTERMEDIATE rac-two                  FAILED OVER         
ora.rac-two.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.rac.db
      1        ONLINE  ONLINE       rac-two                  Open                
      2        ONLINE  OFFLINE                                                   
ora.scan1.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.scan2.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.scan3.vip
      1        ONLINE  ONLINE       rac-two                                      
[grid@rac-two ~]$ 

在节点一重启crs资源，将其加入集群。

[root@rac-one bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac-one bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATADG.dg
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.GIDG.dg
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.TESTDG.dg
               OFFLINE OFFLINE      rac-one                                      
               ONLINE  OFFLINE      rac-two                                      
ora.asm
               ONLINE  ONLINE       rac-one                  Started             
               ONLINE  ONLINE       rac-two                  Started             
ora.gsd
               OFFLINE OFFLINE      rac-one                                      
               OFFLINE OFFLINE      rac-two                                      
ora.net1.network
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.ons
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
ora.registry.acfs
               ONLINE  ONLINE       rac-one                                      
               ONLINE  ONLINE       rac-two                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac-one                                      
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac-two                                      
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac-two                                      
ora.cvu
      1        ONLINE  ONLINE       rac-two                                      
ora.oc4j
      1        ONLINE  ONLINE       rac-two                                      
ora.rac-one.vip
      1        ONLINE  ONLINE       rac-one                                      
ora.rac-two.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.rac.db
      1        ONLINE  ONLINE       rac-two                  Open                
      2        ONLINE  ONLINE       rac-one                  Open                
ora.scan1.vip
      1        ONLINE  ONLINE       rac-one                                      
ora.scan2.vip
      1        ONLINE  ONLINE       rac-two                                      
ora.scan3.vip
      1        ONLINE  ONLINE       rac-two                                      
[root@rac-one bin]# 

至此完成了ocr和votedisk的恢复。

THAT‘s  all！！！！！