Oracle 11g RAC ASM VOTEDISK所在磁盘全部丢失重大故障恢复

Oracle 11g RAC ASM VOTEDISK所在磁盘全部丢失恢复


一、环境描述
系统：AIX 7100-02-03-1334
数据库：Oracle 11.2.0.3.10 RAC ASM


二、故障描述
存储故障导致ASM VOTEDISK所在磁盘全部丢失。
因为OCR和VOTEDISK的丢失，集群服务终止，OHAS进程还在。


三、操作步骤
思路：使用CRS自动备份的文件恢复OCR，VOTEDISK。
1)所有节点停掉CRS服务
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl  stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdb3'
CRS-2673: Attempting to stop 'ora.crf' on 'testdb3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testdb3'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'testdb3'
CRS-2677: Stop of 'ora.cssdmonitor' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.crf' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'testdb3'
CRS-2677: Stop of 'ora.gipcd' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdb3'
CRS-2677: Stop of 'ora.gpnpd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'testdb3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testdb3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
root@testdb3:/#
root@testdb3:/#ps -ef |grep grid
    root  2884826  3146642   0 15:56:44  pts/0  0:00 grep grid


2)在一个节点上以NOCRS方式启动CRS，此操作会启动ASM实例。
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'testdb3'
CRS-2676: Start of 'ora.mdnsd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'testdb3'
CRS-2676: Start of 'ora.gpnpd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'testdb3'
CRS-2672: Attempting to start 'ora.gipcd' on 'testdb3'
CRS-2676: Start of 'ora.cssdmonitor' on 'testdb3' succeeded
CRS-2676: Start of 'ora.gipcd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'testdb3'
CRS-2672: Attempting to start 'ora.diskmon' on 'testdb3'
CRS-2676: Start of 'ora.diskmon' on 'testdb3' succeeded
CRS-2676: Start of 'ora.cssd' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'testdb3'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'testdb3'
CRS-2672: Attempting to start 'ora.ctssd' on 'testdb3'
CRS-2676: Start of 'ora.ctssd' on 'testdb3' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'testdb3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'testdb3' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'testdb3'
CRS-2681: Clean of 'ora.asm' on 'testdb3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'testdb3'
CRS-2676: Start of 'ora.asm' on 'testdb3' succeeded


3)新增一块盘，修改权限并在ASM中查看磁盘状态，创建一个新的磁盘组
root@testdb3:/#chown grid:asmadmin /dev/vx/rdmp/hitachi_vsp0_11d4
root@testdb3:/#chmod 660 /dev/vx/rdmp/hitachi_vsp0_11d4
root@testdb4:/#chown grid:asmadmin /dev/vx/rdmp/hitachi_vsp0_11d4
root@testdb4:/#chmod 660 /dev/vx/rdmp/hitachi_vsp0_11d4


root@testdb3:/#su - grid
$ sqlplus  / as sysasm


SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 16:14:51 2014


Copyright (c) 1982, 2011, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options


SQL> set pagesize 10000
SQL> set linesize 300
SQL> alter system set asm_diskstring = '/dev/vx/rdmp/*';


System altered.


SQL> set pagesize 10000 linesize 300
SQL> select group_number, disk_number, state, path from v$asm_disk order by 1,2;


GROUP_NUMBER DISK_NUMBER STATE    PATH
------------ ----------- -------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
           0           0 NORMAL   /dev/rhitachi_v0_11cc
           0           1 NORMAL   /dev/rhitachi_v0_11cd
           0           2 NORMAL   /dev/rhitachi_v0_11ce
           0           3 NORMAL   /dev/rhitachi_v0_11cf
           0           4 NORMAL   /dev/rhitachi_v0_11d0
           0           5 NORMAL   /dev/rhitachi_v0_11d1
           0           6 NORMAL   /dev/rhitachi_v0_11d2
           0           7 NORMAL   /dev/rhitachi_v0_11d3
           0           8 NORMAL   /voting_disk/vote_disk_nfs
           0           9 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11cc
           0          10 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11cd
           0          11 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11ce
           0          12 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11cf
           0          13 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11d0
           0          14 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11d1
           0          15 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11d2
           0          16 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11d3
           0          17 NORMAL   /dev/vx/rdmp/hitachi_vsp0_11d4


18 rows selected.




SQL> create diskgroup OCRVOTE1 external redundancy
  2  disk '/dev/vx/rdmp/hitachi_vsp0_11d4'
  3  ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';


Diskgroup created.


4)修改/etc/oracle/ocr.loc文件，将ocrconfig_loc修改成新增的磁盘组名称，以便将OCR恢复到新的磁盘组中。
root@testdb3:/#cat /etc/oracle/ocr.loc
ocrconfig_loc=+OCRVOTE
local_only=FALSE


root@testdb3:/#vi /etc/oracle/ocr.loc
ocrconfig_loc=+OCRVOTE1
local_only=FALSE


5)从以前OCR备份中恢复OCR
root@testdb3:/#/oracle/app/11.2.0/grid/bin/ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy


testdb4     2014/07/03 15:55:26     /oracle/app/11.2.0/grid/cdata/testdb/backup00.ocr


testdb4     2014/07/03 11:55:25     /oracle/app/11.2.0/grid/cdata/testdb/backup01.ocr


testdb4     2014/07/03 07:55:25     /oracle/app/11.2.0/grid/cdata/testdb/backup02.ocr


testdb4     2014/07/01 23:55:22     /oracle/app/11.2.0/grid/cdata/testdb/day.ocr


testdb4     2014/06/20 07:54:56     /oracle/app/11.2.0/grid/cdata/testdb/week.ocr


testdb2     2014/05/27 09:37:16     /oracle/app/11.2.0/grid/cdata/testdb/backup_20140527_093716.ocr


testdb2     2014/05/27 09:37:10     /oracle/app/11.2.0/grid/cdata/testdb/backup_20140527_093710.ocr


testdb4     2014/05/26 16:06:13     /oracle/app/11.2.0/grid/cdata/testdb/backup_20140526_160613.ocr


testdb4     2014/05/26 16:05:54     /oracle/app/11.2.0/grid/cdata/testdb/backup_20140526_160554.ocr


testdb4     2014/05/26 16:04:48     /oracle/app/11.2.0/grid/cdata/testdb/backup_20140526_160448.ocr
root@testdb3:/#/oracle/app/11.2.0/grid/bin/ocrconfig  -restore /oracle/app/11.2.0/grid/cdata/testdb/backup00.ocr
root@testdb3:/#/oracle/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3244
         Available space (kbytes) :     258876
         ID                       : 1187520997
         Device/File Name         :  +OCRVOTE1
                                    Device/File integrity check succeeded


                                    Device/File not configured


                                    Device/File not configured


                                    Device/File not configured


                                    Device/File not configured


         Cluster registry integrity check succeeded


         Logical corruption check succeeded


6)创建VOTEDISK
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl  replace votedisk +OCRVOTE1
Successful addition of voting disk 833551381fb94fd0bf09eefda7f1b380.
Successful deletion of voting disk 89210622f0864ff0bf9517205691e679.
Successful deletion of voting disk 55c4ee685a824ff3bf6ce510bf09468e.
Successful deletion of voting disk 159234e88fe64f55bf0d4571362c3b07.
Successfully replaced voting disk group with +OCRVOTE1.
CRS-4266: Voting file(s) successfully replaced
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   833551381fb94fd0bf09eefda7f1b380 (/dev/vx/rdmp/hitachi_vsp0_11d4) [OCRVOTE1]
Located 1 voting disk(s).


7)创建SPFILE并按照以前标准修改ASM参数，重启ASM
$ sqlplus / as sysasm


SQL*Plus: Release 11.2.0.3.0 Production on Tue Jul 15 16:28:37 2014


Copyright (c) 1982, 2011, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options


SQL> create spfile='+OCRVOTE1' from memory;


File created.


SQL> startup force mount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started


Total System Global Area  342388736 bytes
Fixed Size                  2221360 bytes
Variable Size             315001552 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled




8)重启CRS
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl  stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'testdb3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'testdb3'
CRS-2673: Attempting to stop 'ora.ctssd' on 'testdb3'
CRS-2673: Attempting to stop 'ora.asm' on 'testdb3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'testdb3'
CRS-2677: Stop of 'ora.ctssd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.asm' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'testdb3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'testdb3'
CRS-2677: Stop of 'ora.cssd' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'testdb3'
CRS-2677: Stop of 'ora.gipcd' on 'testdb3' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'testdb3'
CRS-2677: Stop of 'ora.gpnpd' on 'testdb3' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'testdb3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'testdb3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
root@testdb3:/#/oracle/app/11.2.0/grid/bin/crsctl  start crs 
CRS-4123: Oracle High Availability Services has been started.


9)启动节点2的CRS
修改节点2的ocr.loc
root@testdb4:/#vi /etc/oracle/ocr.loc
ocrconfig_loc=+OCRVOTE1
local_only=FALSE


root@testdb4:/#/oracle/app/11.2.0/grid/bin/crsctl  start crs 
CRS-4123: Oracle High Availability Services has been started.


10)查看CRS资源状态，已恢复完成
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg
               ONLINE  OFFLINE      testdb3                                      
               ONLINE  OFFLINE      testdb4                                      
ora.DATADG.dg
               ONLINE  OFFLINE      testdb3                                      
               ONLINE  OFFLINE      testdb4                                      
ora.LISTENER.lsnr
               ONLINE  ONLINE       testdb3                                      
               ONLINE  ONLINE       testdb4                                      
ora.OCRVOTE.dg
               ONLINE  OFFLINE      testdb3                                      
               ONLINE  OFFLINE      testdb4                                      
ora.SYSDG.dg
               ONLINE  OFFLINE      testdb3                                      
               ONLINE  OFFLINE      testdb4                                      
ora.asm
               ONLINE  ONLINE       testdb3                  Started             
               ONLINE  ONLINE       testdb4                  Started             
ora.gsd
               OFFLINE OFFLINE      testdb3                                      
               OFFLINE OFFLINE      testdb4                                      
ora.net1.network
               ONLINE  ONLINE       testdb3                                      
               ONLINE  ONLINE       testdb4                                      
ora.ons
               ONLINE  ONLINE       testdb3                                      
               ONLINE  ONLINE       testdb4                                      
ora.registry.acfs
               ONLINE  ONLINE       testdb3                                      
               ONLINE  ONLINE       testdb4                                      
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       testdb3                                      
ora.cvu
      1        ONLINE  ONLINE       testdb3                                      
ora.oc4j
      1        ONLINE  ONLINE       testdb3                                      
ora.scan1.vip
      1        ONLINE  ONLINE       testdb3                                      
ora.testdb.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
      3        ONLINE  OFFLINE                               Instance Shutdown   
ora.testdb3.vip
      1        ONLINE  ONLINE       testdb3                                      
ora.testdb4.vip
      1        ONLINE  ONLINE       testdb4                                      
$ 

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/25964700/viewspace-1653709/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/25964700/viewspace-1653709/