1 故障现象
1.1 使用orccheck命令报错
[root@rac1 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage
1.2 使用ocrconfig -restore恢复仍然报错
[root@rac1 rac-cluster]#ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
PROT-35: The configured OCR locations are not accessible.
2 处理步骤
2.1 强制停掉两个节点的crs(两个节点都要执行)
[root@rac1 rac-cluster]# crsctl stop crs -f
2.2 在节点一上已独占方式启动(只启动ASM实例 但不启动CRS)
[root@rac1 ~]#crsctl start crs -excl -nocrs
2.3 重建原votedisk所在的磁盘组
2.4 使用最新的OCR备份进行恢复
[root@rac1 ~]#ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
[root@rac1 ~]#ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2900
Available space (kbytes) : 259220
ID : 1207909490
Device/File Name : +SYSTEMDG
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#crsctl query css votedisk
Located 0 voting disk(s).
[grid@rac1 ~]$ crsctl replace votedisk +SYSTEMDG
Successful addition of voting disk a0e3b9c0c5934f79bf67aa783618b59d.
Successful addition of voting disk 422a673bdbc94f97bff51f66f84a105e.
Successful addition of voting disk a1b110709eb84f39bf351f1b01aa0c7d.
Successfully replaced voting disk group with +SYSTEMDG.
CRS-4266: Voting file(s) successfully replaced
[root@rac1 rac-cluster]#crsctl query css votedisk
##STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 03bd9e6851cf4fb4bf4c92e24a3d71cc (/dev/asm-diskb) [SYSTEMDG]
2. ONLINE 362f2cfb5b3d4f29bf319c668e0efbe4 (/dev/asm-diskc) [SYSTEMDG]
3. ONLINE 57659c53f4284fbdbfeabafb20c3fbdd (/dev/asm-diskd) [SYSTEMDG]
Notes:如果在恢复时遇到如下错误:
[grid@rac1 ~]$ crsctl replace votedisk +SYSTEMDG
CRS-4602: Failed 27 to add voting file a0e3b9c0c5934f79bf67aa783618b59d.
CRS-4602: Failed 27 to add voting file 422a673bdbc94f97bff51f66f84a105e.
CRS-4602: Failed 27 to add voting file a1b110709eb84f39bf351f1b01aa0c7d.
Failed to replace voting disk group with +SYSTEMDG.
CRS-4000: Command Replace failed, or completed with errors.
--登陆grid用户,检查并修改asm_diskstring参数
SQL> alter system set asm_diskstring='/dev/asm*';
System altered.
--检查并修改spfile参数,并重启ASM
SQL> create spfile from memory;
File created.
2.6 重启crs
[root@rac1 ~]# crsctl stop crs
[root@rac1 ~]# crsctl start has
2.7 由于启动较慢,过几分钟查看状态
但是在启动db的时候报错了.........
PRCR-1079 : Failed to start resource ora.orcl.db
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-00205: error in identifying control file, check alert log for more info
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-00205: error in identifying control file, check alert log for more info
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.orcl.db' on 'rac1' failed
CRS-2674: Start of 'ora.orcl.db' on 'rac2' failed
CRS-2632: There are no more servers to try to place resource 'ora.orcl.db' on that would satisfy its placement policy
--查看了下trace发现SYSTEMDG中的控制文件没了,装RAC的时候就没规划好,导致控制文件放到了votedisk里,但是SYSTEMDG刚刚被重建了,所以丢了。
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.8.222)(PORT=1521))' SCOPE=MEMORY SID='orcl1';
ALTER DATABASE MOUNT /* db agent *//* {1:40511:558} */
This instance was first to mount
NOTE: Loaded library: System
SUCCESS: diskgroup DATADG was mounted
SUCCESS: diskgroup SYSTEMDG was mounted
NOTE: dependency between database orcl and diskgroup resource ora.DATADG.dg is established
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+SYSTEMDG/orcl/controlfile/current.261.904952269'
ORA-17503: ksfdopn:2 Failed to open file +SYSTEMDG/orcl/controlfile/current.261.904952269
ORA-15012: ASM file '+SYSTEMDG/orcl/controlfile/current.261.904952269' does not exist
ORA-205 signalled during: ALTER DATABASE MOUNT /* db agent *//* {1:40511:558} */...
Sat Feb 27 23:48:14 2016
Shutting down instance (abort)
License high water mark = 1
USER (ospid: 21150): terminating the instance
Instance terminated by USER, pid = 21150
Sat Feb 27 23:48:14 2016
Instance shutdown complete
2.8 将数据库启动到nomount阶段
[grid@rac1 trace]$ srvctl start database -d orcl -o nomount
2.9 在节点一上使用rman恢复一份控制文件到SYSTEMDG上
[oracle@rac1 ~]$ rman target /
Recovery Manager: Release 11.2.0.4.0 - Production on Sat Feb 27 23:53:35 2016
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORCL (not mounted)
RMAN> restore controlfile to '+SYSTEMDG' from '+DATADG/orcl/controlfile/current.260.901699963';
Starting restore at 2016/02/27 23:53:43
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=34 instance=orcl1 device type=DISK
channel ORA_DISK_1: copied control file copy
Finished restore at 2016/02/27 23:53:52
2.10 已恢复好的控制文件路径和名称
ASMCMD> pwd
+systemdg/orcl/CONTROLFILE
ASMCMD> ls
current.261.904953225
2.11 节点一上登陆db,修改control_files参数
[oracle@rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 27 23:55:09 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> alter system set control_files='+DATADG/orcl/controlfile/current.260.901699963','+SYSTEMDG/orcl/CONTROLFILE/current.261.904953225' scope=spfile sid='*';
System altered.
2.12 重启数据库,并检查数据库状态
[grid@rac1 trace]$ srvctl stop database -d orcl
[grid@rac1 trace]$ srvctl start database -d orcl
[grid@rac1 trace]$ srvctl status database -d orcl
Instance orcl1 is running on node rac1
Instance orcl2 is running on node rac2
保险起见,查看两个节点上的控制文件参数是否正确:
[oracle@rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 27 23:58:47 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> show parameter control_file
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------------------------------------------------------------------------
control_file_record_keep_time integer 7
control_files string +DATADG/orcl/controlfile/current.260.901699963, +SYSTEMDG/orcl/controlfile/current.261.904953225
[oracle@rac2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 27 23:59:29 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> show parameter control_file
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------------------------------------------------------------------------
control_file_record_keep_time integer 7
control_files string +DATADG/orcl/controlfile/current.260.901699963,