OCR和voting disk损坏恢复

OCR和voting disk损坏恢复

原创魔鬼之武

介绍

有时候服务器或存储的突发情况，导致数据库服务器重启同时出现OCR磁盘丢失报错，导致oracle数据库集群宕机。这时，就需要我们来恢复OCR磁盘和voting disk，从而恢复集群！整个过程中，数据库是无法使用的！

本文数据库版本11.2.0.4 RAC，使用dd命令来模拟ocr磁盘损坏，然后通过ocr的自动备份来恢复ocr信息，然后重建voting disk！

检查当前可用资源

[grid@rac1 ~]$ asmcmd

ASMCMD> lsdg

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name

MOUNTED EXTERN N 512 4096 1048576 20473 18230 0 18230 0 N DBFILE/

MOUNTED EXTERN N 512 4096 1048576 13178 12448 0 12448 0 N FRA/

MOUNTED NORMAL N 512 4096 1048576 6141 5215 2047 1584 0 Y OCR/

检查ocr备份

[grid@rac1 ~]$ ocrconfig -showbackup

 

注意：因为我使用的是虚拟机，且不常使用这个环境，所以这个自动备份时间有点早，但是依然可以使用！

检查vote disk

[grid@rac1 ~]$ crsctl query css votedisk

 

图中显示的是当前votedisk相关信息！

创建pfile文件

创建一个ASM实例的pfile文件，后面会用到！！！ ---- 或者可以手动创建一个

SQL> create pfile= '/tmp/pfile' from spfile;

模拟OCR磁盘损坏

由于OCR磁盘组使用normal冗余模式，所以损坏掉了两个才无法启动！！！

使用DD命令将OCR的磁盘损坏！！！

dd if=/dev/zero of=/dev/mapper/mpathfp1 bs=1M count=100 ----损坏一块照常工作！！！

dd if=/dev/zero of=/dev/mapper/mpathgp1 bs=1M count=100

dd if=/dev/zero of=/dev/mapper/mpathhp1 bs=1M count=100

这里使用dd命令将磁盘头的信息清空！！

等一段时间后，grid下的alter日志会出现如下信息：

[cssd(2912)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/oracleasm/disks/OCR01 will be considered not functional in 99370 milliseconds

[cssd(2957)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /dev/oracleasm/disks/OCR01 will be considered not functional in 49600 milliseconds

.......

[crsd(3534)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:14:33.229:

[/u01/app/grid/bin/oraagent.bin(3693)]CRS-5822:Agent '/u01/app/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:3:6} in /u01/app/grid/log/rac1/agent/crsd/oraagent_grid/oraagent_grid.log.

如果需要更快出现报错，直接重启CRS！！！！

[root@rac1 bin]# ps -elf|grep smon

4 S root 2914 1 3 -40 - - 193678 hrtime 12:38 ? 00:01:21 /u01/app/grid/bin/osysmond.bin

0 S grid 3500 1 0 80 0 - 54961 semtim 12:39 ? 00:00:00 asm_smon_+ASM1

0 S root 9469 2759 0 80 0 - 25814 pipe_w 13:17 pts/0 00:00:00 grep smon

[root@rac1 bin]# ./crsctl stop crs

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-4687: Shutdown command has completed with errors.

CRS-4000: Command Stop failed, or completed with errors.

[root@rac1 bin]# ./crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

进程还在但是无法显示集群任何信息！！！！！

日志文件中会显示：

[crsd(9190)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:11.080:

[crsd(9190)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:11.668:

[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

2021-03-18 13:15:13.158:

[crsd(9203)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:13.170:

[crsd(9203)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:13.785:

[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

2021-03-18 13:15:15.274:

[crsd(9220)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:15.294:

[crsd(9220)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage

]. Details at (:CRSD00111:) in /u01/app/grid/log/rac1/crsd/crsd.log.

2021-03-18 13:15:20.339:

[ohasd(2636)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

2021-03-18 13:15:20.340:

[ohasd(2636)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.

2021-03-18 13:15:20.420:

[ohasd(2636)]CRS-2769:Unable to failover resource 'ora.crsd'.

........

[/u01/app/grid/bin/oraagent.bin(9910)]CRS-5019:All OCR locations are on ASM disk groups [OCR], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".

[client(12353)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.

[client(12406)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.

[client(12424)]CRS-10001:18-Mar-21 13:31 ACFS-9290: Waiting for ASM to shutdown.

由于OCR磁盘损坏了，在正常停止crs时，会一直等待 ASM关闭，所以这里直接手动abort掉！！！

强制停止crs

[root@rac1 bin]# ./crsctl stop crs -f

注意：使用-f 强制停止crs，把2个节点的crs都停止掉！

以独占模式启动crs

在节点1上：以独占模式启动crs

[root@rac1 bin]# ./crsctl start crs -excl -nocrs

参数说明：

-excl 该参数指定使用独占模式

-nocrs 该参数指定忽略查找crs及voting

11.2.0.1:

# $CRS_HOME/bin/crsctl start crs -excl

11.2.0.2 and above:

# $CRS_HOME/bin/crsctl start crs -excl -nocrs

使用创建好的pfile文件重启启动asm实例

这里使用最开始已经准备好的pfile文件(grid用户执行)

SQL>startup pfile='/tmp/pfile';

查询信息可以看到ASM磁盘信息为空：

SQL>select group_number, disk_number, mount_status, header_status, path from v$asm_disk;

创建OCR磁盘组

我们还是使用原来asm磁盘来重新创建一个OCR磁盘组，并且取同样的名称！(grid用户)

sqlplus / as sysasm

SQL>create diskgroup OCR normal redundancy disk '/dev/oracleasm/disks/OCR01','/dev/oracleasm/disks/OCR02', '/dev/oracleasm/disks/OCR03' attribute 'compatible.asm'='11.2.0.0.0';

发现OCR已经上线，但是磁盘组中没有任何文件！！！其他两个磁盘组没有损坏，所以文件依旧存在。

[grid@rac1 ~]$ asmcmd

ASMCMD> lsdg

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name

MOUNTED EXTERN N 512 4096 1048576 20473 18230 0 18230 0 N DBFILE/

MOUNTED EXTERN N 512 4096 1048576 13178 12448 0 12448 0 N FRA/

MOUNTED NORMAL N 512 4096 1048576 6141 5982 2047 1967 0 N OCR/

创建ASM实例的spfile文件

SQL>create spfile='+OCR' from pfile='/tmp/pfile';

还原OCR文件

OCR文件会有自动备份功能：
/u01/app/grid/cdata/raccluster

找一个最新的文件用来做还原！！！！

[root@rac1 ~]# cd /u01/app/grid/bin/

[root@rac1 bin]# ./ocrconfig -restore /u01/app/grid/cdata/raccluster/backup00.ocr

检查OCR：

 

恢复voting disk

如果遇到报错，需要将crs强制停止，然后重新以独占模式启动！！！！

[root@rac1 bin]# ./crsctl replace votedisk +OCR

 

[root@rac1 bin]# ./crsctl query css votedisk

 

检查集群环境

强制关闭crs,然后正常重启2个节点的crs！！！！ 查看日志， 启动需要一定时间！！！

[root@rac1 bin]# ./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

通过下面的命令检查：

crsctl status resource -t

ocrcheck

crsctl query css votedisk

crsctl check cluster -all

检查完成后，简单测试下数据库能否正常使用！

注意：直接重启服务器后，,有可能丢失磁盘标签，导致找不到磁盘，这时通过查看alter日志可以发现，报错信息是找不到OCR磁盘，这是因为使用dd命令有可能将磁盘上得asm磁盘标签给清理了，需要重新将标签补充上！！！！！

oracleasm renamedisk -f /dev/mapper/mpathfp1 ocr01

oracleasm renamedisk -f /dev/mapper/mpathgp1 ocr02

oracleasm renamedisk -f /dev/mapper/mpathhp1 ocr03