超融合硬件损坏导致Oracle RAC异常恢复实录

墨墨导读：一套Oracle RAC环境运行在HW超融合环境中，由于硬件问题导致数据库crash，期间出现了不少数据坏块，本文详述整个恢复过程，希望对大家有帮助。

前几天某客户遇到一个棘手问题：其一套Oracle RAC环境运行在HW超融合环境中，由于硬件问题导致数据库crash，期间出现了不少数据坏块，不过还好客户有RMAN物理备份，因此客户提前进行了全库Restore。

首先我们来看下相关日志信息：

Trying mirror side DATADG_0002.
Reread of blocknum=145005, file=+DATADG/xxxxx/datafile/supervise. found same corrupt data
Reread of blocknum=145005, file=+DATADG/xxxxx/datafile/supervise. found valid data
Sun Apr 26 01:22:35 2020
Hex dump of (file 11, block 230454) in trace file /u01/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_13254.trc
Corrupt block relative dba: 0x02c38436 (file 11, block 230454)
Bad header found during backing up datafile
Data in bad block:
type: 6 format: 2 rdba: 0x0082f736
last change scn: 0x0000.4b5a95d7 seq: 0x1 flg: 0x06
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x95d70601
check value in block header: 0xa3d1
computed block checksum: 0x0
Trying mirror side DATADG_0002.
Reread of blocknum=230454, file=+DATADG/xxxxx/datafile/supdb_etyy.dbf. found same corrupt data
Reread of blocknum=230454, file=+DATADG/xxxxx/datafile/supdb_etyy.dbf. found valid data
Hex dump of (file 11, block 230458) in trace file /u01/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ora_13254.trc
Corrupt block relative dba: 0x02c3843a (file 11, block 230458)

上述类似的大量坏块信息，最终导致数据库挂掉。

我们可以看到客户这里asm diskgroup为normal冗余；当primary extent数据有问题时，Oracle会尝试从mirror extent去获取；如果mirror extent是正常的，那么Oracle会自动进行修复，否则会导致数据丢失，严重的话会导致数据库宕机。

Mon Apr 27 02:01:15 2020
Read of datafile '+DATADG/orclsk/datafile/sysaux.260.956077231' (fno 2) header failed with ORA-01207
Rereading datafile 2 header from mirror side 'DATADG_0000' failed with ORA-01207
Rereading datafile 2 header from mirror side 'DATADG_0006' failed with ORA-01207
Errors in file /u01/app/oracle/diag/rdbms/orclsk/orclsk1/trace/orclsk1_ckpt_4893.trc:
ORA-63999: data file suffered media failure
ORA-01122: database file 2 failed verification check
ORA-01110: data file 2: '+DATADG/orclsk/datafile/sysaux.260.956077231'
ORA-01207: file is more recent than control file - old control file
Errors in file /u01/app/oracle/diag/rdbms/xxxxx/xxxxx1/trace/xxxxx1_ckpt_4893.trc:
ORA-63999: data file suffered media failure
ORA-01122: database file 2 failed verification check
ORA-01110: data file 2: '+DAT