tb级别oracle数据库,记一次TB级别的Oracle Exadata数据库恢复

现在它还提供了新的低成本1/8机架配置。 借助Oracle Exadata，客户可通过整合降低IT成本，使存储能力增加10倍，提升所有应用的性能，因消除系统集成的试错过程而加快上市速度，以及实时做出更好的业务决策

某客户的exadata环境，由于大量硬件故障，导致数据库宕机。经过分析确认是由于某个cell节点的某个disk出现异常，导致部分

cell节点的disk 离线后无法进行asm reblance；最终导致diskgroup 被强制dismount了。如下是alert log

Fri Oct 09 15:44:26 2015

NOTE: process _user4655_+asm1 (4655) initiating offline of disk 46.3915916191 (DATA_KLYX_CD_10_KLYXCEL02) with mask 0x7e in group 1

NOTE: checking PST: grp = 1

ERROR: Disk 18 cannot be offlined, since all the disks [18, 49] with mirrored data would be offline.

ERROR: too many offline disks in PST (grp 2)

GMON checking disk modes for group 1 at 55 for pid 41, osid 4655

Fri Oct 09 15:44:26 2015

NOTE: cache dismounting (not clean) group 2/0x4AD8D253 (DBFS_DG)

ERROR: too many offline disks in PST (grp 1)

NOTE: checking PST for grp 1 done.

WARNING: Disk 46 (DATA_KLYX_CD_10_KLYXCEL02) in group 1 in mode 0x7f is now being taken offline on ASM inst 1

NOTE: initiating PST update: grp = 1, dsk = 46/0xe968239f, mode = 0x6a, op = 4

GMON updating disk modes for group 1 at 56 for pid 41, osid 4655

ERROR: Disk 46 cannot be offlined, since all the disks [46, 72] with mirrored data would be offline.

ERROR: too many offline disks in PST (grp 1)

WARNING: Offline of disk 46 (DATA_KLYX_CD_10_KLYXCEL02) in group 1 and mode 0x7f failed on ASM inst 1

WARNING: Disk 18 (DBFS_DG_CD_10_KLYXCEL02) in group 2 mode 0x7f offline is being aborted

Fri Oct 09 15:44:26 2015

NOTE: process _user4663_+asm1 (4663) initiating offline of disk 46.3915916191 (DATA_KLYX_CD_10_KLYXCEL02) with mask 0x7e in group 1

NOTE: checking PST: grp = 1

WARNING: Offline of disk 18 (DBFS_DG_CD_10_KLYXCEL02) in group 2 and mode 0x7f failed on ASM inst 1

Fri Oct 09 15:44:26 2015

NOTE: halting all I/Os to diskgroup 2 (DBFS_DG)

NOTE: unable to offline disks after getting write error for diskgroup DBFS_DG

NOTE: write to disk 32 succeeded

NOTE: disk 18 had IO error

NOTE: write to disk 6 succeeded

NOTE: halting all I/Os to diskgroup 1 (DATA_KLYX)

NOTE: LGWR doing non-clean dismount of group 1 (DATA_KLYX)

NOTE: LGWR sync ABA=21.5465 last written ABA 21.5465

GMON checking disk modes for group 1 at 57 for pid 44, osid 4663

ERROR: too many offline disks in PST (grp 1)

NOTE: checking PST for grp 1 done.

我们可以看到，cell 2节点的第10号disk存在异常，通过cellcli命令行也可以进一步确认：

CellCLI> list physicaldisk;

20:0            E07Q5N          normal

20:1            E05SMD          normal

20:2            E07GV6          normal

20:3            E060YD          normal

20:4            E05ZQE          normal

20:5            E07L9M          normal

20:6            E07FED          normal

20:7            E05N88          normal

20:8            E08DY5          normal

20:9            E06ADQ          normal

20:10           E08E5C          critical

20:11           E08DYF          normal

通过对该磁盘进行修复检查，确实也发现了坏道，如下所示：

SEAGATE ST360057SSUN600G --> SEAGATE ST360057SSUN600G

Sector 0 --> Sector 0

1,172,123,568 Sectors

Sectors that could not be read:

102,578,137

102,578,139

102,578,140

102,578,141

102,578,142

102,578,149

102,578,152

......

1,172,123,561 sector(s) successfully copied.

7 bad source sectors encountered.

Corresponding destination sectors filled with: UNREADABLESECTOR

修复完毕后，成功mount diskgroup。但是最后检查数据库存在部分数据坏块，如下所示：

SQL>  select * from  V$DATABASE_BLOCK_CORRUPTION;

FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTION_TYPE

---------- ---------- ---------- ------------------ -----------------

1    5186816          3                  0 CORRUPT

1    5186813          1                  0 CORRUPT

1    3343734          2                  0 CORRUPT

25    2391372          1         1.4561E+13 CORRUPT

35     538916          1                  0 CORRUPT

36     543804          3                  0 CORRUPT

对于数据坏块的修复相对比较简单，只是由于是asm环境处理起来相对麻烦一些，加上客户没有备份，所以无法进行blockcover。对于system的几个数据坏块，经过检查发现为审计对象，因此通过truncate然后insert数据即可格式化掉；

另外第25号文件的数据坏块为逻辑坏块，通过查询确认为index，直接rebuild index即可。

至于说最后第第35，36号文件，由于均为数据文件，涉及的对象均为分区表的partiiton，因此处理相对麻烦一些。

这里我通过创建测试表，然后清理掉数据，构造一个数据空块，然后将该数据块复制到asm diskgroup中替换到这几个坏块即可，注意替换之前，需要修改数据坏块的rdba地址和obj id信息，如下是数据块的复制替换过程：

SQL> @copy_block_to_asm.sql

Enter value for file_with_patched_block: /u01/.../blk.dbf

old  15: v_FsFileName := '&file_with_patched_block';

new  15: v_FsFileName := '/u01/.../blk.dbf';

Enter value for file_to_patch_in_asm: +DATA_KLYX/xxxx1/datafile/epmrange107.dbf

old  16: v_AsmFileName := '&file_to_patch_in_ASM';

new  16: v_AsmFileName := '+DATA_KLYX/xxxx1/datafile/epmrange107.dbf';

Enter value for block_to_patch: 543804

old  17: v_offstart := '&block_to_patch';

new  17: v_offstart := '543804';

PL/SQL procedure successfully completed.

最后通过rman检查，确认一切正常。如下是rman的检测结果：

.......

RMAN> backup validate datafile 36;

Starting backup at 16-OCT-15

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=1337 instance=xxx1 device type=DISK

channel ORA_DISK_1: starting full datafile backup set

channel ORA_DISK_1: specifying datafile(s) in backup set

input datafile file number=00036 name=+DATA_KLYX/xxx1/datafile/epmrange107.dbf

channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15

List of Datafiles

=================

File Status Marked Corrupt Empty Blocks Blocks Examined High SCN

---- ------ -------------- ------------ --------------- ----------

36   OK     0              315258       3276800         14561048845049

File Name: +DATA_KLYX/xxx1/datafile/epmrange107.dbf

Block Type Blocks Failing Blocks Processed

---------- -------------- ----------------

Data       0              2926354

Index      0              20

Other      0              35168

......