断电引起redo和数据文件不一致故障恢复---惜分飞

有些时候故障总是来的让人非常意外,这个在准备停机迁移数据库之前的几分钟由于某种原因直接导致主机掉电,再次开机数据库无法启动

Sat Aug 03 23:10:37 2024 Successful mount of redo thread 1, with mount id 3696805928 Database mounted in Exclusive Mode Lost write protection disabled Completed: alter database mount Sat Aug 03 23:10:43 2024 alter database open Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_6808.trc: ORA-01113: 文件 21 需要介质恢复 ORA-01110: 数据文件 21: 'D:\APP\ADMINISTRATOR\ORADATA\XFF\XIFENFEI.DBF' ORA-1113 signalled during: alter database open...

尝试数据库恢复各种报错ORA-600 kdourp_inorder2,ORA-600 3020,ORA-7445 kdxlin等

ALTER DATABASE RECOVER    CONTINUE DEFAULT  Media Recovery Log D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\XFF\ARCHIVELOG\2024_08_03\O1_MF_1_1159998_MBW605HP_.ARC ORA-279 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ... ALTER DATABASE RECOVER    CONTINUE DEFAULT  Media Recovery Log D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\XFF\ARCHIVELOG\2024_08_03\O1_MF_1_1159999_MBW63QBY_.ARC Sat Aug 03 23:22:10 2024 Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xC] [PC:0x14306B54A, kdxlin()+4432] Sat Aug 03 23:22:10 2024 Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xC] [PC:0x14306B54A, kdxlin()+4432] Sat Aug 03 23:22:10 2024 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr25_7740.trc  (incident=132557): ORA-00600: internal error code, arguments: [kdourp_inorder2], [4], [22], [44], [44], [], [], [], [], [], [], [] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\incident\incdir_132557\xff_pr25_7740_i132557.trc ERROR: Unable to normalize symbol name for the following short stack (at offset 213): dbgexProcessError()+200<-dbgeExecuteForError()+65<-dbgePostErrorKGE()+2269<-dbkePostKGE_kgsf()+77<-kgeade()+562 <-kgerelv()+151<-kgerev()+45<-kgerec5()+60<-sss_xcpt_EvalFilterEx()+1862<-sss_xcpt_EvalFilter()+174 <-.1.4_5+59<-00007FFCB5E2C92F<-00007FFCB5E3D82D<-00007FFCB5DE916B<-00007FFCB5E3C9EE<-kdxlin()+4432 <-kco_issue_callback()+196<-kcoapl()+746<-kcbr_apply_change()+6156<-kcbr_mapply_change()+1162 <-kcbrapply()+2297<-kcbr_apply_pending()+2931<-krp_slave_apply()+1155<-krp_slave_main()+4010<-ksvrdp()+2580 <-opirip()+904<-opidrv()+906<-sou2o()+98<-opimai_real()+280<-opimai()+191<-BackgroundThreadStart()+646 <-00007FFCB562168D<-00007FFCB5E14629 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr1w_6472.trc  (incident=132485): ORA-07445: exception encountered: core dump [kdxlin()+4432] [ACCESS_VIOLATION] [ADDR:0xC] [PC:0x14306B54A] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\incident\incdir_132485\xff_pr1w_6472_i132485.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr2o_7472.trc  (incident=132709): ORA-07445: exception encountered: core dump [kdxlin()+4432] [ACCESS_VIOLATION] [ADDR:0xC] [PC:0x14306B54A] Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\incident\incdir_132709\xff_pr2o_7472_i132709.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Sat Aug 03 23:22:11 2024 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr18_7812.trc  (incident=132301): ORA-00600: internal error code, arguments: [3020], [62], [517633], [260564481], [], [], [], [] ORA-10567: Redo is inconsistent with data block (file# 62, block# 517633, file offset is 4240449536 bytes) ORA-10564: tablespace HSEMR_TAB ORA-01110: data file 62: 'D:\APP\ADMINISTRATOR\ORADATA\XFF\EMR006.DBF' ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK' Sat Aug 03 23:22:56 2024 Slave exiting with ORA-10562 exception Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr25_7740.trc: ORA-10562: Error occurred while applying redo to data block (file# 64, block# 508263) ORA-10564: tablespace HSEMR_TAB ORA-01110: data file 64: 'D:\APP\ADMINISTRATOR\ORADATA\XFF\HSEMR_TAB008.DBF' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 467202 ORA-00600: internal error code, arguments: [kdourp_inorder2], [4], [22], [44], [44], [], [], [] Sat Aug 03 23:22:56 2024 Slave exiting with ORA-10562 exception Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr1w_6472.trc: ORA-10562: Error occurred while applying redo to data block (file# 65, block# 498512) ORA-10564: tablespace HSEMR_TAB ORA-01110: data file 65: 'D:\APP\ADMINISTRATOR\ORADATA\XFF\EMR009.DBF' ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 467200 ORA-00607: Internal error occurred while making a change to a data block ORA-00602: internal programming exception ORA-07445: exception encountered: core dump [kdxlin()+4432] [ACCESS_VIOLATION] [ADDR:0xC] Sat Aug 03 23:22:57 2024 Media Recovery failed with error 448 Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_pr00_6732.trc: ORA-00283: recovery session canceled due to errors ORA-00448: normal completion of background process Sat Aug 03 23:22:57 2024 ORA-600 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ... ALTER DATABASE RECOVER CANCEL ORA-1112 signalled during: ALTER DATABASE RECOVER CANCEL ...

通过分析确认有部分数据文件和redo信息不匹配,导致无法正常recover成功

SQL> recover datafile 77; 完成介质恢复。 SQL> recover datafile 78; ORA-00283: 恢复会话因错误而取消 ORA-00600: 内部错误代码, 参数: [3020], [78], [473221], [327628933], [], [], [], [], [], [], [], [] ORA-10567: Redo is inconsistent with data block (file# 78, block# 473221, file offset is 3876626432 bytes) ORA-10564: tablespace HSEMR_TAB ORA-01110: 数据文件 78: 'D:\APP\ADMINISTRATOR\ORADATA\XFF\HIS23.DBF' ORA-10560: block type 'FIRST LEVEL BITMAP BLOCK' SQL> recover datafile 66; ORA-00279: 更改 6029114092 (在 08/03/2024 19:44:05 生成) 对于线程 1 是必需的 ORA-00289: 建议: D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\XFF\ARCHIVELOG\2024_08_03\O1_MF_1_115999 9_MBW63QBY_.ARC ORA-00280: 更改 6029114092 (用于线程 1) 在序列 #1159999 中 指定日志: {<RET>=suggested | filename | AUTO | CANCEL} auto 已应用的日志。 完成介质恢复。 SQL> recover datafile 65; ORA-00279: 更改 6029114092 (在 08/03/2024 19:44:05 生成) 对于线程 1 是必需的 ORA-00289: 建议: D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\XFF\ARCHIVELOG\2024_08_03\O1_MF_1_115999 9_MBW63QBY_.ARC ORA-00280: 更改 6029114092 (用于线程 1) 在序列 #1159999 中 指定日志: {<RET>=suggested | filename | AUTO | CANCEL} auto ORA-00283: 恢复会话因错误而取消 ORA-10562: Error occurred while applying redo to data block (file# 65, block# 498544) ORA-10564: tablespace HSEMR_TAB ORA-01110: 数据文件 65: 'D:\APP\ADMINISTRATOR\ORADATA\XFF\EMR009.DBF' ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 467200 ORA-00607: 当更改数据块时出现内部错误 ORA-00602: 内部编程异常错误 ORA-07445: 出现异常错误: 核心转储 [kdxlin()+4432] [ACCESS_VIOLATION] [ADDR:0xC] [PC:0x14306B54A] [UNABLE_TO_READ] [] ORA-01112: 未启动介质恢复

对于最终无法正常recover成功数据文件,使用Oracle数据库恢复利器：Oracle Recovery Tools工具快速调整scn

---

然后重建ctl，recover 数据库并open成功

Sun Aug 04 01:01:51 2024 Successful mount of redo thread 1, with mount id 3696824638 Completed: CREATE CONTROLFILE REUSE DATABASE "XFF" NORESETLOGS FORCE LOGGING ARCHIVELOG     MAXLOGFILES 16     MAXLOGMEMBERS 3     MAXDATAFILES 200     MAXINSTANCES 8     MAXLOGXFFTORY 23360 LOGFILE   GROUP 1 'D:\APP\ADMINISTRATOR\ORADATA\XFF\REDO01.LOG'  SIZE 50M BLOCKSIZE 512,   GROUP 2 'D:\APP\ADMINISTRATOR\ORADATA\XFF\REDO02.LOG'  SIZE 50M BLOCKSIZE 512,   GROUP 3 'D:\APP\ADMINISTRATOR\ORADATA\XFF\REDO03.LOG'  SIZE 50M BLOCKSIZE 512 DATAFILE   'D:\APP\ADMINISTRATOR\ORADATA\XFF\SYSTEM01.DBF',   'D:\APP\ADMINISTRATOR\ORADATA\XFF\SYSAUX01.DBF', …… ALTER DATABASE RECOVER  database  Media Recovery Start  started logmerger process Only allocated 127 recovery slaves (requested 128) Parallel Media Recovery started with 127 slaves Sun Aug 04 01:01:56 2024 Recovery of Online Redo Log: Thread 1 Group 2 Seq 1160002 Reading mem 0   Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\XFF\REDO02.LOG Completed: ALTER DATABASE RECOVER  database  Sun Aug 04 01:02:20 2024 alter database open Beginning crash recovery of 1 threads  parallel recovery started with 32 processes Started redo scan Completed redo scan  read 1946 KB redo, 0 data blocks need recovery Started redo application at  Thread 1: logseq 1160002, block 2, scn 6029119350 Recovery of Online Redo Log: Thread 1 Group 2 Seq 1160002 Reading mem 0   Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\XFF\REDO02.LOG Completed redo application of 0.00MB Completed crash recovery at  Thread 1: logseq 1160002, block 3895, scn 6029139793  0 data blocks read, 0 data blocks written, 1946 redo k-bytes read Initializing SCN for created control file Database SCN compatibility initialized to 3 Sun Aug 04 01:02:21 2024 LGWR: STARTING ARCH PROCESSES Sun Aug 04 01:02:21 2024 ARC0 started with pid=71, OS id=2772 ARC0: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC0: STARTING ARCH PROCESSES Sun Aug 04 01:02:22 2024 ARC1 started with pid=72, OS id=7996 Sun Aug 04 01:02:22 2024 ARC2 started with pid=73, OS id=2900 Sun Aug 04 01:02:22 2024 ARC3 started with pid=74, OS id=6856 Archived Log entry 1 added for thread 1 sequence 1160000 ID 0xc4814d77 dest 1: ARC1: Archival started ARC2: Archival started ARC2: Becoming the 'no FAL' ARCH ARC2: Becoming the 'no SRL' ARCH ARC1: Becoming the heartbeat ARCH Thread 1 advanced to log sequence 1160003 (thread open) Thread 1 opened at log sequence 1160003   Current log# 1 seq# 1160003 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\XFF\REDO01.LOG Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Sun Aug 04 01:02:23 2024 SMON: enabling cache recovery Archived Log entry 2 added for thread 1 sequence 1160002 ID 0xc4814d77 dest 1: Archived Log entry 3 added for thread 1 sequence 1160001 ID 0xc4814d77 dest 1: ARC3: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE [7808] Successfully onlined Undo Tablespace 2. Undo initialization finished serial:0 start:7657234 end:7657703 diff:469 (4 seconds) Dictionary check beginning Tablespace 'TEMP' #3 found in data dictionary, but not in the controlfile. Adding to controlfile. Dictionary check complete Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed SMON: enabling tx recovery ********************************************************************* WARNING: The following temporary tablespaces contain no files.          Txff condition can occur when a backup controlfile has          been restored.  It may be necessary to add files to these          tablespaces.  That can be done using the SQL statement:             ALTER TABLESPACE <tablespace_name> ADD TEMPFILE             Alternatively, if these temporary tablespaces are no longer          needed, then they can be dropped.            Empty temporary tablespace: TEMP ********************************************************************* Database Characterset is ZHS16GBK No Resource Manager plan active ********************************************************** WARNING: Files may exists in db_recovery_file_dest that are not known to the database. Use the RMAN command CATALOG RECOVERY AREA to re-catalog any such files. If files cannot be cataloged, then manually delete them using OS command. One of the following events caused txff: 1. A backup controlfile was restored. 2. A standby controlfile was restored. 3. The controlfile was re-created. 4. db_recovery_file_dest had previously been enabled and    then disabled. ********************************************************** replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC Sun Aug 04 01:02:27 2024 QMNC started with pid=75, OS id=7884 LOGSTDBY: Validating controlfile with logical metadata LOGSTDBY: Validation complete Completed: alter database open

后续处理异常表,lob,index等数据,客户业务测试都ok,完成本次恢复工作