Hdr: 21458357 11.2.0.4.0 RDBMS 11.2.0.4.0 DATAGUARD_PSBY PRODID-5 PORTID-23
Abstract: TERMINAL RECOVERY LEFT TRSTAMP IN DATAFILE HEADERS AFTER FLASHBACK
*** 07/16/15 08:42 am ***
----
terminal
PROBLEM:
--------
Customer created Guarantee restore point before they open the standby DB as
a read-write DB.
After they were done with their task, they wanted to n use that GRP to roll
back standby and got the following errors.
AOTS-M: RFS[3]: No connections allowed during/after terminal recovery
Errors during flashback
ORA-283: recovery session canceled due to errors
ORA-38861: flashback recovery stopped before reaching recovery target
ORA-16016: archived log for thread 1 sequence# 398341 unavailable
DIAGNOSTIC ANALYSIS:
--------------------
Sequence Of Events:-
====================
> On "Mon Jul 13 13:19:21 2015" Standby recovery was using "current logfile"
clause.
> On "Tue Jul 14 15:52:18 2015" it Started recovery of Thread 1 seq 398341
message in alert ==> "Recovery of Online Redo Log: Thread 1 Group 18 Seq
398341 Reading mem 0"
> AT "Tue Jul 14 15:53:24 2015" Recovery was cancelled when it was applying
logfile Thread 1 seq 398341
> Recovered data files to a consistent state at change 14228487620253
> Tue Jul 14 15:53:48 2015 Created guaranteed restore point
BEFORE_OPEN_STND_TO_PRIMARY
> Tue Jul 14 15:55:36 2015 alter database recover managed standby database
finish was issued to acitvate the database
> Again it tried to read the Thread 1 seq 398341 "Identified End-Of-Redo
(failover) for thread 1 sequence 398341 at SCN 0xffff.ffffffff"
> Incomplete Recovery applied until change 14228487668121
> Standby became primary SCN: 14228487620252
> Tue Jul 14 17:12:00 2015 flashback to restore point
BEFORE_OPEN_STND_TO_PRIMARY
> Tue Jul 14 17:33:19 2015 Created guaranteed restore point
BEFORE_SP5_STANDBY
Tue Jul 14 17:39:12 2015 Media Recovery Waiting for thread 1 sequence 398341
Cannot recover datafiles with current logs containing different Terminal
Recovery Stamps (TRSTAMP)
Analysis:-
==========
> From the above sequence of events it appears that managed recovery on
standby was stopped for the first time when recovery for thread 1 sequence
398341 was in progress.
> Then standby was activated and again it tried to read the Thread 1 seq
398341 "Identified End-Of-Redo (failover) for thread 1 sequence 398341 at SCN
0xffff.ffffffff"
> This file thread 1 sequence 398341 was later archived on the standby
database.
> Then the later the database was flashed back to
BEFORE_OPEN_STND_TO_PRIMARY, even at that point it was looking for recovery
from thread 1 sequence 398341
> Even when restore point BEFORE_SP5_STANDBY was created, recovery was
waiting for recovery from thread 1 sequence 398341 to maintain the datafiles
at a consistent state.
> While reading thread 1 sequence 398341 it reported "Cannot recover
datafiles with current logs containing different Terminal Recovery Stamps
(TRSTAMP)"
> When the file generated on the primary was applied recovery was able to
continue applying the logs.
It appears that when recovery was cancelled the first time at "Tue Jul 14
15:52:18 2015" datafiles were not consistent and were needing recovery from
sequence 398341 which could have led to the issue. but am not too sure
because the datafile header bits for recovery were not checked.
Could not find a shut abort issue like specified in bug 16005924
Like in Alter Database Activate Standby Database failed with Ora-1196 (Doc
ID 1150720.1) ..........but here it was " Recovered data files to a
consistent state at change 14228487620253"
Incomplete Recovery applied until change 14228487668121 time 07/14/2015
15:54:46
Standby terminal recovery start SCN: 14228487620253
RESETLOGS after incomplete recovery UNTIL CHANGE 14228487668121
Standby became primary SCN: 14228487620252 ===> Standby became primary SCN:
14228487620252
This looks like a bug cause from above it looks like recovery was performed
till 14228487668121 and standby became primary at a lower scn 14228487620252
so requiring recovery from sequence 398341
WORKAROUND:
-----------
We did some manual recovery on standby
- Opted for a manual recovery on standby
recovery standby database
- supped the logfile
'/orabkup/cts/fs01/ctsdb01/CTSDB01_DG2_PRD/archivelog/2015_07_14/o1_mf_1_39834
1_btbdg4n8_.arc';
The recovery proceeding
RELATED BUGS:
-------------
REPRODUCIBILITY:
----------------
TEST CASE:
----------
STACK TRACE:
------------
SUPPORTING INFORMATION:
-----------------------
24 HOUR CONTACT INFORMATION FOR P1 BUGS:
----------------------------------------
9086351040
DIAL-IN INFORMATION:
--------------------
IMPACT DATE:
------------
*** 07/16/15 08:43 am ***
*** 07/16/15 09:23 am ***
*** 07/16/15 09:25 am ***
*** 07/16/15 09:28 am ***
*** 07/16/15 09:39 am *** (CHG: Sta->16)
*** 07/16/15 11:04 am *** (CHG: Sta->10)
*** 07/16/15 11:04 am ***
*** 07/16/15 11:09 am ***
*** 07/16/15 12:48 pm ***
*** 07/18/15 11:34 am *** (CHG: Sta->16)
*** 07/18/15 11:34 am ***
*** 07/18/15 03:40 pm ***
*** 07/20/15 02:12 pm ***
*** 07/20/15 04:45 pm *** (CHG: Sta->10 Asg->BMSMITH SubComp->DATAGUARD_PSBY)
*** 07/20/15 04:45 pm ***
*** 07/22/15 11:14 am ***
*** 07/22/15 11:14 am *** (CHG: Sta->16)
*** 07/22/15 11:15 am ***
*** 07/22/15 11:59 am ***
*** 07/22/15 11:59 am *** (CHG: Sta->10)
*** 07/23/15 12:02 pm ***
*** 07/23/15 12:30 pm ***
*** 07/23/15 12:47 pm ***
*** 07/23/15 12:47 pm *** (CHG: Sta->16)
*** 07/27/15 09:09 am ***
*** 07/27/15 09:09 am ***
*** 07/28/15 05:37 pm *** (CHG: Sta->10)
*** 07/28/15 05:37 pm ***
*** 09/15/15 12:53 pm *** (ADD: Impact/Symptom->RECOVERY - RECOVERY ...)
*** 09/15/15 01:31 pm *** (CHG: Sta->33)
*** 09/15/15 01:31 pm ***
*** 09/15/15 01:42 pm *** (CHG: Sta->32)
*** 09/15/15 01:42 pm ***
Key Symptoms/Summary/Rediscovery:
Customer was putting standby in read write state using steps in following
document:
How To Open Physical Standby For Read Write Testing and Flashback
(Doc ID 805438.1)
They converted back to physical standby role then later converted again to
read write state. But attempting to go back to physical standby role after
this point fails on the FLASHBACK DATABASE TO RESTORE POINT step with:
ORA-283: recovery session canceled due to errors
ORA-38861: flashback recovery stopped before reaching recovery target
ORA-16016: archived log for thread 1 sequence# 398341 unavailable
Explain why this is not a bug:
The primary had not shipped any new redo for the standby to apply while it
was previously in physical standby state. This is needed for RVWR process to
generate any flashback logs for later usage by FLASHBACK DATABASE command.
Follow all steps in Note 805438.1 including 7 thru 8 before later converting
again to read write state.
For 11g and up, recommendation is instead to use Snapshot Standby process
documented in 'Managing Physical and Snapshot Standby Databases' Chapter of
the Data Guard Concepts and Administration guide.
*** 12/11/16 06:21 pm ***