oracle asm ora 15041,ORA-15041，ORA-01274 故障解决实例

上周五去客户处巡检时，发现原来配置的DG备库未与主库同步，特此记录一下

在5月14日13：58出现故障，3398日志无法应用，以下是alert日志

Thu May 14 13:58:01 2015

Errors in file /u01/app/Oracle/diag/rdbms/tcdg/tc/trace/tc_pr00_3473760.trc:

ORA-01119: error in creating database file '+data'

ORA-17502: ksfdcre:4 Failed to create file +data

ORA-15041: diskgroup "DATA" space exhausted

File #108 added to control file as 'UNNAMED00108'.

Originally created as:

'+DATA/tc/datafile/oa2015.387.879688649'

Recovery was unable to create the file as:

'+data'

MRP0: Background Media Recovery terminated with error 1274

Errors in file /u01/app/oracle/diag/rdbms/tcdg/tc/trace/tc_pr00_3473760.trc:

ORA-01274: cannot add datafile '+DATA/tc/datafile/oa2015.387.879688649' - file could not be created

Managed Standby Recovery not using Real Time Apply

Recovery interrupted!

Recovered data files to a consistent state at change 10293976250522

Thu May 14 13:58:06 2015

MRP0: Background Media Recovery process shutdown (tc)

由于空间不足，出现了ORA-15041错误，然后MRP0进程被终止

再来看trace文件内容：

*** 2015-05-14 13:58:01.562

----- START Event Driven Actions Dump ----

---- END Event Driven Actions Dump ----

----- START DDE Actions Dump -----

Executing SYNC actions

Executing ASYNC actions

----- END DDE Actions Dump (total 0 csec) -----

ORA-01119: error in creating database file '+data'

ORA-17502: ksfdcre:4 Failed to create file +data

ORA-15041: diskgroup "DATA" space exhausted

File #108 added to control file as 'UNNAMED00108'.

Originally created as:

'+DATA/tc/datafile/oa2015.387.879688649'

Recovery was unable to create the file as:

'+data'

*** 2015-05-14 13:58:01.598 4329 krsh.c

MRP0: Background Media Recovery terminated with error 1274

ORA-01274: cannot add datafile '+DATA/tc/datafile/oa2015.387.879688649' - file could not be created

*** 2015-05-14 13:58:01.613 4329 krsh.c

Managed Standby Recovery not using Real Time Apply

*** 2015-05-14 13:58:01.616

MRP: Prodding archiver at standby for thread 1 seq 3398

----- Redo read statistics for thread 1 -----

Read rate (ASYNC): 1631875714Kb in 8455733.63s => 0.19 Mb/sec

Total redo bytes: 1631903362Kb Longest record: 53Kb, moves: 546631/3909060988 moved: 2869Mb (0%)

Longest LWN: 61559Kb, reads: 53311640

Last redo scn: 0x095c.c0cc389a (10293976250522)

Change vector header moves = 505014709/3028421495 (1%)

----------------------------------------------

*** 2015-05-14 13:58:01.626

Media Recovery drop redo thread 1

KCBR: Redo cache copies/changes = 7576594/7576586

Wait to push change maps to slaves = 136500s

*** 2015-05-14 13:58:02.545

Completed Media Recovery

Checking to start in-flux buffer recovery from SCN 2396.3234383719 to SCN  (non-inclusive) 2396.3234609306

Influx recovery found in-flux buffers

*** 2015-05-14 13:58:02.622

Influx Media Recovery add redo thread 1

Managed Standby Recovery: Standby online log for thr 1  seq 3397 not found. Looking whether archived..

Looking for archived log thr 1 seq 3397 recovery branch id 869789191 that contains SCN 10293976024935

Checking whether scn 10293976024935 in los 10293976024935 and nxs 10293976125751

Managed Standby Recovery: Opening archived log /oraarch/ARC_CRM5_3397_869789191_1.log during invocation of recoverable recovery

*** 2015-05-14 13:58:03.937

Resized overflow buffer to 2435K (for 2435K LWN)

Resized overflow buffer to 3923K (for 3923K LWN)

Resized overflow buffer to 14242K (for 14242K LWN)

*** 2015-05-14 13:58:06.105

MRP: Prodding archiver at standby for thread 1 seq 3398

Managed Recovery: Not Active posted.

由于redo apply的进程被终止，因此应用到3397归档后，之后的日志都未被应用，但是归档日志都已经顺利传到了备库

SQL> select thread#,sequence#，applied from v$archived_log where sequence#>3391;

THREAD#  SEQUENCE# APPLIED

---------- ---------- ---------

1      3392 YES

1      3393 YES

1      3394 YES

1      3395 YES

1      3396 YES

1      3397 YES

1      3398 NO

1      3399 NO

1      3400 NO

1      3401 NO

1      3402 NO

1      3403 NO

1      3404 NO

1      3405 NO

1      3406 NO

1      3407 NO

1      3408 NO

1      3409 NO

1      3410 NO

1      3411 NO

1      3412 NO

1      3413 NO

还可以通过下面的SQL语句去分析备库落后了多少归档日志未完成应用。

SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied",

2  (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"

FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN

(SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,

(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN

(SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL

WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;  3    4    5    6    7

Thread Last Sequence Received Last Sequence Applied Difference

---------- ---------------------- --------------------- ----------

1                  3531                  3397        134

此时归档应用停在了3398上面，加上之后的，共有134个归档没应用

这个时间点在主库执行的操作是添加数据文件：

Thu May 14 13:57:28 2015

ALTER TABLESPACE oa2015

ADD DATAFILE '+DATA'

SIZE 16G

AUTOEXTEND OFF

Thu May 14 13:58:05 2015

Completed: ALTER TABLESPACE oa2015

ADD DATAFILE '+DATA'

SIZE 16G

AUTOEXTEND OFF

备库磁盘空间确实不足，主库要求创建的数据文件为2个16G的文件，但是备库磁盘组可用空间只剩3G左右了

ASMCMD> lsdg

State    Type    Rebal  Sector  Block      AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N        512  4096  1048576  1638400    3265                0            3265              0            N  DATA/

和客户沟通了下，他们打算把DG停掉，暂时不用了，因此也没有做进一步处理，如果不想让归档日志传递到备库，可以设置主库参数log_archived_dest_state_2=defer。如果要解决故障，就要先添加磁盘到磁盘组，然后重新启用MRP进程，应用归档日志。