【问题记录】standby MRP进程挂了 报IO错误 重启MRP仍会自动停掉

standby 报错

Sun Sep 22 07:18:17 2013

KCF: write/open errorblock=0x58530 online=1

     file=69 /dev/rrac_lv16_5g

     error=27070 txt: 'IBM AIX RISCSystem/6000 Error: 5: I/O error

Additional information:1'

Sun Sep 22 07:19:40 2013

KCF: write/open errorblock=0x5852f online=1

     file=69 /dev/rrac_lv16_5g

...

 

Sun Sep 22 07:22:03 2013

Errors with log/arch/1_104730_751481221.dbf

MRP0: Background MediaRecovery terminated with error 12801        --MRP进程挂了

Sun Sep 22 07:22:04 2013

Errors in file /u01/oracle/admin/eptdb/bdump/eptdb_mrp0_667684.trc:

ORA-12801: error signaledin parallel query server P002

ORA-01579: write erroroccurred during recovery

 

 

发现问题后，尝试把MRP起起来

Sun Sep 22 09:14:01 2013

Completed: alter databaserecover managed standby database disconnect from session

Sun Sep 22 09:14:01 2013

Media Recovery Log/arch/1_104730_751481221.dbf

Media Recovery Log/arch/2_98851_751481221.dbf

 

...

 

之后又挂起

Sun Sep 22 09:24:10 2013

KCF: write/open error block=0x4b302online=1

     file=58/dev/rrac_lv09_20g

     error=27070 txt: ‘IBM AIX RISC System/6000Error: 5: I/O error

Additional information: 1’

KCF: write/open error block=0x4b303online=1

    file=58 /dev/rrac_lv09_20g

     error=27070 txt: ‘IBM AIX RISC System/6000Error: 5: I/O error

Additional information: 1’

KCF: write/open error block=0x4b304online=1

    file=58 /dev/rrac_lv09_20g

     error=27070 txt: ‘IBM AIX RISC System/6000Error: 5: I/O error

--感觉是IO的问题

...

Sun Sep 22 09:26:20 2013

Errors in file/u01/oracle/admin/eptdb/bdump/eptdb_p002_610360.trc:

ORA-01579: write erroroccurred during recovery

Sun Sep 22 09:26:20 2013

Errors in file/u01/oracle/admin/eptdb/bdump/eptdb_p001_921670.trc:

ORA-01579: write erroroccurred during recovery

Sun Sep 22 09:26:23 2013

Write error has occurred during recovery

Sun Sep 22 09:26:23 2013

Write error has occurred during recovery

Sun Sep 22 09:26:23 2013

Write error has occurred during recovery

Sun Sep 22 09:26:23 2013

Errors with log /arch/2_98852_751481221.dbf

MRP0: Background Media Recovery terminatedwith error 12801

Sun Sep 22 09:26:23 2013

Errors in file/u01/oracle/admin/eptdb/bdump/eptdb_mrp0_942194.trc:

ORA-12801: error signaled in parallel queryserver P000

ORA-01579: write error occurred duringrecovery

 

...

Sun Sep 22 09:56:51 2013

Errors in file/u01/oracle/admin/eptdb/udump/eptdb_rfs_880812.trc:

ORA-00345: redo log writeerror block 32589 count 2042

ORA-00312: online log 8thread 1: '/dev/rstb_redo1_2_1g'

ORA-27070: asyncread/write failed

IBM AIX RISC System/6000 Error: 5: I/Oerror

Additional information: 1

Sun Sep 22 09:57:59 2013

Primary database is in MAXIMUM PERFORMANCEmode

Sun Sep 22 09:57:59 2013

ARC3: Standby redo logfile selected forthread 2 sequence 98863 for destination LOG_ARCHIVE_DEST_2

Sun Sep 22 09:57:59 2013

RFS[25198]: Successfully opened standby log10: '/dev/rstb_redo1_4_1g'

Sun Sep 22 10:02:00 2013

Errors in file/u01/oracle/admin/eptdb/udump/eptdb_rfs_880812.trc:

ORA-00600: internal errorcode, arguments: [kcrrpicc.4], [], [], [], [], [], [], []

Sun Sep 22 10:02:04 2013

Redo Shipping Client Connected as PUBLIC

 

主库报错：

Sun Sep 22 10:00:08 2013

Errors in file/u01/oracle/admin/eptdb/bdump/eptdb1_lns1_1130638.trc:

ORA-00270: error creatingarchive log

LGWR: Error 270 closingarchivelog file 'standby1'

LNS: Standby redo logfile selected forthread 1 sequence 104743 for destination LOG_ARCHIVE_DEST_2

Sun Sep 22 10:03:54 2013

Errors in file/u01/oracle/admin/eptdb/bdump/eptdb1_lns1_1130638.trc:

ORA-00340: IO errorprocessing online log  of thread

Sun Sep 22 10:03:54 2013

LGWR: I/O error 340archiving log 1 to 'standby1'

 

此时LOG_ARCHIVE_DEST_2变为error的状态：

querydb1:/home/oracle>$orz dgarcdest

DEST_NAME            STATUS    DATABASE_MODE   DESTINATION

-------------------- ------------------------ --------------------

LOG_ARCHIVE_DEST_1   VALID    OPEN            /arch

LOG_ARCHIVE_DEST_2   ERROR     MOUNTED-STANDBY standby1

 

发现数据文件、归档都不能写，由于这些文件都是存储在裸设备上的，怀疑是放裸设备的卷出了问题。

将该问题向主机组反映后说是由于内存不足导致的，但是当时查过内存以及交换内存，都不是很紧张。

 

解决：

后来将操作系统重启，问题就解决了。

 

备库重新开启apply应用：

SQL> alter database recover managed standbydatabase disconnect from session;

 

SQL> SELECT PROCESS,STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM V$MANAGED_STANDBY order by 1;

PROCESS  STATUS          THREAD# SEQUENCE#     BLOCK#     BLOCKS

--------- ------------ -------------------- ---------- ----------

ARCH          CLOSING                2      98890         190465        553

ARCH          CLOSING                1     104764         190465       1778

ARCH          CLOSING                1     104766         190465        607

ARCH          CLOSING                2      98891         190465        962

ARCH          CLOSING                1     104767         190465        463

MRP0         APPLYING_LOG             1     104768               0           0

--没有RFS进程

--原因是主库的LOG_ARCHIVE_DEST_2还是error状态

 

在主库使LOG_ARCHIVE_DEST_2重新生效：

alter system set LOG_ARCHIVE_DEST_2=enable;

 

再次在备库查询进程：

SQL> SELECT PROCESS,STATUS, THREAD#, SEQUENCE#, BLOCK#, BLOCKS FROM V$MANAGED_STANDBY order by 1;

PROCESS  STATUS          THREAD# SEQUENCE#     BLOCK#     BLOCKS

--------- ------------ -------------------- ---------- ----------

ARCH          CLOSING                2      98890         190465        553

ARCH          CLOSING                1     104764         190465       1778

ARCH          CLOSING                1     104766         190465        607

ARCH          CLOSING                2      98891         190465        962

ARCH          CLOSING                1     104767         190465        463

MRP0         WAIT_FOR_LOG           1     104768               0           0

RFS     IDLE                        1     104768         142875       6312

RFS     IDLE                        0         0               0           0

RFS     IDLE                        0         0               0           0

RFS     IDLE                        0         0               0           0

RFS     IDLE                        0         0               0           0

RFS     IDLE                        0         0               0           0

RFS     IDLE                        2      98892           67433      1892

RFS     IDLE                        0         0               0           0