oracle 僵死的进程,Oracle Instance僵死导致不能正常恢复

Oracle

Instance僵死导致不能正常恢复

——同构System Copy异常之一

感谢我的OCP老师

【异常现象】：

svr:oraqas>

brrestore -b bejrhdub.anr -m full -d rman_util

BR0401I BRRESTORE 7.00

(18)

BR0405I Start of file restore:

rejtwnwo.rsb 2012-10-31 17.01.18

BR0564W File

/oracle/PRD/sapbackup/cntrlPRD.dbf cannot be restored/verified with

other data files using RMAN

BR0564W File

/oracle/PRD/origlogA/cntrl/cntlrPRD.dbf cannot be restored/verified

with other data files using RMAN

BR0564W File

/oracle/PRD/origlogB/cntrl/cntrlPRD.dbf cannot be restored/verified

with other data files using RMAN

BR0564W File

/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf cannot be restored/verified

with other data files using RMAN

BR0567I These files can be

restored/verified in a separate BRRESTORE run

BR0565I Due to running database

instance control file cannot be restored during restore using

RMAN

BR0566I Stop the database

instance and restore the control file separately

BR0456I Probably the database

must be recovered due to restore from online backup

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.18

BR0407I Restore of database:

PRD

BR0408I BRRESTORE action ID:

rejtwnwo

BR0409I BRRESTORE function ID:

rsb

BR0449I Restore mode:

FULL

BR0419I Files will be restored

from backup: bejrhdub.anr 2012-10-18 00.01.01

BR0416I 43 files found to

restore, total size 9922100.336 MB

BR0424I Files will not be

decompressed

BR0421I Restore device type:

rman_util

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.18

BR0256I Enter 'c[ont]' to

continue, 's[top]' to cancel BRRESTORE:

c

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.42

BR0257I Your reply:

'c'

BR0259I Program execution will

be continued...

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.42

BR0551I Starting restore from

normal database backup using RMAN...

BR0278E Command output of

'SHELL=/bin/sh /oracle/PRD/102_64/bin/rman nocatalog':

Recovery Manager: Release

10.2.0.2.0 - Production on Wed Oct 31 17:01:42 2012

Copyright (c) 1982, 2005,

Oracle.  All rights reserved.

RMAN>

RMAN> connect

target *

connected to target database

(not started)

RMAN>

**end-of-file**

RMAN>

host command

complete

RMAN>

2> 3> 4>

5> 6>

7>

RMAN-00571:

===========================================================

RMAN-00569: ===============

ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571:

===========================================================

RMAN-03002: failure of allocate

command at 10/31/2012 17:01:43

RMAN-06403:

could not obtain a fully authorized session

ORA-01034:

ORACLE not available

RMAN>

Recovery Manager

complete.

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.43

BR0279E Return code from

'SHELL=/bin/sh /oracle/PRD/102_64/bin/rman nocatalog':

1

BR0522E 0 of 43 files/save sets

processed by RMAN

BR0536E RMAN call for database

instance PRD failed

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.43

BR0553E Restore from normal

database backup using RMAN failed

BR0406I End of file restore:

rejtwnwo.rsb 2012-10-31 17.01.43

BR0280I BRRESTORE time stamp:

2012-10-31 17.01.43

BR0404I BRRESTORE terminated

with errors

【解决办法】:

(1) ps

-ef|grep ora_

可以看到ora相关进程是正常的，给人以intance是active的假象，后面会发现，其实已死

从另一角度，从上面的log可以知道：ORA-01034: ORACLE not

available(细心很重要)

(2)select

status from v$instance;  发现是not

available的，证明instance其实已死，RMAN的原理尚不清，但它与这个instance去交互，发现是死的，自然没法给你恢复，这是root

cause

(3)ps

-ef|grep ora_看到的进程清单，

SQL> !ps -ef|grep ora_

oraqas 188464  1  0

00:27:35  -

0:00 ora_mmon_PRD

oraqas 204966  1  0

00:27:35  -

0:00 ora_reco_PRD

oraqas 209036  1  0

00:27:35  -

0:00 ora_ckpt_PRD

oraqas 241904  1  0

00:27:35  -

0:00 ora_psp0_PRD

oraqas 278644  1  0

00:27:35  -

0:00 ora_dbw0_PRD

oraqas 233876  1  0

00:27:35  -

0:00 ora_mmnl_PRD

oraqas 242128  1  0

00:27:35  -

0:00 ora_cjq0_PRD

oraqas 254378  1  0

00:27:35  -

0:00 ora_smon_PRD

oraqas 262636  1  0

00:27:35  -

0:00 ora_lgwr_PRD

oraqas 266508  1  0

00:27:35  -

0:01 ora_mman_PRD

oraqas 315718  1  0

00:27:35  -

0:00 ora_pmon_PRD

逐一kill掉：

kill -9 188464

……

kill -9 315718

(4)exit

#su -

oraqas

sqlplus /nolog

connect /as sysdba

startup nomount;

alter

database mount;

select

status from v$instance;  #MOUNTED

(5)exit

brrestore -b bejrhdub.anr -m

full -d rman_util

starting............................