Instance terminated by PMON

DB Instance terminated by PMON

LOG FILE
--------------
Filename = log_20140103.xml

<msg time='2014-01-03T22:05:42.673+08:00' org_id='oracle' comp_id='rdbms'
msg_id='1251525200' type='INCIDENT_ERROR' group='Generic Internal Error'
level='1' host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11'
prob_key='ORA 600 [kcbo_dump_q]' upstream_comp='' downstream_comp='CACHE_RCV'
ecid='' errid='138960' detail_path='/opt/oracle/diag/rdbms/selaprods/SELAPRODS/trace/SELAPRODS_dbw2_32210.trc'>
<txt>Errors in file /opt/oracle/diag/rdbms/selaprods/SELAPRODS/trace/SELAPRODS_dbw2_32210.trc  (incident=138960):
ORA-00600: internal error code, arguments: [kcbo_dump_q], [2], [16385], [0], [155640944], [0], [0], [], [], [], [], []
ORA-00600: internal error code, arguments: [kcbo_switch_mq_bg_3], [16385], [155640944], [0], [], [], [], [], [], [], [], []
</txt>
</msg>
<msg time='2014-01-03T22:05:42.677+08:00' org_id='oracle' comp_id='rdbms'
msg_id='dbgexProcessError:1176:3370026720' type='TRACE' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11'>
<txt>Incident details in: /opt/oracle/diag/rdbms/selaprods/SELAPRODS/incident/incdir_138960/SELAPRODS_dbw2_32210_i138960.trc
</txt>
</msg>
<msg time='2014-01-03T22:05:42.984+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32182'>
<txt>Trace dumping is performing id=[cdmp_20140103220542]
</txt>
</msg>
<msg time='2014-01-03T22:15:43.909+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>Process 0x0x5c16d2ee0 appears to be hung while dumping
</txt>
</msg>
<msg time='2014-01-03T22:15:43.909+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>Current time = 1338869583, process death time = 1338809572 interval = 60000
</txt>
</msg>
<msg time='2014-01-03T22:15:43.910+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>Attempting to kill process 0x0x5c16d2ee0 with OS pid = 32210
</txt>
</msg>
<msg time='2014-01-03T22:15:43.910+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>OSD kill succeeded for process 0x5c16d2ee0
</txt>
</msg>
<msg time='2014-01-03T22:15:46.925+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>PMON (ospid: 32168): terminating the instance due to error 471
</txt>
</msg>
<msg time='2014-01-03T22:15:52.869+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>Termination issued to instance processes. Waiting for the processes to exit
</txt>
</msg>
<msg time='2014-01-03T22:15:58.886+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>Instance termination failed to kill one or more processes
</txt>
</msg>
<msg time='2014-01-03T22:15:58.887+08:00' org_id='oracle' comp_id='rdbms'
client_id='' type='UNKNOWN' level='16'
host_id='oracnbj011.corp.novocorp.net' host_addr='10.51.150.11' module=''
pid='32168'>
<txt>Instance terminated by PMON, pid = 32168
</txt>
</msg>


TRACE FILE
----------------
Filename = SELAPRODS_dbw2_32210_i138959.trc

VERSION
---------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /opt/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: oracnbj011.corp.novocorp.net
Release: 2.6.18-194.el5
Version: #1 SMP Tue Mar 16 21:52:39 EDT 2010
Machine: x86_64
Instance name: SELAPRODS
Redo thread mounted by this instance: 1
Oracle process number: 12
Unix process pid: 32210, image: oracle@oracnbj011.corp.novocorp.net (DBW2)


ORA-00600: internal error code, arguments: [kcbo_switch_mq_bg_3], [16385], [155640944], [0], [], [], [], [], [], [], [], []

========= Dump for incident 138959 (ORA 600 [kcbo_switch_mq_bg_3]) ========

*** 2014-01-03 22:05:36.291
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.

----- Call Stack Trace -----
skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp
<- ksfdmp <- dbgexPhaseII <- dbgexProcessError <- dbgeExecuteForError <- dbgePostErrorKGE
<- 1615 <- dbkePostKGE_kgsf <- kgeadse <- kgerinv_internal <- kgerinv
<- kgeasnmierr <- kcbo_switch_mq_bg1 <- kcbkubc <- kcbbic1 <- kcbbiop
<- kcbbdrv <- ksbabs <- ksbrdp <- opirip <- opidrv
<- sou2o <- opimai_real <- ssthrdmain <- main <- libc_start_main

<- start

On : 11.2.0.1 version, RDBMS The instance crashed with  the following error occurs. ERROR ----------------------- ORA-00600: internal error code, arguments: [kcbo_dump_q], [2], [16385], [0], [155640944], [0], [0], [], [], [], [], [] ORA-00600: internal error code, arguments: [kcbo_switch_mq_bg_3], [16385], [155640944], [0], [], [], [], [], [], [], [], [] STEPS ----------------------- The issue can not be reproduced at will. BUSINESS IMPACT ----------------------- The issue has the following business impact: Due to this issue, the instance crashed.

Dear Customer, This issue is caused by an array overrun in the Oracle code not being handled properly. Patch 1021445 can fix this problem. Unfortunately, the patch is not available on your platform. We highly recommend to upgrade to 11.2.0.3 or higher. Regards, William Chen

 

Oracle 技术支持

- 早于 5 小时前

[ODM Proposed Solution Justif]

1. The issue was fixed in 11.2.0.3. 2. patch 10214450 is not available on this linux x86-64 platform. 3. There is no workaround can bypass this issue.

最终原因还是由于oracle 11.2.01版本的BUG引起的，临时解决方法就是把crash掉的DB再起来，

根本解决方法就是升级DB版本到11.2.03

Dear Customer, This issue is caused by an array overrun in the Oracle code not being handled properly. Patch 1021445 can fix this problem. Unfortunately, the patch is not available on your platform. We highly recommend to upgrade to 11.2.0.3 or higher. Regards, William Chen

 

Oracle 技术支持

- 早于 5 小时前

[ODM Proposed Solution Justif]

1. The issue was fixed in 11.2.0.3. 2. patch 10214450 is not available on this linux x86-64 platform. 3. There is no workaround can bypass this issue.