Alert信息:
Cursor pin S wait ON X,ora-600
Sat Aug 18 05:40:02 2012
Errors in file /u01/oracle/app/oracle/admin/prdb/udump/prdb1_ora_3813910.trc:
ORA-00600: internal error code, arguments: [kkshgnc-nextchild], [], [], [], [], [], [], []
/u01/oracle/app/oracle/admin/prdb/udump/prdb1_ora_3813910.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options
ORACLE_HOME = /u01/oracle/app/oracle/product/10.2.0/db
System name: AIX
Node name: prdb1
Release: 3
Version: 5
Machine: 00C048954C00
Instance name: prdb1
Redo thread mounted by this instance: 1
Oracle process number: 136
Unix process pid: 3813910, image: oracleprdb1@prdb1
Call Stack信息:
*** 2012-08-18 05:40:02.939
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [kkshgnc-nextchild], [], [], [], [], [], [], []
No current SQL statement being executed.
ksedst <- ksedmp <- ksfdmp <- kgerinv <- kgeasnmierr
<- kkshgnc <- kksSearchChildList <- 0628 <- kksfbc <- kkspsc0
<- kksParseCursor <- opiosq0 <- opiosq <- opiodr <- rpidrus
<- skgmstack <- rpidru <- rpiswu2 <- rpidrv <- rpisplu
<- rpispl <- kqdGetCursor <- kqldcar <- kglslod <- kqlslod
<- kqlsublod <- kqllod <- kglobld <- kglobpn <- kglpim
<- kglpin <- kglgob <- kkdcloc <- kkdcacr <- qksqbMarkDPKeys
<- qksqbMarkDPKeys <- qksqbMarkDPKeys <- qksqbMarkDPKeys <- opiSem <- opiDeferredSem
<- opitca <- kksFullTypeCheck <- rpiswu2 <- kksSetBindType <- kksfbc
<- opiexe <- kpoal8 <- opiodr <- ttcpip <- opitsk
<- opiino <- opiodr <- opidrv <- sou2o <- opimai_real
<- main <- start
SO: 700000d0af3dff8, type: 4, owner: 700000d0e9910a0, flag: INIT/-/-/0x00
(session) sid: 5453 trans: 0, creator: 700000d0e9910a0, flag: (41) USR/- BSY/-/-/-/-/-
DID: 0001-0088-00001A70, short-term DID: 0001-0088-00001A71
txn branch: 0
oct: 3, prv: 0, sql: 700000c4b6373f0, psql: 700000c5168db60, user: 73/AICBS
service name: prdb
O/S info: user: aiapp, term: unknown, ospid: 1234, machine: pc-zjjacrm15.site
program: JDBC Thin Client
application name: JTC crm-app-g15-c4b7-srv4 13463@pc-zjjacrm15., hash value=1017042931
action name: JAVA, hash value=3039773399
在MOS上查找又是bug,最近遇到好几个cursor:pin s wait on x相关的bug,发现Bug:6795880。大概在Instance crash的前一天,DB 出现大量Cursor:
pin S wait on X,当时通过查看,所有的session是被一个session A堵塞,session A在等待“'kksfbc child completion”。
由于此库最近也遭受另一个bug,所以当时就把堵塞的session给kill掉了,结果一天后Instance Crash了。
看看MOS description:
A session may go into an infinite spin just after a wait
for 'kksfbc child completion'. The spin occurs with
a stack including kksSearchChildList -> kkshgnc where
kksSearchChildList loops forever.
This problem can also lead to internal error such as any of
ORA-600 [kksSearchChildList1], ORA-600 [kksSearchChildList2]
ORA-600 [kksSearchChildList3], ORA-600 [kkshgnc-nextchild]
Solution:
1,Apply Patch:8575528
2,设置参数并且reboot数据库 set "_cursor_features_enabled=10"