ORA-600(4000)

oracle 11gR2 启动报错：
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Process ID: 21644
Session ID: 1426 Serial number: 3

++++alter信息如下：
Fri Dec 27 14:28:59 2013
SMON: enabling cache recovery
Errors in file /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc  (incident=112161):
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/incident/incdir_112161/coredb_ora_21644_i112161.trc
Fri Dec 27 14:29:06 2013
Errors in file /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Errors in file /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 21644): terminating the instance due to error 704
Instance terminated by USER, pid = 21644
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (21644) as a result of ORA-1092
Fri Dec 27 14:29:08 2013
ORA-1092 : opitsk aborting process

++++dbv检测：
[oracle@dg_136 ~]$ dbv file=/tol/oradata/datafile/coredb/system01.dbf
DBVERIFY: Release 11.2.0.1.0 - Production on Fri Dec 27 14:58:21 2013
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
DBVERIFY - Verification starting : FILE = /tol/oradata/datafile/coredb/system01.dbf
DBVERIFY - Verification complete

Total Pages Examined         : 264960
Total Pages Processed (Data) : 218039
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 22126
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 6165
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 18630
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 496559453 (2.496559453)

+++++查看coredb_ora_21644.trc
** ACTION NAME:() 2013-12-27 14:28:41.140
 
 parallel recovery setup failed: using serial mode
Thread 1 checkpoint: logseq 46403, block 2, scn 9086475510
  cache-low rba: logseq 46403, block 40010
    on-disk rba: logseq 46403, block 43232, scn 9086494047
  start recovery at logseq 46403, block 40010, scn 0

*** 2013-12-27 14:28:41.327
Started writing zeroblks thread 1 seq 46403 blocks 43232-43239

*** 2013-12-27 14:28:41.345
Completed writing zeroblks thread 1 seq 46403

*** 2013-12-27 14:28:41.360
==== Redo read statistics for thread 1 ====
Total physical reads (from disk and memory): 4096Kb
-- Redo read_disk statistics --
Read rate (ASYNC): 1611Kb in 0.22s => 7.15 Mb/sec
Longest record: 2Kb, moves: 0/6706 (0%)
Longest LWN: 109Kb, moves: 0/67 (0%), moved: 0Mb
Last redo scn: 0x0002.1d98e55e (9086494046)
----------------------------------------------
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 2
Average hash chain = 1833/1831 = 1.0
Max compares per lookup = 1
Avg compares per lookup = 9869/11700 = 0.8
----------------------------------------------

*** 2013-12-27 14:28:41.400
KCRA: start recovery claims for 1833 data blocks

*** 2013-12-27 14:28:42.430
KCRA: blocks processed = 1833/1833, claimed = 1833, eliminated = 0

*** 2013-12-27 14:28:42.448
Recovery of Online Redo Log: Thread 1 Group 5 Seq 46403 Reading mem 0

*** 2013-12-27 14:28:42.556
Completed redo application of 1.29MB

*** 2013-12-27 14:28:55.112
Completed recovery checkpoint
IR RIA: redo_size 1649664 bytes, time_taken 1390 ms
----- Recovery Hash Table Statistics ---------
Hash table buckets = 32768
Longest hash chain = 2
Average hash chain = 1833/1831 = 1.0
Max compares per lookup = 2
Avg compares per lookup = 11704/11700 = 1.0
----------------------------------------------
Recovery sets nab of thread 1 seq 46403 to 43232 with 8 zeroblks

*** 2013-12-27 14:29:00.278
Incident 112161 created, dump file: /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/incident/incdir_112161/coredb_ora_21644_i112161.trc
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []

ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []

*** 2013-12-27 14:29:06.871
USER (ospid: 21644): terminating the instance due to error 704

++++coredb_ora_21644_i112161.trc内容如下：
Dump continued from file: /opt/app/oracle/admin/coredb/diag/rdbms/dg_111/coredb/trace/coredb_ora_21644.trc
ORA-00600: internal error code, arguments: [4000], [1], [], [], [], [], [], [], [], [], [], []

========= Dump for incident 112161 (ORA 600 [4000]) ========

*** 2013-12-27 14:29:00.292
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=6apq2rjyxmxpj) -----
select line#, sql_text from bootstrap$ where obj# != :1

----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------

*** 2013-12-27 14:29:00.423
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFFFDDEA2C8 ? 000000001 ?
                                                   7FFFFDDEE7C8 ? 000000000 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?

。。。。
============
Plan Table
============
---------------------------------------+-----------------------------------+
| Id  | Operation          | Name      | Rows  | Bytes | Cost  | Time      |
---------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT   |           |       |       |       |           |
| 1   |  TABLE ACCESS FULL | BOOTSTRAP$|       |       |       |           |
---------------------------------------+-----------------------------------+

Content of other_xml column
===========================
  db_version     : 11.2.0.1
  parse_schema   : SYS
  plan_hash      : 867914364
  plan_hash_2    : 382997521
  Outline Data:
  /*+
    BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.1')
      DB_VERSION('11.2.0.1')
      RBO_OUTLINE
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "BOOTSTRAP$"@"SEL$1")
    END_OUTLINE_DATA
  */
 。。。。。。
 PINNED BUFFER HISTORY (oldest pin first)
---------------------
BH (0xa6faef68) file#: 1 rdba: 0x00400208 (1/520) class: 4 ba: 0xa67b0000
  set: 31 pool 3 bsz: 8192 bsi: 0 sflg: 2 pwc: 0,0
  dbwrid: 0 obj: 59 objn: -1 tsn: 0 afn: 1 hint: f
  hash: [0xd69fee08,0xd69fee08] lru: [0xa6faedf0,0xa6faf180]
  ckptq: [NULL] fileq: [NULL] objq: [0xc8e5b670,0xc8e5b670]
  st: XCURRENT md: NULL tch: 1
  flags:
  LRBA: [0x0.0.0] LSCN: [0x0.0] HSCN: [0xffff.ffffffff] HSUB: [65535]
  cr pin refcnt: 0 sh pin refcnt: 0
  buffer tsn: 0 rdba: 0x00400208 (1/520)
  scn: 0x0002.1d98d857 seq: 0x01 flg: 0x04 tail: 0xd8571001
  frmt: 0x02 chkval: 0x7485 type: 0x10=DATA SEGMENT HEADER - UNLIMITED
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x00000000A67B0000 to 0x00000000A67B2000
0A67B0000 0000A210 00400208 1D98D857 04010002  [......@.W.......]
0A67B0010 00007485 00000000 00000000 00000000  [.t..............]
0A67B0020 00000000 00000001 00000007 00001020  [............ ...]
0A67B0030 00000000 00000003 00000007 0040020C  [..............@.]
0A67B0040 00000000 00000000 00000001 00000003  [................]
0A67B0050 00000001 007F8D23 00000001 00000001  [....#...........]
0A67B0060 00000000 0000003B 40000000 00400209  [....;......@..@.]
0A67B0070 00000007 00000000 00000000 00000000  [................]
0A67B0080 00000000 00000000 00000000 00000000  [................]
        Repeat 250 times
0A67B1030 00000000 00010000 00010001 00000001  [................]
0A67B1040 00000000 00000003 0040020B 0040020B  [..........@...@.]
0A67B1050 00000000 00000000 00000000 00000000  [................]
        Repeat 249 times
0A67B1FF0 00000000 00000000 00000000 D8571001  [..............W.]
  Extent Control Header
  -----------------------------------------------------------------
  Extent Header:: spare1: 0      spare2: 0      #extents: 1      #blocks: 7
                  last map  0x00000000  #maps: 0      offset: 4128
      Highwater::  0x0040020c  ext#: 0      blk#: 3      ext size: 7
  #blocks in seg. hdr's freelists: 1
  #blocks below: 3
  mapblk  0x00000000  offset: 0
      Disk Lock:: Locked by xid:  0x0001.000.007f8d23
     Map Header:: next  0x00000000  #extents: 1    obj#: 59     flag: 0x40000000
  Extent Map
  -----------------------------------------------------------------
   0x00400209  length: 7

  nfl = 1, nfb = 1 typ = 1 nxf = 0 ccnt = 3
  SEG LST:: flg: USED   lhd: 0x0040020b ltl: 0x0040020b

The buffer with tsn: 0 rdba: 0x00400208 has already been dumped

END OF PINNED BUFFER HISTORY

++++仔细观察上面trace文件的内容，有用的信息如下
select line#, sql_text from bootstrap$ where obj# != :1
rdba: 0x00400208 (1/520)
Disk Lock:: Locked by xid:  0x0001.000.007f8d23
obj#: 59

正常来说，bootstrap$ 的内容是固定不变的，不会在这个对象上出现事务

++++那我们查一下file 1 block 520 这个块
BBED> set file 1 block 520
        FILE#           1
        BLOCK#          520


BBED> p locker_ktech
struct locker_ktech, 8 bytes                @80      
   ub2 kxidusn                              @80       0x0001
   ub2 kxidslt                              @82       0x0000
   ub4 kxidsqn                              @84       0x007f8d23


BBED> p ktech.flag_ktech
ub4 flag_ktech                              @88       0x00000001 (NONE)

+++发现这个块确实有事务，跟xid对应的上
+++接下来我们手工把这个锁清除了看看
BBED> m /x 00 offset 88
 File: /tol/oradata/datafile/coredb/system01.dbf (1)
 Block: 520              Offsets:   88 to  599           Dba:0x00400208
------------------------------------------------------------------------
 00000000 01000000 00000000 3b000000 00000040 09024000 07000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 
 BBED> sum apply;
Check value for File 1, Block 520:
current = 0x7484, required = 0x7484

BBED> p offset 88
ktech.flag_ktech
----------------
ub4 flag_ktech                              @88       0x00000000 (NONE)

++++再次open，已经可以正常启动
SQL> alter database open;

Database altered.