Oracle 一次ORA-00600[13013]

起初后台报错:
昨晚9点至早上10点:

ORA-00600: , : [13013], [5001], [126801], [109382409], [101], [109382437], [17], [], [], [], [], []
Incident details in: /opt/app/oracle/diag/rdbms/jcd/jcd/incident/incdir_99948/jcd_ora_23550_i99948.trc
Non critical error ORA-48113 caught while writing to trace file "/opt/app/oracle/diag/rdbms/jcd/jcd/incident/incdir_99948/jcd_ora_23550_i99948.trc"
ORA-00600: , : [13013], [5001], [126801], [109382408], [102], [109382433], [17], [], [], [], [], []
Incident details in: /opt/app/oracle/diag/rdbms/jcd/jcd/incident/incdir_99980/jcd_ora_445_i99980.trc
ORA-00600: , : [13013], [5001], [126801], [109382409], [123], [109382437], [17], [], [], [], [], []
Incident details in: /opt/app/oracle/diag/rdbms/jcd/jcd/incident/incdir_101652/jcd_ora_8586_i101652.trc
ORA-00600: , : [13013], [5001], [126801], [109382409], [168], [109382437], [17], [], [], [], [], []
ORA-00600: , : [13013], [5001], [126801], [109382406], [99], [109382433], [17], [], [], [], [], []
ORA-00600: , : [13013], [5001], [126801], [109382405], [152], [109382433], [17], [], [], [], [], []

该600错误号可以拆分:
ORA-600 [13013]
Format: ORA-600 [13013] [a] [b] [c] [d] [e] [f]
This format relates to Oracle Server 8.0.3 and above
    Arg [a] Passcount
    Arg [b] Data Object number
    Arg [c] Tablespace Decimal Relative DBA (RDBA) of block containing the row to be updated
    Arg [d] Row Slot number
    Arg [e] Decimal RDBA of block being updated (Typically same as [c])
    Arg [f] Code

ORA-00600: internal error code, arguments: [13013], [5001], [268], [8457615], [5], [8457615], [17], [], [], [], [], []

根据报错信息确定相对文件号与相对块号:

select dbms_utility.data_block_address_file(109382409) rfile,
    dbms_utility.data_block_address_block(109382409) rblocks
     from dual;
     RFILE    RBLOCKS
---------- ----------
        26     330505

select dbms_utility.data_block_address_file(109382408) rfile,
    dbms_utility.data_block_address_block(109382408) rblocks
     from dual;       

     RFILE    RBLOCKS
---------- ----------
        26     330504

select dbms_utility.data_block_address_file(109382406) rfile,
    dbms_utility.data_block_address_block(109382406) rblocks
     from dual;      
     RFILE    RBLOCKS
---------- ----------
        26     330502

select dbms_utility.data_block_address_file(109382405) rfile,
    dbms_utility.data_block_address_block(109382405) rblocks
     from dual;          

   RFILE    RBLOCKS
---------- ----------
        26     330501

确认对象:

SQL> set linesize 999 pagesize 9999
SQL> select owner,segment_name,segment_type,tablespace_name from dba_extents 
    where file_id =26
    and 330505 between block_id and block_id + blocks -1;        

OWNER      SEGMENT_NAME    SEGMENT_TYPE TABLESPACE_NAME
---------- ----------    ----------   --------------------
XHSD_JCD   JCD_PUB_YHBMB    TABLE      TS_PUB_DATA

同时dbv校验26号文件:
无物理坏块!

尝试执行分析:

analyze table XHSD_JCD.JCD_PUB_YHBMB validate structure cascade online;
ERROR at line 1:
ORA-08104: this index object 130036 is being online built or rebuilt

确认130036索引:

OBJECT_NAME   OBJECT_TYPE
---------------------------------------------------------
PK_JCD_PUB_YHBMB   INDEX

//与客户沟通,此时正在重建索引.无法进行分析.

客户重建索引后仍无法继续分析:（经反馈其创建索引会话被杀掉.）

analyze table XHSD_JCD.JCD_PUB_YHBMB validate structure cascade online;
ERROR at line 1:
ORA-08104: this index object 130036 is being online built or rebuilt

执行清理索引创建操作:
SQL> DECLARE
    isClean BOOLEAN;
    BEGIN
    isClean := DBMS_REPAIR.ONLINE_INDEX_CLEAN(130036);
    END;
    /

再次手动进行rebulid:
alter index XHSD_JCD.PK_JCD_PUB_YHBMB rebuild

再次进行分析资源被占用,对象被hang住sql占用:
analyze table XHSD_JCD.JCD_PUB_YHBMB validate structure cascade online;
analyze table XHSD_JCD.JCD_PUB_YHBMB validate structure
ANALYZE INDEX  XHSD_JCD.PK_JCD_PUB_YHBMB VALIDATE STRUCTURE;


确定锁定对象会话:
 select l.session_id,o.owner,o.object_name 
 from v$locked_object l,dba_objects o where l.object_id=o.object_id;   
SESSION_ID OWNER      OBJECT_NAME
---------- ---------- --------------------
       780 XHSD_JCD   JCD_PUB_YHBMB
       627 XHSD_JCD   JCD_PUB_YHBMB
      1152 XHSD_JCD   JCDMS_DZTSYCCL


操作系统层面杀锁:
select a.spid,b.sid,b.serial#,b.username from v$process a,v$session b where a.addr=b.paddr and b.sid=780;
select a.spid,b.sid,b.serial#,b.username from v$process a,v$session b where a.addr=b.paddr and b.sid=627;
select a.spid,b.sid,b.serial#,b.username from v$process a,v$session b where a.addr=b.paddr and b.sid=1152;

kill -9进行操作系统层面杀锁

再次执行分析成功:
analyze table XHSD_JCD.JCD_PUB_YHBMB validate structure cascade online;
analyze table XHSD_JCD.JCD_PUB_YHBMB validate structure
ANALYZE INDEX  XHSD_JCD.PK_JCD_PUB_YHBMB VALIDATE STRUCTURE;

此时客户给出问题SQL,执行SQL 会话hang住

UPDATE XHSD_JCD.PUB_YHBMB SET DLBZ = '1' WHERE DLH='00560';

后台出现:
ORA-07445: exception encountered: core dump [kduudm()+1798] [SIGSEGV] [ADDR:0x1E1000000] [PC:0x4BCB2A2] [Address not mapped to object] []
//对于该错误代码,Mos并无相关记录.
--------------
Wed Mar 03 11:49:46 2021
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x1E1000000] [PC:0x4BCB2A2, kduudm()+1798] [flags: 0x0, count: 1]
Errors in file /opt/app/oracle/diag/rdbms/jcd/jcd/trace/jcd_ora_9960.trc  (incident=100772):
ORA-07445: exception encountered: core dump [kduudm()+1798] [SIGSEGV] [ADDR:0x1E1000000] [PC:0x4BCB2A2] [Address not mapped to object] []
Incident details in: /opt/app/oracle/diag/rdbms/jcd/jcd/incident/incdir_100772/jcd_ora_9960_i100772.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Mar 03 11:49:48 2021
Dumping diagnostic data in directory=[cdmp_20210303114948], requested by (instance=1, osid=9960), summary=[incident=100772].
Wed Mar 03 11:49:49 2021
Sweep [inc][100772]: completed
Sweep [inc2][100772]: completed
--------------

询问客户视图创建SQL:

create or replace view pub_yhbmb as
select DLH,BM,YHM,KL,SYBZ,QYRQ,ZZRQ,DLBZ,SM
    from jcd_PUB_YHBMB;

对原表执行delete也hang住: 
delete from  jcd_pub_yhbmb  WHERE DLH='00560';

select语句数据获取正常:
select * from XHSD_JCD.jcd_pub_yhbmb

尝试跟踪delete以及update语句,但sql_trace以及10046无输出.

最终尝试手动通过CTS创建一张测试表进行DML测试可以执行:
create table XHSD_JCD.test as select * from XHSD_JCD.jcd_PUB_YHBMB;
delete from  XHSD_JCD.test  WHERE DLH='00560';

最终使用CTS重建业务表业务正常.

记录排查过程.

相关Mos:
How to Use DBMS_REPAIR.ONLINE_INDEX_CLEAN For The Cleanup Of Online Index Failing With ORA-08104?(文档 ID 1378173.1)
How to Cleanup and Rebuild an Interrupted Online Index Rebuild - ORA-8104 , ORA-8106 (文档 ID 272735.1)
Session Was Killed During The Rebuild Of Index ORA-08104 (文档 ID 375856.1)