Oracle RAC 集群异常故障处理系列--案例（2）ORA-00600 [ktubko_1]

实例崩溃重启 ORA-00600 [ktubko_1] ORA-00600 [4506]

情况说明

业务反应出现每隔15分钟左右就会出现断联，数据库环境为三节点 RAC 12c版本。

故障描述

检查数据库告警日志发现三个节点alert间隔10到30分钟左右就会出现实例重启和ora-00600错误。报错信息如下：

2024-01-15T12:00:09.818988+08:00
ORACLE Instance xxxxdb2 (pid = 137) - Error 600 encountered while recovering transaction (1, 24) on object 427040.
2024-01-15T12:00:09.819155+08:00
Errors in file /u01/app/oracle/diag/rdbms/xxxxcdb/xxxxcdb2/trace/xxxxcdb2_smon_27712.trc:
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-15T12:00:10.172785+08:00
Dumping diagnostic data in directory=[cdmp_20240115120010], requested by (instance=2, osid=27712 (SMON)), summary=[incident=2576433].
2024-01-15T12:00:25.626083+08:00
Errors in file /u01/app/oracle/diag/rdbms/xxxxcdb/xxxxcdb2/trace/xxxxcdb2_m000_76207.trc  (incident=2579744) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [4506], [2], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xxxxcdb/xxxxcdb2/incident/incdir_2579744/sxsbcdb2_m000_76207_i2579744.trc

TRC 日志如下：

2024-01-15T12:00:10.172785+08:00
Incident 3176730 created, dump file: /u01/app/oracle/diag/rdbms/xxxxcdb/xxxxcdb2/incident/incdir_3176730/xxxxcdb2_smon_87659_i3176730.trc

2024-01-15T12:00:10.172785+08:00
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []

ORACLE Instance xxxxcdb2 (pid = 164) - Error 600 encountered while recovering transaction (1, 24) on object 427040.

2024-01-15T12:00:10.172785+08:00(CDB$ROOT(1))
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0)
----- Error Stack Dump -----
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.

----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst()+119         call     kgdsdst()            7FFC942011F8 000000002
                                                   7FFC941E2C60 ? 7FFC941E2D78 ?
                                                   000000000 000000082 ?

故障分析

通过对比mos匹配 600报错，发现相关错误说明 RAC 3个节点内的SGA大小不一致：
节点1 SGA_TARGET=329G
节点2 SGA_TARGET=329G
节点3 SGA_TARGET=419G
将SGA大小统一修改为329G后仍旧出现实例崩溃重启故障。
根据日志内报错都指向 Smon 进程在恢复事务时 object_id= 427040 的对象出现了错误。

ORACLE Instance xxxxdb2 (pid = 137) - Error 600 encountered while recovering transaction (1, 24) on object 427040.

通过隐含参数关闭smon的事务恢复进程，避免实例持续崩溃重启，以打开数据库的状态进行进一步检查。

alter system set "_smu_debug_mode"=1024;

根据日志中给出object_id 在数据库内找到相应的对象类型及对象名

select owner,object_name,object_type from dba_objects where object_id=427040;

OWNER                   OBJECT_NAME          OBJECT_TYPE
----------- --------------------------- -----------------------------------
SYS                   WRH$_LATCH_BL_PK     INDEX

发现对象是awr快照表相关索引，查出索引名称

select table_name,index_name from dba_indexes where index_name='WRH$_LATCH_BL_PK';


TABLE_NAME      INDEX_NAME
-------------- --------------------------------------------------------------------------
WRH$_LATCH_BL   WRH$_LATCH_BL_PK

解决方案

重新创建索引

alter index WRH$_LATCH_BL_PK rebuild online;

重建完成后关闭smon隐含参数

alter system reset "_smu_debug_mode";

根据查出的对象类型不同建议以下方式来解决：

1. 如果受影响的object_type是索引，删除并重新创建索引。
2. 如果object_type表/表分区受到影响，对表/表分区进行导出备份，truncate表并导入表或者在表空间中将表/表分区在线移动。
3. 如果可以承受逻辑不一致，则通过重新创建段来修复问题。

后续观察实例状态及告警日志未出现崩溃现象，问题解决。