sqlerrm切mysql_惜分飞 - 提供7*24专业数据库(Oracle,SQL Server,MySQL等)恢复和Oracle技术服务@Tel:+86 13429648788...

有朋友asm运行一段时间asm实例会报错导致数据库实例异常

Wed Dec 23 08:31:55 2020

Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_asmb_6729.trc:

ORA-00600: internal error code, arguments: [kffmLoad_1], [4365], [1], [], [], [], [], []

Wed Dec 23 08:31:55 2020

Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_asmb_6729.trc:

ORA-00600: internal error code, arguments: [kffmLoad_1], [4365], [1], [], [], [], [], []

Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_asmb_29743.trc:

ORA-00600: internal error code, arguments: [kffmLoad_1], [670], [1], [], [], [], [], []

Wed Dec 23 09:10:22 2020

Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_asmb_29743.trc:

ORA-00600: internal error code, arguments: [kffmLoad_1], [670], [1], [], [], [], [], []

Wed Dec 23 09:10:22 2020

Wed Dec 23 10:18:33 2020

Errors in file /u01/app/oracle/admin/+ASM/udump/+asm1_ora_25890.trc:

ORA-00600: internal error code, arguments: [kffmVerify_4], [0], [0], [887], [1005986561], [1352], [1], [0]

对应的trace文件

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

and Real Application Testing options

ORACLE_HOME = /u01/app/oracle/product/10.2.0/db

System name:Linux

Node name:shb01

Release:2.6.18-348.el5

Version:#1 SMP Wed Nov 28 21:22:00 EST 2012

Machine:x86_64

Instance name: +ASM1

Redo thread mounted by this instance: 0

Oracle process number: 29

Unix process pid: 26337, image: oracle@xff01 (TNS V1-V3)

*** ACTION NAME:() 2020-12-22 19:03:41.272

*** MODULE NAME:(sp_ocap@xff01 (TNS V1-V3)) 2020-12-22 19:03:41.272

*** SERVICE NAME:() 2020-12-22 19:03:41.272

*** SESSION ID:(143.1) 2020-12-22 19:03:41.272

*** 2020-12-22 19:03:41.272

ksedmp: internal or fatal error

ORA-00600: internal error code, arguments: [kffmVerify_4], [0], [0], [1657], [1005987045], [152], [1], [0]

Current SQL statement for this session:

DECLARE

fileType varchar2(16);

fileName varchar2(1024);

blkSz number;

fileSz number;

hdl number;

plksz number;

BEGIN

fileName := '+DATA4/xifenfei/onlinelog/group_6.1657.1005987045';

BEGIN

dbms_diskgroup.getfileattr(fileName,fileType,fileSz, blkSz);

dbms_diskgroup.open(fileName,'r',fileType,blkSz,hdl,plkSz,fileSz);

EXCEPTION

WHEN OTHERS then

:rc := SQLCODE;

:err_msg := SQLERRM;

return;

END;

:handle := hdl;

:bsz := blkSz;

:bcnt := fileSz;

:rc := 0;

END;

----- PL/SQL Call Stack -----

object line object

handle number name

0x15ce59360 96 package body SYS.X$DBMS_DISKGROUP

0x15cd88568 12 anonymous block

----- Call Stack Trace -----

calling call entry argument values in hex

location type point (? means dubious value)

-------------------- -------- -------------------- ----------------------------

ksedst()+31 call ksedst1() 000000000 ? 000000001 ?

7FFFBDFC3450 ? 7FFFBDFC34B0 ?

7FFFBDFC33F0 ? 000000000 ?

ksedmp()+610 call ksedst() 000000000 ? 000000001 ?

7FFFBDFC3450 ? 7FFFBDFC34B0 ?

7FFFBDFC33F0 ? 000000000 ?

ksfdmp()+21 call ksedmp() 000000003 ? 000000001 ?

7FFFBDFC3450 ? 7FFFBDFC34B0 ?

7FFFBDFC33F0 ? 000000000 ?

kgerinv()+161 call ksfdmp() 000000003 ? 000000001 ?

7FFFBDFC3450 ? 7FFFBDFC34B0 ?

7FFFBDFC33F0 ? 000000000 ?

kgeasnmierr()+163 call kgerinv() 0068996E0 ? 009AA2670 ?

7FFFBDFC34B0 ? 7FFFBDFC33F0 ?

000000000 ? 000000000 ?

kffmVerify()+379 call kgeasnmierr() 0068996E0 ? 009AA2670 ?

7FFFBDFC34B0 ? 7FFFBDFC33F0 ?

000000000 ? 000000000 ?

kfioIdentify()+1276 call kffmVerify() 000000000 ? 00000000D ?

000000001 ?

927B814400000004 ?

3BF624E500000679 ?

000000000 ?

ksfd_osmopn()+1138 call kfioIdentify() 7FFFBDFC4820 ? 15DB873F4 ?

15DB87556 ? 000000200 ?

7FFF00000003 ? 15DB873C8 ?

ksfdopn()+1014 call ksfd_osmopn() 7FFFBDFC4820 ? 00000002D ?

000000200 ? 000000003 ?

2B3800020000 ? 15F3031F0 ?

kfpkgDGOpenFile()+2 call ksfdopn() 7FFFBDFC4820 ? 00000002D ?

301 000000200 ? 000000003 ?

000020000 ? 15F3031F0 ?

pevm_icd_call_commo call kfpkgDGOpenFile() 2B383F459FA8 ? 00000002D ?

n()+1003 2B383F439070 ? 000000003 ?

000020000 ? 15F3031F0 ?

pfrinstr_ICAL()+228 call pevm_icd_call_commo 7FFFBDFC5700 ? 000000000 ?

n() 000000001 ? 000000001 ?

000000007 ? 7FFF00000000 ?

pfrrun_no_tool()+65 call pfrinstr_ICAL() 2B383F459FA8 ? 005DBD8AA ?

2B383F45A010 ? 000000001 ?

000000007 ? 7FFF00000000 ?

pfrrun()+906 call pfrrun_no_tool() 2B383F459FA8 ? 005DBD8AA ?

2B383F45A010 ? 000000001 ?

000000007 ? 7FFF00000000 ?

plsql_run()+841 call pfrrun() 2B383F459FA8 ? 000000000 ?

2B383F45A010 ? 7FFFBDFC5700 ?

000000007 ? 15CD77BD6 ?

peicnt()+298 call plsql_run() 2B383F459FA8 ? 000000001 ?

000000000 ? 7FFFBDFC5700 ?

000000007 ? 900000000 ?

kkxexe()+503 call peicnt() 7FFFBDFC5700 ? 2B383F459FA8 ?

2B383F438830 ? 7FFFBDFC5700 ?

2B383F4367D8 ? 900000000 ?

opiexe()+4691 call kkxexe() 2B383F4561D8 ? 2B383F459FA8 ?

2B383F438830 ? 15C160BD8 ?

0040D677F ? 900000000 ?

kpoal8()+2273 call opiexe() 000000049 ? 000000003 ?

7FFFBDFC6950 ? 000000001 ?

0040D677F ? 900000000 ?

opiodr()+984 call kpoal8() 00000005E ? 000000017 ?

7FFFBDFC9830 ? 000000001 ?

000000001 ? 900000000 ?

ttcpip()+1012 call opiodr() 00000005E ? 000000017 ?

7FFFBDFC9830 ? 000000000 ?

0059C35D0 ? 900000000 ?

opitsk()+1322 call ttcpip() 0068A13B0 ? 7FFFBDFC75A0 ?

7FFFBDFC9830 ? 000000000 ?

7FFFBDFC9328 ? 7FFFBDFC9998 ?

opiino()+1026 call opitsk() 000000003 ? 000000000 ?

7FFFBDFC9830 ? 000000001 ?

000000000 ? 4E6111C00000001 ?

opiodr()+984 call opiino() 00000003C ? 000000004 ?

7FFFBDFCA9F8 ? 000000001 ?

000000000 ? 4E6111C00000001 ?

opidrv()+547 call opiodr() 00000003C ? 000000004 ?

7FFFBDFCA9F8 ? 000000000 ?

0059C3080 ? 4E6111C00000001 ?

sou2o()+114 call opidrv() 00000003C ? 000000004 ?

7FFFBDFCA9F8 ? 000000000 ?

0059C3080 ? 4E6111C00000001 ?

opimai_real()+163 call sou2o() 7FFFBDFCA9D0 ? 00000003C ?

000000004 ? 7FFFBDFCA9F8 ?

0059C3080 ? 4E6111C00000001 ?

main()+116 call opimai_real() 000000002 ? 7FFFBDFCAA60 ?

000000004 ? 7FFFBDFCA9F8 ?

0059C3080 ? 4E6111C00000001 ?

__libc_start_main() call main() 000000002 ? 7FFFBDFCAA60 ?

+244 000000004 ? 7FFFBDFCA9F8 ?

0059C3080 ? 4E6111C00000001 ?

_start()+41 call __libc_start_main() 0007230B8 ? 000000002 ?

7FFFBDFCABB8 ? 000000000 ?

0059C3080 ? 000000002 ?

--------------------- Binary Stack Dump ---------------------

结合mos信息ORA-600[KFFMVERIFY_4] OR ORA-600 [kffmLoad_1], [131635] REPORTED ON THE ASMINSTANCE (Doc ID 794103.1)的描述,由于多个进程/现场使用dbms_diskgroup访问不同磁盘组之时可能触发

BUG:6377738 – ASMB ORA-00600 [KFFMVERIFY_4]

BUG:8328467 – ASM CRASHED WITH ORA-600[KFFMVERIFY_4] OR [KFFMVERIFY_4] AND [KFFMLOAD_1]

从而导致asm实例crash,引起数据库异常.结合客户这边的情况,确认他们是使用了多个SharePlex程序同步数据,而且redo放在多个磁盘组中,从而出现该问题.临时解决方案为把所有的redo和归档放一个磁盘组,这样多个SharePlex进程调用dbms_diskgroup访问redo/arch不会触发该bug.