ORA-00600: internal error code, arguments: [16703], [1403], [28], [...

联系：手机/微信(+86 13429648788) QQ(107644445)

标题：10g数据库遭遇ORA-600 16703

作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

ORA-600 16703 
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi PL/SQL Release 10.2.0.3.0 - Production CORE    10.2.0.3.0  Production TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio NLSRTL Version 10.2.0.3.0 - Production

Fri Jan  4 10:11:05 2019 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Jan  4 10:11:05 2019 SMON: enabling cache recovery Fri Jan  4 10:11:05 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952 .trc: ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan  4 10:11:06 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952 .trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan  4 10:11:06 2019 Error 704 happened during db open , shutting down database USER: terminating instance due to error 704 Instance terminated by USER, pid = 1826952 ORA-1092 signalled during: alter database open ...

看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []，但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28

分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.

--节点1 Fri Jan  4 06:11:00 2019 Database Characterset is ZHS16GBK replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=29, OS id =2883786 Fri Jan  4 06:11:07 2019 SMON: Parallel transaction recovery tried Fri Jan  4 06:11:07 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106 .trc: ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], [] Fri Jan  4 06:11:09 2019 ORA-600 signalled during: ALTER DATABASE OPEN... Fri Jan  4 06:11:09 2019 Trace dumping is performing id =[cdmp_20190104061109] Fri Jan  4 06:11:30 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550 .trc: ORA-00604: error occurred at recursive SQL level 1 ORA-20001: the IP can not logon ORA-06512: at line 36 Fri Jan  4 06:12:35 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508 .trc: ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], [] Fri Jan  4 06:12:40 2019 Trace dumping is performing id =[cdmp_20190104061240] Fri Jan  4 06:16:10 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540 .trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] Fri Jan  4 06:16:10 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546 .trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] Fri Jan  4 06:16:11 2019 Trace dumping is performing id =[cdmp_20190104061611] Fri Jan  4 06:16:15 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636 .trc: ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan  4 06:16:16 2019 Non-fatal internal error happenned while SMON was doing logging scn-> time mapping. SMON encountered 1 out of maximum 100 non-fatal internal errors. Fri Jan  4 06:17:28 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774 .trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] ………… Fri Jan  4 06:56:47 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636 .trc: ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan  4 06:56:49 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030 .trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] ORA-06544: PL /SQL : internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], [] ORA-06544: PL /SQL : internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], [] Fri Jan  4 06:56:50 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524 .trc: ORA-00474: SMON process terminated with error Fri Jan  4 06:56:50 2019 PMON: terminating instance due to error 474 Fri Jan  4 06:56:50 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590 .trc: ORA-00474: SMON process terminated with error Fri Jan  4 06:56:50 2019 System state dump is made for local instance System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192 .trc Fri Jan  4 06:56:51 2019 Shutting down instance (abort)

节点一重启触发故障之后，节点二也开始报错，然后数据库直接挂掉

--节点2 Fri Jan  4 06:21:19 2019 Trace dumping is performing id =[cdmp_20190104062118] Fri Jan  4 06:21:22 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440 .trc: ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], [] Fri Jan  4 06:21:24 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262 .trc: ORA-00604: error occurred at recursive SQL level 1 ORA-20001: the IP can not logon ORA-06512: at line 36 Fri Jan  4 06:21:56 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888 .trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] Fri Jan  4 06:24:38 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340 .trc: ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], [] Fri Jan  4 06:24:44 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340 .trc: ORA-12012: error on auto execute of job 351 ORA-12008: error in materialized view refresh path ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], [] Fri Jan  4 06:28:48 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456 .trc: ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], [] Fri Jan  4 06:28:49 2019 ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan  4 07:30:25 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800 .trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], [] ………… Fri Jan  4 07:31:23 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956 .trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] Fri Jan  4 07:31:24 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956 .trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] Fri Jan  4 07:31:24 2019 PMON: terminating instance due to error 472 Fri Jan  4 07:31:24 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432 .trc: ORA-00472: PMON  process terminated with error

两个节点再次启动报ORA-600 16703错误

Fri Jan  4 07:31:50 2019 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Jan  4 07:31:50 2019 SMON: enabling cache recovery Fri Jan  4 07:31:50 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516 .trc: ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan  4 07:31:52 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516 .trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan  4 07:31:52 2019 Error 704 happened during db open , shutting down database USER: terminating instance due to error 704 Fri Jan  4 07:31:52 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764 .trc: ORA-00704: bootstrap process failure Instance terminated by USER, pid = 3813516 ORA-1092 signalled during: ALTER DATABASE OPEN...

10046跟踪启动过程

---

分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动

procedure     DBMS_DBMONITOR wrapped a000000 369 abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd 7 15a 171 VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2 IQkFph+2IhlEBrPrXe14giDow /HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx /pqFncbX/tLwp0NY5E Uu4E54MZ34yVtDQybwljVqp06KHqWN /ZwZJpvT +2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ +E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D VtDc4f6

通过对损坏的字典进行恢复,实现数据0丢失