前两天,客户数据库集群突然无法访问。一个节点的alert日志中出现大量ORA-07445: exception encountered: core dump [kghlkremf+0024] [SIGSEGV] [Address not mapped to object]和ORA-00476: RECO process terminated with error的错误。同时该节点实例使用sqlplus无法登陆
从另一个节点中可以看到有很多DFS lock handle等待事件,P1的值为1128857605,可以判断为CI(Cross-Instance)锁。对于这种锁,最简单粗暴的解决方法是重启两个节点。
我们看一下这个问题.
环境:
Oracle 10.2.0.4 RAC
AIX 5.3
故障现象:
1.节点2的数据库alert日志中首先数个服务器进程出现ORA-07445报错,之后后台进程RECO也出现ORA-07445错误,然后pmon进程不断的去恢复RECO进程
Errors in file /oracle/admin/orcl/udump/orcl2_ora_446622.trc:
ORA-07445: 出现异常错误: 核心转储 [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0xB4B000000023208] [] []
Tue Sep 23 09:49:24 2014
Trace dumping is performing id=[cdmp_20140923094924]
Tue Sep 23 09:49:33 2014
Errors in file /oracle/admin/orcl/udump/orcl2_ora_299228.trc:
ORA-07445: 出现异常错误: 核心转储 [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0xB4B000000023208] [] []
Tue Sep 23 09:49:43 2014
Errors in file /oracle/admin/orcl/udump/orcl2_ora_127730.trc:
ORA-07445: 出现异常错误: 核心转储 [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0xB4B000000023208] [] []
Tue Sep 23 09:49:51 2014
PMON failed to acquire latch, see PMON dump
Tue Sep 23 09:49:53 2014
Errors in file /oracle/admin/orcl/udump/orcl2_ora_484038.trc:
ORA-07445: 出现异常错误: 核心转储 [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0xB4B000000023208] [] []
Tue Sep 23 09:50:03 2014
Errors in file /oracle/admin/orcl/bdump/orcl2_reco_115088.trc:
ORA-07445: exception encountered: core dump [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0xB4B000000023208] [] []
Tue Sep 23 09:50:13 2014
Errors in file /oracle/admin/orcl/bdump/orcl2_pmon_115700.trc:
ORA-00476: RECO process terminated with error
Tue Sep 23 09:50:23 2014
Errors in file /oracle/admin/orcl/bdump/orcl2_pmon_115700.trc:
ORA-00476: RECO process terminated with error
Tue Sep 23 09:50:33 2014
Errors in file /oracle/admin/orcl/bdump/orcl2_pmon_115700.trc:
ORA-00476: RECO process terminated with error
Tue Sep 23 09:50:43 2014
Errors in file /oracle/admin/orcl/bdump/orcl2_pmon_115700.trc:
ORA-00476: RECO process terminated with error
Tue Sep 23 09:50:53 2014
2.查看udump目录中生成的trc文件,以及reco进程的trace文件。在call stack中都出现kghadd_reserved_ext,kghget_reserved_ext信息
*** 2014-09-23 09:50:03.110
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0xB4B000000023208] [] []
No current SQL statement being executed.
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst+001c bl ksedst1 000000000 ? 104A2CD0D ?
ksedmp+0290 bl ksedst 104A2C690 ?
ssexhd+03e0 bl ksedmp 300001D15 ?
000044C0 ? 00000000
kghget_reserved_ext bl kghadd_reserved_ext 000000000 ? 000000000 ?
ent+0214 ent 000000000 ? 000000000 ?
FFFFFFFFFFF5F40 ? 104AC36D4 ?
kghgex+0534 bl kghget_reserved_ext 000000000 ? 70000038F799CF8 ?
ent FFFFFFFFFFF5FB0 ?
2488C54000000000 ?
100203234 ?
kghfnd+0584 bl kghgex 700000389E9AAC0 ? 0000009A4 ?
000000001 ? 110000FF8 ?
000000001 ? 7000000100EE190 ?
FFFFFFFFFFF6110 ?
kghalo+0a24 bl kghfnd 102DCCC40 ? 000000000 ?
000000000 ? 44442D4D4F4E ?
问题分析:
这个问题是由Oracle的bug导致,参考Oracle文档 ID 468456.1
其原因是由于使用到bloom filter算法的SQL查询在执行的时候出现异常中断。这是一个内存错误导致,当内存重新分配给其他一些客户端时,原有的链接仍旧往这个内存中写入数据,所以导致了ORA-07445错误
关于bloom filter算法可以在网上搜索查找
只是有个问题,关于[kghlkremf+0024] 这个错误中的0024这个信号是从哪里查找的?
在Oracle的468456.1文档中,描述该问题出现的版本是Version 10.2.0.1 to 10.2.0.3。对于10.2.0.4版本,原话是这样的It can be provided only as part of a complete patchset so the first version that will include this fix is 10.2.0.4.
就是说这个问题将作为一个完整patch的一部分给用户提供,因此修复这个问题的第一个版本会是10.2.0.4
从这个现象来看,也许这个bug在10.2.0.4中可能并没有被修复.
解决该问题的方法是禁用bloom filter算法
alter system set "_bloom_filter_enabled"=false scope=both;
动态修改即可生效
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/23850820/viewspace-1281786/,如需转载,请注明出处,否则将追究法律责任。
转载于:http://blog.itpub.net/23850820/viewspace-1281786/