关闭

ASMB的BUG(ORA-04030 kfmditer)导致数据库宕机

标签: databaseoracleracasm宕机
8214人阅读 评论(0) 收藏 举报
分类:
ASMB的BUG(ORA-04030 kfmditer)导致数据库宕机
现象:
客户的一个重要生产系统RAC的一个实例宕机,查看alert日志:

Fri Jun 21 17:05:52 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc (incident=31397):
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc

Fri Jun 21 17:05:55 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_rbal_11389.trc (incident=31389):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31389/jyj1_rbal_11389_i31389.trc
Fri Jun 21 17:06:14 2013
Instance terminated by ASMB, pid = 11391

查看asmb trace文件:
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc (incident=31397):
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc
Fri Jun 21 17:05:52 2013
Trace dumping is performing id=[cdmp_20130621170552]
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc:
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)
ASMB (ospid: 11391): terminating the instance due to error 4030
System state dump is made for local instance
System State dumped to trace file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_diag_11345.trc
Fri Jun 21 17:05:53 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lms0_11363.trc (incident=31301):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31301/jyj1_lms0_11363_i31301.trc
Fri Jun 21 17:05:53 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lmon_11359.trc (incident=31277):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31277/jyj1_lmon_11359_i31277.trc
Fri Jun 21 17:05:53 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lms1_11367.trc (incident=31309):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31309/jyj1_lms1_11367_i31309.trc
Fri Jun 21 17:05:54 2013
ORA-1092 : opitsk aborting process
Fri Jun 21 17:05:54 2013
License high water mark = 327
Fri Jun 21 17:05:55 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_rbal_11389.trc (incident=31389):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31389/jyj1_rbal_11389_i31389.trc
Fri Jun 21 17:06:14 2013
Instance terminated by ASMB, pid

jyj1_asmb_11391_i31397.trc:

Dump file /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /opt/app/ora11gR1db
System name: Linux
Node name: KSJYJ_DB01
Release: 2.6.18-164.el5
Version: #1 SMP Thu Sep 3 04:15:13 EDT 2009
Machine: x86_64
Instance name: jyj1
Redo thread mounted by this instance: 1
Oracle process number: 24
Unix process pid: 11391, image: oracle@KSJYJ_DB01 (ASMB)


*** 2013-06-21 17:05:52.045
*** SESSION ID:(532.1) 2013-06-21 17:05:52.046
*** CLIENT ID:() 2013-06-21 17:05:52.046
*** SERVICE NAME:(SYS$BACKGROUND) 2013-06-21 17:05:52.046
*** MODULE NAME:() 2013-06-21 17:05:52.046
*** ACTION NAME:() 2013-06-21 17:05:52.046
 
Dump continued from file: /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)

========= Dump for incident 31397 (ORA 4030) ========

*** 2013-06-21 17:05:52.046
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.

skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp
 <- ksfdmp <- dbgexPhaseII <- dbgexProcessError <- dbgeExecuteForError <- dbgePostErrorKGE
 <- 1774 <- dbkePostKGE_kgsf <- kgesev <- kgesec3 <- kghnospc
 <- kghalf <- kfmdIterInit <- kfkIterInit <- kfnbIostatiterOp <- 110
 <- kfnbRun <- ksbrdp <- opirip <- opidrv <- sou2o

Process state
-----------------------

SO: 0x940dd1b98, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0x940dd1b98, name=process, file=ksu.h LINE:10286, pg=0
 (process) Oracle pid:24, ser:1, calls cur/top: 0x920f28eb8/0x920f28eb8
 flags: (0x6) SYSTEM
 int error: 0, call error: 0, sess error: 0, txn error 0
 (post info) last post received: 0 0 34
 last post received-location: ksr2.h LINE:594 ID:ksrpublish
 last process to post me: 950dfd540 47 2
 last post sent: 0 0 64
 last post sent-location: kso2.h LINE:316 ID:ksoreq_reply
 last process posted by me: 930e5c948 1 0
 (latch info) wait_event=0 bits=0
 Process Group: DEFAULT, pseudo proc: 0x950e4c060
 O/S info: user: oracle, term: UNKNOWN, ospid: 11391
 OSD pid info: Unix process pid: 11391, image: oracle@KSJYJ_DB01 (ASMB)
Dump of memory from 0x00000009D0DC0A10 to 0x00000009D0DC0C18


分析:
从报错信息(ORA-04030)看来,怀疑是Oracle的BUG导致的,因为以前碰到过类似的ASMB进程内存泄露的BUG,
于是搜索metalink关键词:asmb 04030
发现第一篇就跟客户的问题吻合。
ASMB process grows raising ora-4030 intermittently (Doc ID 735180.1)
ASMB process grows on memory, eventually leading to ora-4030 errors
which causes DB crash.

The reported error:
ORA-04030: out of process memory when trying to allocate 552 Bytes (callheap,kfmditer)
 
In the ASMB process heapdump we can see most of memory chunks are for 'kfmditer',
example:

 BreakDown
 ~~~~~~~~~
 Type     Count   Sum        Average
 ~~~~     ~~~~~   ~~~        ~~~~~~~
 Free     285684  142841492  500.00
 kfmditer 285685  157698132  552.00   <-- 在ASMB的HEAPDUMP中也看到了绝大多数都为kfmditer的内存片

 Total = 300539624 bytes 293495.73k 286.62MB
 
 这个BUG在11.1以后的大版本中都有出现,但是在以下的patchset中被修复:
 
 This issue is fixed in

11.2.0.1 (Base Release)
11.1.0.7.1 (Patch Set Update)
10.2.0.5 (Server Patch Set)
11.1.0.7 Patch 11 on Windows Platforms
11.1.0.7 RAC Recommended Patch Bundle #1
11.1.0.6 Patch 11 on Windows Platforms

如果不想做patchset升级的话,也可以直接打个小Patch 6851110可以解决这个问题。
You can check if Patch 6851110 is available for your patchset release and
O/S environment.:  Patch 6851110

解决方法:
在客户的数据库上打patch  6851110,经过持续观察一段时间,该问题未再现。
1
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:153839次
    • 积分:2697
    • 等级:
    • 排名:第14327名
    • 原创:111篇
    • 转载:14篇
    • 译文:1篇
    • 评论:16条
    文章分类
    最新评论