swap耗尽引起ora- 471 （reason unknown）

Alert日志出现如下错误。
Thu Jan 17 16:04:01 2013
Thread 1 advanced to log sequence 706 (LGWR switch)
Current log# 1 seq# 706 mem# 0: /db/oracle11g/oradata/benguo/redo01.log
Thu Jan 17 16:06:00 2013
Thread 1 advanced to log sequence 707 (LGWR switch)
Current log# 4 seq# 707 mem# 0: /db/oracle11g/oradata/benguo/redo04.log
Thu Jan 17 20:03:53 2013
PMON (ospid: 72500): terminating the instance due to error 471
Instance terminated by PMON, pid = 72500

[oracle@server135 ~]$ oerr ora 471
00471, 00000, "DBWR process terminated with error"
// *Cause: The database writer process died
// *Action: Warm start instance

Process startup failed, error stack:
Errors in file /db/oracle11g/diag/rdbms/benguo/benguo/trace/benguo_psp0_41589.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Thu Jan 17 04:45:51 2013
Process m001 died, see its trace file

上面三个错误都和resource和内存有关，而导致了m001这个sga自动管理的子进程都dead了。

Dmesg的日志
Swap cache stats: add 103075105, delete 102890577, find 143284867/153505239
Free swap = 0kB
Total swap = 67108856kB
16777215 pages RAM
281381 pages reserved
3319924 pages shared
14482885 pages non-shared
Out of memory: kill process 16941 (dbus-daemon) score 56855 or a child
Killed process 16944 (dbus-daemon) vsz:2425828kB, anon-rss:2182360kB, file-rss:0kB
gvfs-gdu-volume[17260]: segfault at 18 ip 0000003ed1016220 sp 00007fffdec3b678 error 4 in libgdu.so.0.0.0[3ed1000000+38000]

看见dmesg的日志swap基本全部耗尽了。
[root@*****~]# free -m
total used free shared buffers cached
Mem: 64436 61326 3110 0 3 7647
-/+ buffers/cache: 53674 10762
Swap: 65535 58986 6549

等待n久终于登录到oracle中,感觉memory_target和memory_max_size都还是很正常，想对于64g的物理内存，这个值并不是很大。

SQL> show parameter sga;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
lock_sga boolean FALSE
pre_page_sga boolean FALSE
sga_max_size big integer 25856M
sga_target big integer 0
SQL> show parameter memor
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
hi_shared_memory_address integer 0
memory_max_target big integer 25856M
memory_target big integer 25856M
shared_memory_address integer 0
SQL> show parameter pga;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target big integer 0

虽然已经清楚了swap耗尽，引起了dbwn进程无法分配资源而导致实例崩溃，不过这个内存消耗来源却没有很详细的信息，由于并发的请求并不高，按理server完全应该能负载应用的请求。

后面由于server swap耗尽，导致相应极慢，而考虑到时间要求不得不采取最快的重启服务器方法，好在后续重启server后一切都正常，不过是何种原因消耗尽内存却不得了之，对于xiaoyu这个倒是一个损失。