ORA-27300，ORA-27301，ORA-27302，ORA-04030导致crashed database

今天检查数据库的时候发现的alert日志报错，OS:HP UNIX 11.31 DB:Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 – 64bit Prodction RAC

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn5

ORA-27303: additional information: skgpspawn5

Mon Mar 14 15:23:00 EAT 2011

Process J001 died, see its trace file

Mon Mar 14 15:23:00 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:23:00 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 11

ORA-27301: OS failure message: Resource temporarily unavailable

ORA-27302: failure occurred at: skgpspawn5

ORA-27303: additional information: skgpspawn5

Mon Mar 14 15:46:09 EAT 2011

Process J000 died, see its trace file

Mon Mar 14 15:46:09 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:46:09 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Mon Mar 14 15:47:09 EAT 2011

Process startup failed, error stack:

Mon Mar 14 15:47:09 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

Mon Mar 14 15:47:10 EAT 2011

Process J000 died, see its trace file

Mon Mar 14 15:47:10 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:47:10 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Process startup failed, error stack:

Mon Mar 14 15:53:07 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

Mon Mar 14 15:53:08 EAT 2011

Process J001 died, see its trace file

Mon Mar 14 15:53:08 EAT 2011

kkjcre1p: unable to spawn jobq slave process

Mon Mar 14 15:53:08 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

 

Process startup failed, error stack:

Mon Mar 14 15:53:23 EAT 2011

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:

ORA-27300: OS system dependent operation:fork failed with status: 12

ORA-27301: OS failure message: Not enough space

ORA-27302: failure occurred at: skgpspawn3

Mon Mar 14 15:53:24 EAT 2011

Process m000 died, see its trace file

Mon Mar 14 15:53:24 EAT 2011

ksvcreate: Process(m000) creation failed

 

Errors in file /oracle/admin/scrk/bdump/scrk2_dbw0_3639.trc:

ORA-04030: out of process memory when trying to allocate 1049136 bytes (KSFD vec heap,KS

FD contig I/O buf)

Mon Mar 14 15:53:55 EAT 2011

System state dump is made for local instance

Mon Mar 14 15:53:55 EAT 2011

DBW0: terminating instance due to error 4030

Mon Mar 14 15:53:55 EAT 2011

System State dumped to trace file /oracle/admin/scrk/bdump/scrk2_diag_3620.trc

Mon Mar 14 15:53:58 EAT 2011

Shutting down instance (abort)

License high water mark = 174

Mon Mar 14 15:54:00 EAT 2011

Instance terminated by DBW0, pid = 3639

Mon Mar 14 15:54:03 EAT 2011

Instance terminated by USER, pid = 9322

Mon Mar 14 16:25:27 EAT 2011

Starting ORACLE instance (normal)

通过查询metalink [ID 579365.1] ：

ORA-27300: OS system dependent operation:fork failed with status: 11错误是由于：Maximum number of PROCESSES allowed per user may be too low (for example: maxuproc on AIX, maxuprc on HP and Solaris.  On Solaris 10, check project.max-lwps)。不过alert日志的ORA-27303: additional information: 是skgpspawn5不是文档中的skgpspawn3。由于对OS不熟，此问题暂时搁置，以备以后查询。

ORA-27300: OS system dependent operation:fork failed with status: 12 错误是由于：STATUS 12 - ENOMEM Not enough core / memory ;During an exec or a break, the program asked for more memory than the one available by the system. This error also occurs when there are too many segmentation registers which are required for the arrangement of text data or stack segments.

ORA-04030: out of process memory when trying to allocate 1049136 bytes (KSFD vec heap,KS

FD contig I/O buf)是由于dbw0没有请求到足够的内存导致了实力的重启。根据metalink  [ID 580552.1]在system log应当还有WARNING: Sorry, no swap space to grow stack for pid ***** (oracle)的警告信息，可是在此时间段内没有发现，此问题暂时搁置以待复查。