oracle: ORA-27300, ORA-27301, ORA-27302 on AIX (文档 ID 1541121.1)

双节点的aix下的11204rac的警告日志:

Sat Aug 04 07:21:01 2018
WARNING: Heavy swapping observed on system in last 5 mins.
pct of memory swapped in [1.11%] pct of memory swapped out [1.00%].
Please make sure there is no memory pressure and the SGA and PGA 
are configured correctly. Look at DBRM trace file for more details.
Sat Aug 04 07:31:53 2018
Process startup failed, error stack:
Errors in file /oracle/app/oracle/diag/rdbms/nc65prd/nc65prd2/trace/nc65prd2_psp0_2425524.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sat Aug 04 07:31:54 2018
Process m000 died, see its trace file
Sat Aug 04 07:33:45 2018
Process startup failed, error stack:
Errors in file /oracle/app/oracle/diag/rdbms/nc65prd/nc65prd2/trace/nc65prd2_psp0_2425524.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Sat Aug 04 07:33:46 2018
Process W000 died, see its trace file
Process startup failed, error stack:
Errors in file /oracle/app/oracle/diag/rdbms/nc65prd/nc65prd2/trace/nc65prd2_psp0_2425524.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Process W000 died, see its trace file
Process startup failed, error stack:
 

服务器内存128g,swap空间65g.sga为30g.作为单纯的数据库服务器,swap空间被用尽的可能性不大.查询mos,发下如下问题

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
IBM AIX on POWER Systems (64-bit)
***Checked for relevance on 03-July-2017***

SYMPTOMS

Received following errors in alert log:


ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

- AIX server became unusable.

- OS commands such as ps returns malloc or fork errors.

- Can not connect to the database

- Command line hangs

- No Oracle or user process is using abnormal amount of memory.

- There are many defunct kernel processes

 

CHANGES

 

CAUSE

In this case, this is an IBM bug:

IV37048 CIFS_FS LEAVES BEHIND DEFUNCT KERNEL PROCESSES.
 

SOLUTION

Apply IBM patch: IV37048 CIFS_FS LEAVES BEHIND DEFUNCT KERNEL PROCESSES

The IBM patch will prevent zombies (defunct processes) and prevents symptoms.

 
Additional potential patch for AIX 7.1:
IV37469: CIFS_FS LEAVES BEHIND DEFUNCT KERNEL PROCESSES APPLIES TO AIX 7100-03

REFERENCES

NOTE:579365.1 - Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors
NOTE:372157.1 - How to enable Large Page Feature on AIX-Based Systems
NOTE:282036.1 - Minimum Software Versions and Patches Required to Support Oracle Products on IBM Power Systems
NOTE:465002.1 - Database Crash With Error ORA-00490
NOTE:580552.1 - Database Crashes With ORA-04030 ORA-07445 ORA-27300 ORA-27301 ORA-27302
NOTE:560309.1 - Database Cannot Start Due to Lack of Memory