WARNING: Heavy swapping observed on system in last 5 mins.

问题背景：
客户反馈数据宕机，协助查看原因

1、查看alert日志

 

 1 opiodr aborting process unknown ospid (32412) as a result of ORA-609
 2 Mon Dec 30 08:53:48 2019
 3 WARNING: Heavy swapping observed on system in last 5 mins.
 4 pct of memory swapped in [3.27%] pct of memory swapped out [8.91%].
 5 Please make sure there is no memory pressure and the SGA and PGA 
 6 are configured correctly. Look at DBRM trace file for more details.
 7 Errors in file /u01/app/oracle/diag/rdbms/ecology/ecology/trace/ecology_dbrm_25203.trc (incident=300076):
 8 ORA-00700: soft internal error, arguments: [kskvmstatact: excessive swapping observed], [], [], [], [], [], [], [], [], [], [], []
 9 Incident details in: /u01/app/oracle/diag/rdbms/ecology/ecology/incident/incdir_300076/ecology_dbrm_25203_i300076.trc
10 Mon Dec 30 08:54:04 2019
11 Dumping diagnostic data in directory=[cdmp_20191230085404], requested by (instance=1, osid=25203 (DBRM)), summary=[incident=300076].
12 Mon Dec 30 08:54:42 2019

 

 

关于WARNING: Heavy swapping observed on system in last 5 mins.

Applies to:
Oracle Database - Enterprise Edition - Version 11.2.0.3 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.
Goal
The document explains why you may see the following WARNING in the alert log after upgrading to 11.2.0.3.

Wed Apr 25 10:10:50 2012
WARNING: Heavy swapping observed on system in last 5 mins.
pct of memory swapped in [7.16%] pct of memory swapped out [0.56%].
Please make sure there is no memory pressure and the SGA and PGA
are configured correctly. Look at DBRM trace file for more details.
Fix
The reason you may now see these informational messages in the alert is due to the fact that in 11.2.0.3 an enhancement fix was included for unpublished
Bug 10220118 - Print warning to alert log when system is swapping (Doc ID 10220118.8)

Please refer to
Note 10220118.8 - Bug 10220118 - Print warning to alert log when system is swapping

 

Description

  This fix is really an enhancement.
  With this fix a warning is printed to the alert log (and DBRM trace)
  if excessive swapping is observed on the system.
  The warning will not be printed more than once per hour.
  

The same issues could have been happening on 11.2.0.2 but they simply would not have been reported.
There is no cause for concern and we suggest that the databases are just monitored as usual.

If you look in the alert log, you will see that the message does not happen every hour. 
If SWAP was causing any serious issue we could also see ORA-4031/ORA-4030 and would then be able to use the DBRM trace in conjunction with obtaining further diagnostics.

 

- If your Platform. is IBM-AIX then This is not the only reason for this alert log file. There is a known port-specific bug for IBM AIX on POWER Systems (64-bit):

Bug 14731911 : FALSE SWAP WARNING MESSAGES PRINTED TO ALERT.LOG ON AIX
Base Bug 11801934 : WRONG PAGE-IN AND PAGE-OUT OS VM STATS IN AIX.

Solution for AIX
Apply patch 11801934 if available for your platform. or request a patch to Oracle Support in a new Service Request.

关于ORA-00700
According to the MOS note ID 737878.1, ORA-700 errors are referred to as soft asserts in 11g. Basically these are internal errors that do not cause anything fatal to occur in the process, and so the process can continue. Soft asserts are triggered when the caller wants to make a note of the fact that something unexpected has happened, but would like to continue on because the failure is not fatal to the process or the instance. In such cases, the process just records the error, takes the corresponding dumps and moves on as though nothing happened.