aix oracle 内存耗尽,AIX swap耗尽导致oracle实例down

早上发现数据库实例无法访问，ping 数据库服务器网络正常，无法telnet登陆进去。通过管理口进去重启没相应，强制重启后进入系统检查如下：# errptIDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTIONAFA89905   0830100915 I O grpsvcs  ...显示全部

早上发现数据库实例无法访问，ping 数据库服务器网络正常，无法telnet登陆进去。通过管理口进去重启没相应，强制重启后进入系统检查如下：# errptIDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION

AFA89905   0830100915 I O grpsvcs        Group Services daemon started

97419D60   0830100915 I O topsvcs        Topology Services daemon started

A6DF45AA   0830100915 I O RMCdaemon      The daemon is started.

BE0A03E5   0830100515 P H sysplanar0     ENVIRONMENTAL PROBLEM

9DBCFDEE   0830100715 T O errdemon       ERROR LOGGING TURNED ON

BE0A03E5   0830094715 P H sysplanar0     ENVIRONMENTAL PROBLEM

C5C09FFA   0830033915 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED

C5C09FFA   0830033915 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED

C5C09FFA   0830033915 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED

C5C09FFA   0830033915 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED

C5C09FFA   0830033915 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED

C5C09FFA   0830033915 P S SYSVMM         SOFTWARE PROGRAM ABNORMALLY TERMINATED

E18E984F   0830032415 P S SRC            SOFTWARE PROGRAM ERROR

CB4A951F   0830032415 I S SRC            SOFTWARE PROGRAM ERROR

AA8AB241   0830032215 T O clstrmgrDANGER OPERATOR NOTIFICATION

AA8AB241   0801021415 T O clstrmgrES     OPERATOR NOTIFICATION

51E537B5   0729003315 P H sysplanar0     platform_dump saved to file

A6D1BD62   0729003215 I H unspecified    Firmware Event

291D64C3   0729003115 I H sysplanar0     Platform dump data

3C81E43F   0724030315 P U topsvcs        Late in sending heartbeat

3C81E43F   0714030315 P U topsvcs        Late in sending heartbeat

3C81E43F   0627030215 P U topsvcs        Late in sending heartbeat

然后# errpt -aj C5C09FFA|pg 查看错误

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

Date/Time:       Sun Aug 30 03:39:33 CST 2015

Sequence Number: 876

Machine Id:      00F75D6A4C00

Node Id:         calldb2

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

Recommended Actions

DEFINE ADDITIONAL PAGING SPACE

REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data

PROGRAM

expdp

USER'S PROCESS ID:

29098094

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

2408

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

Date/Time:       Sun Aug 30 03:39:33 CST 2015

Sequence Number: 875

Machine Id:      00F75D6A4C00

Node Id:         calldb2

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

Recommended Actions

DEFINE ADDITIONAL PAGING SPACE

REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data

PROGRAM

oracle

USER'S PROCESS ID:

25559232

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

3668

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

Date/Time:       Sun Aug 30 03:39:33 CST 2015

Sequence Number: 874

Machine Id:      00F75D6A4C00

Node Id:         calldb2

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

Recommended Actions

DEFINE ADDITIONAL PAGING SPACE

REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data

PROGRAM

oracle

USER'S PROCESS ID:

27000900

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

9616

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

Date/Time:       Sun Aug 30 03:39:33 CST 2015

Sequence Number: 873

Machine Id:      00F75D6A4C00

Node Id:         calldb2

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

Recommended Actions

DEFINE ADDITIONAL PAGING SPACE

REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data

PROGRAM

oracle

USER'S PROCESS ID:

30408938

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

40336

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

Date/Time:       Sun Aug 30 03:39:33 CST 2015

Sequence Number: 872

Machine Id:      00F75D6A4C00

Node Id:         calldb2

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

Recommended Actions

DEFINE ADDITIONAL PAGING SPACE

REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data

PROGRAM

oracle

USER'S PROCESS ID:

33947854

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

1536

---------------------------------------------------------------------------

LABEL:          PGSP_KILL

IDENTIFIER:     C5C09FFA

Date/Time:       Sun Aug 30 03:39:33 CST 2015

Sequence Number: 871

Machine Id:      00F75D6A4C00

Node Id:         calldb2

Class:           S

Type:            PERM

WPAR:            Global

Resource Name:   SYSVMM

Description

SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes

SYSTEM RUNNING OUT OF PAGING SPACE

Failure Causes

INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM

PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

Recommended Actions

DEFINE ADDITIONAL PAGING SPACE

REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data

PROGRAM

oracle

USER'S PROCESS ID:

29491234

PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS

0

#

提示oracle 进程导致swap耗尽。。各位有谁能给予解决思路？收起