RHEL 7上Oracle 莫名宕机 ORA-27300 ORA-27301 ORA-27302 ORA-27157

现象:

Oracle 执行job:expdp,数据库实例直接宕掉,报错如下:

ORA-27300: OS system dependent operation:semctl failed with status: 22

ORA-27301: OS failure message: Invalid argument

ORA-27302: failure occurred at: sskgpwrm1

ORA-27157: OS post/wait facility removed

ORA-27300: OS system dependent operation:semop failed with status: 36

ORA-27301: OS failure message: Identifier removed

ORA-27302: failure occurred at: sskgpwwait1

数据库版本11.2.0.4,操作系统版本:rhel 7

检查操作系统日志,定时任务日志并无异常;

原因:

OnRedHat 7.x, systemd-logind service introduced a new feature to remove all IPC objects when a user fully logs out.

The feature is controled by the option RemoveIPC in the /etc/systemd/logind.conf configuration file,

see man logind.conf(5) for details.

在 RHEL 7.x的版本里,systemd-logind 服务控制着用户的行为,当用户退出,会清理该用户的所有内存段.那么定时任务的数据泵就有问题了,调用的内存段是 oracle 的;

查看removeipc的说明:

RemoveIPC=

Controls whether System V and POSIX IPC objects belonging to the user shall be removed when the user fully logs out. Takes a boolean argument. If enabled, the user

may not consume IPC resources after the last of the user's sessions terminated. This covers System V semaphores, shared memory and message queues, as well as POSIX

 

shared memory and message queues. Note that IPC objects of the root user are excluded from the effect of this setting. Defaults to "yes".

The default value for RemoveIPC in RHEL7.2 is yes.

默认值为yes; 所以当有定时任务时:

As a result, when the last oracle or grid user disconnects, the OS removes shared memory segments and semaphores for those users.

As Oracle ASM and Databases use shared memory segments for SGA, removing shared memory segments will crash the Oracle ASM and database instances.

解决办法:

1) Set RemoveIPC=no in /etc/systemd/logind.conf

2) 重启服务

systemctl daemon-reload

systemctl restart systemd-logind

参考MOS:

ORA-27300 ORA-27301 ORA-27302 ORA-27157 Database Crash (Doc ID 438205.1)

Failed Install of RAC with ASM: ORA-27300 ORA-27302 ORA-27300 ORA-27301 ORA-27302 (Doc ID 2099563.1)