ora12700linux,苏大文正 ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302 BUG

Problem Description

---------------------------------------------------

Tue Sep 01 04:05:33 2020

skgxpvfynet: mtype: 61 process 417356 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)

Errors in file /u01/app/oracle/diag/rdbms/syntong/syntong1/trace/syntong1_w001_417356.trc (incident=96021):

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:sendmsg failed with status: 105

ORA-27301: OS failure message: No buffer space available

ORA-27302: failure occurred at: sskgxpsnd2

Incident details in: /u01/app/oracle/diag/rdbms/syntong/syntong1/incident/incdir_96021/syntong1_w001_417356_i96021.trc

opidrv aborting process W001 ospid (417356) as a result of ORA-603

Error Codes

---------------------------------------------------

ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302

根据MOS文档查询，发现匹配文档中的105错误：

Doc ID 579365.1:

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=80914981328617&id=579365.1

STATUS 105 - ENOBUFS - No buffer space available

This error means that a socket cannot be created until resources are freed.

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:sendmsg failed with status: 105

ORA-27301: OS failure message: No buffer space available

ORA-27302: failure occurred at: sskgxpsnd2

See: Oracle Linux: ORA-27301:OS Failure Message: No Buffer Space Available (Doc ID 2041723.1)

文档提供的解决方案：

Doc ID 2041723.1:

CAUSE

This happens due to less space available for network buffer reservation.

SOLUTION

1. On servers with High Physical Memory, the parameter vm.min_free_kbytes should be set in the order of 0.4% of total Physical Memory. This helps in keeping a larger range of defragmented memory pages available for network buffers reducing the probability of a low-buffer-space conditions.

*** For example, on a server which is having 256GB RAM, the parameter vm.min_free_kbytes should be set to 1073742 ***

On NUMA Enabled Systems, the value of vm.min_free_kbytes should be multiplied by the number of NUMA nodes since the value is to be split across all the nodes.

On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here 'n' is the number of NUMA nodes.

2. Additionally, the MTU value should be modified as below

#ifconfig lo mtu 16436

To make the change persistent over reboot add the following line in the file /etc/sysconfig/network-scripts/ifcfg-lo :

MTU=16436

Save the file and restart the network service to load the changes

#service network restart

Note : While making the changes in CRS nodes, if network is restarted while CRS is up, it can hung CRS. So cluster services should be stopped prior to the network restart.

vm.min_free_kbytes

该参数表示Linux VM最低保留多少的空闲内存空间，当可用的内存低于配置参数时，系统会进行cache内存的回收，来进行内存的释放。

单位是kb