oracle shm alert,solaris修改shminfo_shmmax参数无效解决

环境介绍：双机

操作系统：solaris 10

数据库版本：oracle 11g R1 64bit

当前主机的主机名称和IP：fly-db2 192.168.20.21

当前备机的主机名称和IP：fly-db1 192.168.20.20

背景：

主备机的物理内存均为94G，近期对备机的物理内存扩容一倍，并修改了数据库层面的相关内存参数，以及主备机的操作系统参数shmsys:shminfo_shmmax，注意是主备的shmsys:shminfo_shmmax参数均已修改，详情请查看下面链接的博文

http://fly1116.blog.51cto.com/8301004/1343788

今晚对fly-db1主机(目前的备机)的物理内存扩容一倍后，重启操作系统，因主备机的shmsys:shminfo_shmmax参数均已经修改，重启操作系统后，理论上该参数设置生效，通过双机软件将数据库从fly-db2切换到fly-db1，切换失败，在双机软件中，查看到out of memory错误，在数据库alert日志，查看到如下报错：WARNING: The system does not seem to be configured

optimally. Creating a segment of size 0x0000000220000000

failed. Please change the shm parameters so that

a segment can be created for this size. While this is

not a fatal issue, creating one segment may improve performance

如果手工启动，则报错如下：

SQL> startup

ORA-27102: out of memory

SVR4 Error: 22: Invalid argument

bash-3.00$ oerr ora 27102

27102, 00000, "out of memory"

// *Cause: Out of memory

// *Action: Consult the trace file for details

从以上报错分析，因为使用的是同一个数据库，操作系统层面只修改了shmsys:shminfo_shmmax参数,定位为当前备机fly-db1该参数设置未生效

1、对比主备机的/etc/system文件，是一致的bash-3.00#hostname

fly-db1

bash-3.00# diff /etc/system /home/oracle/system_flydb2_131220.bak

2、百度之solaris out of memory,查看到以下blog，文章中涉及/etc/project文件，我们可以这么理解/etc/project文件，如果/etc/system和/etc/project中设置了相同的参数，则操作系统会忽略/etc/system的设置，而采用/etc/project的设置，/etc/project文件可以针对特定用户设置只对自己生效的参数

https://blogs.oracle.com/mandalika/entry/oracle_on_solaris_10_fixing   Oracle on Solaris 10 : Fixing the 'ORA-27102: out of memory' Error

3、查看到当前备机fly-db1主机的/etc/project中设置了max-shm-memory的最大大小为60G，即最大的共享内存段为60G，如下所示：bash-3.00# hostname

fly-db1

bash-3.00# cat /etc/project

system:0::::

user.root:1::::

noproject:2::::

default:3::::project.max-shm-memory=(privileged,64424509440,deny)

group.staff:10::::

bash-3.00# bc

scale=5

64424509440/1024/1024/1024

60.00000

4、查看当前主机(fly-db2)的/etc/project内容，未设置max-shm-memory的最大值，所以在该操作系统，生效的是/etc/system的设置bash-3.00#hostname

fly-db2

bash-3.00# cat /etc/project

system:0::::

user.root:1::::

noproject:2::::

default:3::::

group.staff:10::::

5、将当前备机fly-db1的/etc/project修改成和当前主机的/etc/project一样bash-3.00#hostname

fly-db1

bash-3.00# cat /etc/project

system:0::::

user.root:1::::

noproject:2::::

default:3::::

group.staff:10::::

6、重启操作系统bash-3.00# sync

bash-3.00# sync

bash-3.00# sync

bash-3.00# shutdown -i6 -g0 -y

7、双机切换，成功切换到备机fly-db1

以下内容来自：

https://blogs.oracle.com/mandalika/entry/oracle_on_solaris_10_fixing

Symptom:

As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with an ORA-27102:

out of memory error message. The system had enough free memory to serve the needs of Oracle.

SQL> startup

ORA-27102: out of memory

SVR4 Error: 22: Invalid argument

Diagnosis

$ oerr ORA 27102

27102, 00000, "out of memory"

// \*Cause: Out of memory

// \*Action: Consult the trace file for details

Not so helpful. Let's look the alert log for some clues.

% tail -2 alert.log

WARNING: EINVAL creating segment of size 0x000000028a006000

fix shm parameters in /etc/system or equivalent

Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example)

responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in /etc/system.

Prior to Solaris 10, shmsys:shminfo_shmmax parameter has to be set in /etc/system with maximum memory segment value that can

be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10

and later. On a Solaris 10 (or later) system, it can be verified as shown below:

% prtconf | grep Mem

Memory size: 32760 Megabytes

% id -p

uid=59008(oracle) gid=10001(dba) projid=3(default)

% prctl -n project.max-shm-memory -i project 3

project: 3: default

NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT

project.max-shm-memory

privileged 7.84GB - deny -

system 16.0EB max deny -

Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to

create a memory segment (10G) larger than 8G. Hence the failure.So, the solution is to configure the system with a value large

enough for the shared segment being created, so Oracle succeeds in starting up the database instance.

On Solaris 9 and prior releases, it can be done by adding the following line to /etc/system, followed by a reboot for the system

to pick up the new value.

set shminfo_shmmax = 0x000000028a006000

However shminfo_shmmax parameter was obsoleted with the release of Solaris 10; and Sun doesn't recommend setting this parameter

in /etc/system even though it works as expected.

On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is

how we do it on Solaris 10 and later:

% prctl -n project.max-shm-memory -r -v 10G -i project 3

% prctl -n project.max-shm-memory -i project 3

project: 3: default

NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT

project.max-shm-memory

privileged 10.0GB - deny -

system 16.0EB max deny -

Note that changes made with the prctl command on a running system are temporary, and will be lost when the system is rebooted. To make

the changes permanent, create a project with projadd command and associate it with the user account as shown below:

% projadd -p 3 -c 'eBS benchmark' -U oracle -G dba -K 'project.max-shm-memory=(privileged,10G,deny)' OASB

% usermod -K project=OASB oracle

Finally make sure the project is created with projects -l or cat /etc/project commands.

% projects -l

...

...

OASB

projid : 3

comment: "eBS benchmark"

users : oracle

groups : dba

attribs: project.max-shm-memory=(privileged,10737418240,deny)

% cat /etc/project

...

...

OASB:3:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)

With these changes, Oracle would start the database up normally.

SQL> startup

ORACLE instance started.

Total System Global Area 1.0905E+10 bytes

Fixed Size 1316080 bytes

Variable Size 4429966096 bytes

Database Buffers 6442450944 bytes

Redo Buffers 31457280 bytes

Database mounted.

Database opened.