ORACLE RAC 11.2.0.4 ASM加盘导致集群重启之ASM sga设置过小

    最近，一同事为一2节点的ORACLE RAC 11.2.0.4集群ASM加盘，没有注意到ASM的sga设置过小，加盘reblance时导致集群重启。详细描述如下：

    ​1、问题描述

    ​ORACLE RAC 11.2.0.4集群ASM加盘，没有注意到ASM的sga设置过小，加盘reblance时导致集群重启。

    ​2、问题分析

    ​2.1 首先，接到运维监控告警，某某平台发生严重告警

    ​2.2 登陆相关平台查看数据库和集群ASM日志，发现节点1集群发生重启，相关报错日志如下：

Wed Dec 26 17:20:29 2018

ARB0 started with pid=43, OS id=22557 

NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O

NOTE: Attempting voting file refresh on diskgroup ORADATA

Wed Dec 26 17:22:37 2018

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_22557.trc  (incident=208353):

ORA-04031: unable to allocate 7072 bytes of shared memory ("shared pool","unknown object","sga heap(1,1)","ges resource ")

Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208353/+ASM1_arb0_22557_i208353.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Wed Dec 26 17:22:39 2018

Dumping diagnostic data in directory=[cdmp_20181226172239], requested by (instance=1, osid=22557 (ARB0)), summary=[incident=208353].

Wed Dec 26 17:22:46 2018

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc  (incident=208345):

ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")

Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208345/+ASM1_m000_22771_i208345.trc

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc:

ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")

    ​2.3 数据库运维人员注意到，此前的主要操作主要就是给磁盘组加盘，观察ASM告警日志，相关操作记录如下：

Wed Dec 26 17:20:16 2018

SQL> ALTER DISKGROUP ORADATA ADD  DISK '/dev/rdisk/VD-DATA21' SIZE 182272M ,

'/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */ 

NOTE: GroupBlock outside rolling migration privileged region

NOTE: Assigning number (2,20) to disk (/dev/rdisk/VD-DATA21)

NOTE: Assigning number (2,21) to disk (/dev/rdisk/VD-DATA22)

NOTE: requesting all-instance membership refresh for group=2

NOTE: initializing header on grp 2 disk ORADATA_0020

NOTE: initializing header on grp 2 disk ORADATA_0021

NOTE: requesting all-instance disk validation for group=2

Wed Dec 26 17:20:19 2018

NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance.

NOTE: requesting all-instance disk validation for group=2

NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance.

NOTE: initiating PST update: grp = 2

Wed Dec 26 17:20:25 2018

GMON updating group 2 at 7 for pid 42, osid 22238

NOTE: PST update grp = 2 completed successfully 

NOTE: membership refresh pending for group 2/0x6b4139d2 (ORADATA)

GMON querying group 2 at 8 for pid 18, osid 5128

NOTE: cache opening disk 20 of grp 2: ORADATA_0020 path:/dev/rdisk/VD-DATA21

NOTE: cache opening disk 21 of grp 2: ORADATA_0021 path:/dev/rdisk/VD-DATA22

Wed Dec 26 17:20:29 2018

NOTE: Attempting voting file refresh on diskgroup ORADATA

GMON querying group 2 at 9 for pid 18, osid 5128

SUCCESS: refreshed membership for 2/0x6b4139d2 (ORADATA)

Wed Dec 26 17:20:29 2018

SUCCESS: ALTER DISKGROUP ORADATA ADD  DISK '/dev/rdisk/VD-DATA21' SIZE 182272M ,

'/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */

NOTE: starting rebalance of group 2/0x6b4139d2 (ORADATA) at power 1

Starting background process ARB0

Wed Dec 26 17:20:29 2018

ARB0 started with pid=43, OS id=22557 

NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O

NOTE: Attempting voting file refresh on diskgroup ORADATA

    ​2.4 到此，基本上可以确定是ASM加盘，ASM在进行reblance的时候，由于ASM的SGA设置过小导致的集群重启

    ​3、问题处理

    ​调整相关集群的ASM的SGA大小到2G，重新启动集群，ASM开始正常的reblance:

Wed Dec 26 17:56:32 2018

NOTE: client SYXPTDB1:SYXPTDB registered, osid 1074, mbr 0x1

Wed Dec 26 18:35:33 2018

ALTER SYSTEM SET memory_max_target='2G' SCOPE=SPFILE SID='*';

ALTER SYSTEM SET memory_target='2G' SCOPE=SPFILE SID='*';

ALTER SYSTEM SET sga_max_size='2G' SCOPE=SPFILE SID='*';

Wed Dec 26 20:50:11 2018

NOTE: GroupBlock outside rolling migration privileged region

NOTE: requesting all-instance membership refresh for group=2

Wed Dec 26 20:50:14 2018

NOTE: membership refresh pending for group 2/0x1edaa5d7 (ORADATA)

Wed Dec 26 20:50:17 2018

GMON querying group 2 at 7 for pid 18, osid 26306

SUCCESS: refreshed membership for 2/0x1edaa5d7 (ORADATA)

NOTE: Attempting voting file refresh on diskgroup ORADATA

 

    ​到此问题处理完成！