最近,一同事为一2节点的ORACLE RAC 11.2.0.4集群ASM加盘,没有注意到ASM的sga设置过小,加盘reblance时导致集群重启。详细描述如下:
1、问题描述
ORACLE RAC 11.2.0.4集群ASM加盘,没有注意到ASM的sga设置过小,加盘reblance时导致集群重启。
2、问题分析
2.1 首先,接到运维监控告警,某某平台发生严重告警
2.2 登陆相关平台查看数据库和集群ASM日志,发现节点1集群发生重启,相关报错日志如下:
Wed Dec 26 17:20:29 2018
ARB0 started with pid=43, OS id=22557
NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O
NOTE: Attempting voting file refresh on diskgroup ORADATA
Wed Dec 26 17:22:37 2018
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_22557.trc (incident=208353):
ORA-04031: unable to allocate 7072 bytes of shared memory ("shared pool","unknown object","sga heap(1,1)","ges resource ")
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208353/+ASM1_arb0_22557_i208353.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Dec 26 17:22:39 2018
Dumping diagnostic data in directory=[cdmp_20181226172239], requested by (instance=1, osid=22557 (ARB0)), summary=[incident=208353].
Wed Dec 26 17:22:46 2018
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc (incident=208345):
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_208345/+ASM1_m000_22771_i208345.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_m000_22771.trc:
ORA-04031: unable to allocate 12312 bytes of shared memory ("shared pool","unknown object","KKSSP^1232","kglseshtTable")
2.3 数据库运维人员注意到,此前的主要操作主要就是给磁盘组加盘,观察ASM告警日志,相关操作记录如下:
Wed Dec 26 17:20:16 2018
SQL> ALTER DISKGROUP ORADATA ADD DISK '/dev/rdisk/VD-DATA21' SIZE 182272M ,
'/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */
NOTE: GroupBlock outside rolling migration privileged region
NOTE: Assigning number (2,20) to disk (/dev/rdisk/VD-DATA21)
NOTE: Assigning number (2,21) to disk (/dev/rdisk/VD-DATA22)
NOTE: requesting all-instance membership refresh for group=2
NOTE: initializing header on grp 2 disk ORADATA_0020
NOTE: initializing header on grp 2 disk ORADATA_0021
NOTE: requesting all-instance disk validation for group=2
Wed Dec 26 17:20:19 2018
NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance.
NOTE: requesting all-instance disk validation for group=2
NOTE: skipping rediscovery for group 2/0x6b4139d2 (ORADATA) on local instance.
NOTE: initiating PST update: grp = 2
Wed Dec 26 17:20:25 2018
GMON updating group 2 at 7 for pid 42, osid 22238
NOTE: PST update grp = 2 completed successfully
NOTE: membership refresh pending for group 2/0x6b4139d2 (ORADATA)
GMON querying group 2 at 8 for pid 18, osid 5128
NOTE: cache opening disk 20 of grp 2: ORADATA_0020 path:/dev/rdisk/VD-DATA21
NOTE: cache opening disk 21 of grp 2: ORADATA_0021 path:/dev/rdisk/VD-DATA22
Wed Dec 26 17:20:29 2018
NOTE: Attempting voting file refresh on diskgroup ORADATA
GMON querying group 2 at 9 for pid 18, osid 5128
SUCCESS: refreshed membership for 2/0x6b4139d2 (ORADATA)
Wed Dec 26 17:20:29 2018
SUCCESS: ALTER DISKGROUP ORADATA ADD DISK '/dev/rdisk/VD-DATA21' SIZE 182272M ,
'/dev/rdisk/VD-DATA22' SIZE 182272M /* ASMCA */
NOTE: starting rebalance of group 2/0x6b4139d2 (ORADATA) at power 1
Starting background process ARB0
Wed Dec 26 17:20:29 2018
ARB0 started with pid=43, OS id=22557
NOTE: assigning ARB0 to group 2/0x6b4139d2 (ORADATA) with 1 parallel I/O
NOTE: Attempting voting file refresh on diskgroup ORADATA
2.4 到此,基本上可以确定是ASM加盘,ASM在进行reblance的时候,由于ASM的SGA设置过小导致的集群重启
3、问题处理
调整相关集群的ASM的SGA大小到2G,重新启动集群,ASM开始正常的reblance:
Wed Dec 26 17:56:32 2018
NOTE: client SYXPTDB1:SYXPTDB registered, osid 1074, mbr 0x1
Wed Dec 26 18:35:33 2018
ALTER SYSTEM SET memory_max_target='2G' SCOPE=SPFILE SID='*';
ALTER SYSTEM SET memory_target='2G' SCOPE=SPFILE SID='*';
ALTER SYSTEM SET sga_max_size='2G' SCOPE=SPFILE SID='*';
Wed Dec 26 20:50:11 2018
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=2
Wed Dec 26 20:50:14 2018
NOTE: membership refresh pending for group 2/0x1edaa5d7 (ORADATA)
Wed Dec 26 20:50:17 2018
GMON querying group 2 at 7 for pid 18, osid 26306
SUCCESS: refreshed membership for 2/0x1edaa5d7 (ORADATA)
NOTE: Attempting voting file refresh on diskgroup ORADATA
到此问题处理完成!