磁盘组add disk,报错:ORA-15041 thrown in ARB0 for group number

磁盘组add disk,报错:ORA-15041 thrown in ARB0 for group number


在ASM中,每一个文件的extent都均匀的分布在它所在磁盘组的所有磁盘上,无论是在文件第一次创建或是文件创建之后的重新resize都是如此,这也意味着我们始终能保持磁盘组中的每一个磁盘上都有一个平衡的空间分配。

当我们ASM磁盘空间不够时,一般我们都会选择扩容,往磁盘组中添加新的磁盘,然后等待完成rebalance即可。但是,当我们没有注意到一些情况时,可能就会扩容失败。下面就说一种可能之前没有注意到的地方,导致扩容失败的场景。

故障现象

有套oracle 11.2.0.4 2节点rac的环境,监控显示,磁盘组data的剩余空间告警。所以登录环境,查看具体信息:
磁盘组详细信息:

grid@drrac01 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  N         512   4096  1048576   1966080    27640            32768           -2564              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576   3584000     6791                0            6791              0             N  DATADG01/
MOUNTED  HIGH    N         512   4096  1048576     10240     8984             4096            1629              0             Y  OCR_VOTING/

可以看到,data磁盘组为normal冗余,剩余空间只剩20多个G,Usable_file_MB已经成为负值,说明已经不能保证数据的冗余度,如果此时有块磁盘offline了,那么就有可能丢数据。所以继续磁盘组扩容。

经过主机分配资源,我们往磁盘组中添加磁盘:
alter diskgroup data add disk ‘xxx’ rebalance power 10;

但是,还没等返回成功信息,就报错了。报错如下:
ERROR: ORA-15041 thrown in ARB0 for group number 1
Errors in file /u01/oracle/app/grid/base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_16655.trc:
ORA-15041: diskgroup “DATA” space exhausted

问题分析

[oracle@oracle11g ~]$ oerr ora 15041
15041, 00000, “diskgroup “%s” space exhausted”
// *Cause: The diskgroup ran out of space.
// *Action: Add more disks to the diskgroup, or delete some existing files.
//
官方的意思指出,磁盘组空间耗尽。我们应该添加更多的磁盘或者删除一些文件。
和我们的场景有点类似,我们确实是空间已经耗尽(Usable_file_MB为负值)。但是我们确实是在添加磁盘,但是报错了,所以官方解释对我们没有帮助。
而且存储空间问题不是直接存储扩容就完事了?可惜,本例无论你添加再多的磁盘也无济于事。

参考mos:ORA-15041 IN A DISKGROUP ALTHOUGH FREE_MB REPORTS SUFFICIENT SPACE (Doc ID 460155.1)
我们很容易就能验证:
1、添加新磁盘前,60块磁盘大小都一致,为32GB;
2、磁盘状态都正常,没有DROPPING / HUNG状态的磁盘;
3、我们添加新磁盘的命令根本就没成功。
那么导致它的根因是什么哪?
我们通过查询,会发现有个别磁盘的free_mb过小,甚至为0。由于rebalance需要每个磁盘的一些可用空间,最终会导致normal模式下由于部分disk无法分配有效空间触发ORA-15041。
这里有个更严重的问题:
手工或自动rebalance的前提条件是每个磁盘必须要有50M~100M的可用空间,目前部分disk的free_mb=0,所以即使表空间使用率再高,情况再紧急,无论你有再多的新盘使劲地往DATA_DG添加也无济于事。

根因:
当磁盘组中有部分磁盘的free_mb<300MB的时候,rebalance就会报错,因为重新平衡操作需要每个磁盘中的一些可用空间,也就会报错ORA-15041。

解决方法

本案例通过下面直接手工rebalance的方式是无法成功的:
alter diskgroup name rebalance power level;
建议处理的方式有两种:
1.删除磁盘组上部分数据,以释放空间来满足rebalance的需要;
2.新建ASM磁盘组,进行disk group迁移。
删除数据文件的风险过大,时间窗口和多余磁盘允许下,建议使用迁移磁盘组的方式处理。

解决过程

1、查看asm磁盘使用率:
set linesize 300;
set pagesize 20;
col name for a15;
col state for a15;
col header_status for a15;
col path for a40;
col failgroup for a15;
select dk.group_number,dk.name,dk.header_status,dk.state,dk.mode_status,dk.failgroup,dk.os_mb,dk.total_mb,dk.free_mb,dk.path from v$asm_disk dk where group_number=1 order by dk.group_number,dk.failgroup,dk.path;

GROUP_NUMBER NAME	     HEADER_STATUS   STATE	     MODE_STATUS    FAILGROUP		 OS_MB	 TOTAL_MB    FREE_MB PATH
------------ --------------- --------------- --------------- -------------- --------------- ---------- ---------- ---------- ----------------------------------------
	   1 DATA_0000	     MEMBER	     NORMAL	     ONLINE	    DATA_0000		 32768	    32768	1338 /dev/asm-diskaa
	   1 DATA_0001	     MEMBER	     NORMAL	     ONLINE	    DATA_0001		 32768	    32768	 893 /dev/asm-diskab
	   1 DATA_0002	     MEMBER	     NORMAL	     ONLINE	    DATA_0002		 32768	    32768	 432 /dev/asm-diskac
	   1 DATA_0003	     MEMBER	     NORMAL	     ONLINE	    DATA_0003		 32768	    32768	   0 /dev/asm-diskad
	   1 DATA_0004	     MEMBER	     NORMAL	     ONLINE	    DATA_0004		 32768	    32768      11239 /dev/asm-diskae
	   1 DATA_0005	     MEMBER	     NORMAL	     ONLINE	    DATA_0005		 32768	    32768      10779 /dev/asm-diskaf
	   1 DATA_0006	     MEMBER	     NORMAL	     ONLINE	    DATA_0006		 32768	    32768      10322 /dev/asm-diskag
	   1 DATA_0007	     MEMBER	     NORMAL	     ONLINE	    DATA_0007		 32768	    32768	9879 /dev/asm-diskah
	   1 DATA_0008	     MEMBER	     NORMAL	     ONLINE	    DATA_0008		 32768	    32768	9409 /dev/asm-diskai
	   1 DATA_0009	     MEMBER	     NORMAL	     ONLINE	    DATA_0009		 32768	    32768	9423 /dev/asm-diskaj
	   1 DATA_0010	     MEMBER	     NORMAL	     ONLINE	    DATA_0010		 32768	    32768	9414 /dev/asm-diskak
	   1 DATA_0011	     MEMBER	     NORMAL	     ONLINE	    DATA_0011		 32768	    32768	9415 /dev/asm-diskal
	   1 DATA_0012	     MEMBER	     NORMAL	     ONLINE	    DATA_0012		 32768	    32768	9419 /dev/asm-diskam
	   1 DATA_0013	     MEMBER	     NORMAL	     ONLINE	    DATA_0013		 32768	    32768	9410 /dev/asm-diskan
	   1 DATA_0014	     MEMBER	     NORMAL	     ONLINE	    DATA_0014		 32768	    32768	9413 /dev/asm-diskao
	   1 DATA_0015	     MEMBER	     NORMAL	     ONLINE	    DATA_0015		 32768	    32768	9413 /dev/asm-diskap
	   1 DATA_0016	     MEMBER	     NORMAL	     ONLINE	    DATA_0016		 32768	    32768	9414 /dev/asm-diskaq

GROUP_NUMBER NAME	     HEADER_STATUS   STATE	     MODE_STATUS    FAILGROUP		 OS_MB	 TOTAL_MB    FREE_MB PATH
------------ --------------- --------------- --------------- -------------- --------------- ---------- ---------- ---------- ----------------------------------------
	   1 DATA_0017	     MEMBER	     NORMAL	     ONLINE	    DATA_0017		 32768	    32768	9412 /dev/asm-diskar
	   1 DATA_0018	     MEMBER	     NORMAL	     ONLINE	    DATA_0018		 32768	    32768	9414 /dev/asm-diskas
	   1 DATA_0019	     MEMBER	     NORMAL	     ONLINE	    DATA_0019		 32768	    32768	9410 /dev/asm-diskat
	   1 DATA_0020	     MEMBER	     NORMAL	     ONLINE	    DATA_0020		 32768	    32768	9416 /dev/asm-diskau
	   1 DATA_0021	     MEMBER	     NORMAL	     ONLINE	    DATA_0021		 32768	    32768	9414 /dev/asm-diskav
	   1 DATA_0022	     MEMBER	     NORMAL	     ONLINE	    DATA_0022		 32768	    32768	9414 /dev/asm-diskaw
	   1 DATA_0023	     MEMBER	     NORMAL	     ONLINE	    DATA_0023		 32768	    32768	9412 /dev/asm-diskax
	   1 DATA_0024	     MEMBER	     NORMAL	     ONLINE	    DATA_0024		 32768	    32768	9417 /dev/asm-diskay
	   1 DATA_0025	     MEMBER	     NORMAL	     ONLINE	    DATA_0025		 32768	    32768	9418 /dev/asm-diskaz
	   1 DATA_0026	     MEMBER	     NORMAL	     ONLINE	    DATA_0026		 32768	    32768	9405 /dev/asm-diskba
	   1 DATA_0027	     MEMBER	     NORMAL	     ONLINE	    DATA_0027		 32768	    32768	9414 /dev/asm-diskbb
	   1 DATA_0028	     MEMBER	     NORMAL	     ONLINE	    DATA_0028		 32768	    32768	9412 /dev/asm-diskbc
	   1 DATA_0029	     MEMBER	     NORMAL	     ONLINE	    DATA_0029		 32768	    32768	9415 /dev/asm-diskbd
	   1 DATA_0030	     MEMBER	     NORMAL	     ONLINE	    DATA_0030		 32768	    32768	9415 /dev/asm-diskbe
	   1 DATA_0031	     MEMBER	     NORMAL	     ONLINE	    DATA_0031		 32768	    32768	9412 /dev/asm-diskbf
	   1 DATA_0032	     MEMBER	     NORMAL	     ONLINE	    DATA_0032		 32768	    32768	9420 /dev/asm-diskbg
	   1 DATA_0033	     MEMBER	     NORMAL	     ONLINE	    DATA_0033		 32768	    32768	9416 /dev/asm-diskbh

GROUP_NUMBER NAME	     HEADER_STATUS   STATE	     MODE_STATUS    FAILGROUP		 OS_MB	 TOTAL_MB    FREE_MB PATH
------------ --------------- --------------- --------------- -------------- --------------- ---------- ---------- ---------- ----------------------------------------
	   1 DATA_0034	     MEMBER	     NORMAL	     ONLINE	    DATA_0034		 32768	    32768	9415 /dev/asm-diskbi
	   1 DATA_0035	     MEMBER	     NORMAL	     ONLINE	    DATA_0035		 32768	    32768	9417 /dev/asm-diskbj
	   1 DATA_0036	     MEMBER	     NORMAL	     ONLINE	    DATA_0036		 32768	    32768	9415 /dev/asm-diskbk
	   1 DATA_0037	     MEMBER	     NORMAL	     ONLINE	    DATA_0037		 32768	    32768	9418 /dev/asm-diskbl
	   1 DATA_0038	     MEMBER	     NORMAL	     ONLINE	    DATA_0038		 32768	    32768	9416 /dev/asm-diskbm
	   1 DATA_0039	     MEMBER	     NORMAL	     ONLINE	    DATA_0039		 32768	    32768	9419 /dev/asm-diskbn
	   1 DATA_0040	     MEMBER	     NORMAL	     ONLINE	    DATA_0040		 32768	    32768	9414 /dev/asm-diskbo
	   1 DATA_0041	     MEMBER	     NORMAL	     ONLINE	    DATA_0041		 32768	    32768	9414 /dev/asm-diskbp
	   1 DATA_0042	     MEMBER	     NORMAL	     ONLINE	    DATA_0042		 32768	    32768	9411 /dev/asm-diski
	   1 DATA_0043	     MEMBER	     NORMAL	     ONLINE	    DATA_0043		 32768	    32768	9413 /dev/asm-diskj
	   1 DATA_0044	     MEMBER	     NORMAL	     ONLINE	    DATA_0044		 32768	    32768	9416 /dev/asm-diskk
	   1 DATA_0045	     MEMBER	     NORMAL	     ONLINE	    DATA_0045		 32768	    32768	9422 /dev/asm-diskl
	   1 DATA_0046	     MEMBER	     NORMAL	     ONLINE	    DATA_0046		 32768	    32768	9415 /dev/asm-diskm
	   1 DATA_0047	     MEMBER	     NORMAL	     ONLINE	    DATA_0047		 32768	    32768	9417 /dev/asm-diskn
	   1 DATA_0048	     MEMBER	     NORMAL	     ONLINE	    DATA_0048		 32768	    32768	9411 /dev/asm-disko
	   1 DATA_0049	     MEMBER	     NORMAL	     ONLINE	    DATA_0049		 32768	    32768	9423 /dev/asm-diskp
	   1 DATA_0050	     MEMBER	     NORMAL	     ONLINE	    DATA_0050		 32768	    32768	9415 /dev/asm-diskq

GROUP_NUMBER NAME	     HEADER_STATUS   STATE	     MODE_STATUS    FAILGROUP		 OS_MB	 TOTAL_MB    FREE_MB PATH
------------ --------------- --------------- --------------- -------------- --------------- ---------- ---------- ---------- ----------------------------------------
	   1 DATA_0051	     MEMBER	     NORMAL	     ONLINE	    DATA_0051		 32768	    32768	9414 /dev/asm-diskr
	   1 DATA_0052	     MEMBER	     NORMAL	     ONLINE	    DATA_0052		 32768	    32768	9865 /dev/asm-disks
	   1 DATA_0053	     MEMBER	     NORMAL	     ONLINE	    DATA_0053		 32768	    32768      10323 /dev/asm-diskt
	   1 DATA_0054	     MEMBER	     NORMAL	     ONLINE	    DATA_0054		 32768	    32768      10774 /dev/asm-disku
	   1 DATA_0055	     MEMBER	     NORMAL	     ONLINE	    DATA_0055		 32768	    32768      11226 /dev/asm-diskv
	   1 DATA_0056	     MEMBER	     NORMAL	     ONLINE	    DATA_0056		 32768	    32768	   0 /dev/asm-diskw
	   1 DATA_0057	     MEMBER	     NORMAL	     ONLINE	    DATA_0057		 32768	    32768	 432 /dev/asm-diskx
	   1 DATA_0058	     MEMBER	     NORMAL	     ONLINE	    DATA_0058		 32768	    32768	 895 /dev/asm-disky
	   1 DATA_0059	     MEMBER	     NORMAL	     ONLINE	    DATA_0059		 32768	    32768	1350 /dev/asm-diskz
	   1 DATA_0060	     MEMBER	     NORMAL	     ONLINE	    DATA_0060		512000	   512000     388283 /dev/asm-disksdce

61 rows selected.

可以看到结果中总共61块盘,前60块盘都为32G,最后一块为新添加的512G,normal冗余模式下,每块磁盘都是默认添加,自己是自己的failgroup。但是前60块磁盘的磁盘使用率都不一致,按照ASM使用率均衡的原理,所有磁盘的剩余空间应该都一样才对。在磁盘组中每个磁盘大小不一致的情况下,才有可能使用率可能不一致。但是这里有大有小,而且有两块盘的剩余空间已经为0。
手工或自动rebalance的前提条件是每个磁盘必须要有50M~100M的可用空间,目前部分disk的free_mb=0,无法rebalance,所以报错ORA-15041。无论你有再多的新盘使劲地往DATA_DG添加也无济于事。
所以现在想办法要把那两块盘的free_mb调整为不是0.那么唯一的办法就是把数据文件挪到其他位置,释放点空间。
2、resize 释放空间。

SQL> set pages 999 lines 300
SELECT /*+ parallel(8) */ d.tablespace_name "TB Name",d.status "Status",TO_CHAR((a.bytes / 1024 / 1024/1024),'99,999,990.90') "Size (G)",
TO_CHAR((a.bytes - f.bytes) / 1024 / 1024/1024,'99,999,990.90') "Used G", round(((a.bytes - f.bytes) / a.bytes) * 100, 2) "% Used"  FROM sys.dba_tablespaces d,sys.sm$ts_avail a, sys.sm$ts_free f WHERE d.tablespace_name =a.tablespace_name AND f.tablespace_name (+) =d.tablespace_name order by ((a.bytes - f.bytes) / a.bytes) desc;SQL>   2  

TB Name 		       Status	 Size (G)	Used G		   % Used
------------------------------ --------- -------------- -------------- ----------
ZFPT_KF 		       ONLINE	       4,140.61       3,770.38	    91.06
SYSAUX			       ONLINE		  31.65 	 26.50	    83.72
ZFPT_ZW 		       ONLINE		  30.00 	 23.43	    78.11
SYSTEM			       ONLINE		   3.94 	  2.61	    66.22
DUBBO			       ONLINE		   0.10 	  0.01	     9.63
UNDOTBS1		       ONLINE		  68.63 	  5.77	      8.4
ZFPT_CFG		       ONLINE		   3.00 	  0.01	      .37
USERS			       ONLINE		   3.00 	  0.00	      .08
CMS_DATA		       ONLINE		  30.00 	  0.02	      .06
UNDOTBS2		       ONLINE		  94.00 	  0.04	      .04
OGG			       ONLINE		   5.00 	  0.00	      .02

11 rows selected.

SQL> select file_name,file_id,tablespace_name from dba_data_files where tablespace_name='CMS_DATA';

FILE_NAME					      FILE_ID TABLESPACE_NAME
-------------------------------------------------- ---------- ------------------------------
+DATA/hyzf/datafile/cms_data.1353.997352165		    6 CMS_DATA

SQL> select file_name,file_id,tablespace_name,bytes/1024/1024/1024 from dba_data_files where tablespace_name='CMS_DATA';

FILE_NAME					      FILE_ID TABLESPACE_NAME		     BYTES/1024/1024/1024
-------------------------------------------------- ---------- ------------------------------ --------------------
+DATA/hyzf/datafile/cms_data.1353.997352165		    6 CMS_DATA					       30

SQL> alter database datafile 6 resize 1G;

Database altered.

SQL> select file_name,file_id,tablespace_name,bytes/1024/1024/1024 from dba_data_files where tablespace_name='UNDOTBS2';

FILE_NAME					      FILE_ID TABLESPACE_NAME		     BYTES/1024/1024/1024
-------------------------------------------------- ---------- ------------------------------ --------------------
+DATA/hyzf/datafile/undotbs2.265.997301983		    5 UNDOTBS2					       30
+DATA/hyzf/datafile/undotbs2.1369.997353267		   22 UNDOTBS2					       30
+DATA/hyzf/datafile/undotbs2.256.997432009		   24 UNDOTBS2					       30
+DATA/hyzf/datafile/undotbs2.1303.1027270551		   52 UNDOTBS2						1
+DATA/hyzf/datafile/undotbs2.1302.1027270555		   53 UNDOTBS2						1
+DATA/hyzf/datafile/undotbs2.1301.1027270557		   54 UNDOTBS2						1
+DATA/hyzf/datafile/undotbs2.1300.1027270561		   55 UNDOTBS2						1

7 rows selected.

SQL> alter database datafile 5 resize 1G;

Database altered.

SQL> alter database datafile 22 resize 1G;           

Database altered.

SQL> alter database datafile 24 resize 1G;

Database altered.

SQL> select file_name,file_id,tablespace_name,bytes/1024/1024/1024 from dba_data_files where tablespace_name='UNDOTBS1';

FILE_NAME					      FILE_ID TABLESPACE_NAME		     BYTES/1024/1024/1024
-------------------------------------------------- ---------- ------------------------------ --------------------
+DATA/hyzf/datafile/undotbs1.261.997301819		    3 UNDOTBS1					       30
+DATA/hyzf/datafile/undotbs1.1308.1027270475		   47 UNDOTBS1					    7.125
+DATA/hyzf/datafile/undotbs1.1307.1027270509		   48 UNDOTBS1					    7.375
+DATA/hyzf/datafile/undotbs1.1306.1027270513		   49 UNDOTBS1					     7.75
+DATA/hyzf/datafile/undotbs1.1305.1027270515		   50 UNDOTBS1						8
+DATA/hyzf/datafile/undotbs1.1304.1027270523		   51 UNDOTBS1					    8.375

6 rows selected.

SQL> alter database datafile 3 resize 6G;  

Database altered.

此时可以观察到,那两块盘的free_mb,已经不为0了,通过日志,可以看到rebalance可以继续了。

Wed Jan 19 17:35:35 2022
NOTE: ASM did background COD recovery for group 1/0x90482d19 (DATA)
NOTE: starting rebalance of group 1/0x90482d19 (DATA) at power 10
Starting background process ARB0
Wed Jan 19 17:35:38 2022
ARB0 started with pid=34, OS id=20360
NOTE: assigning ARB0 to group 1/0x90482d19 (DATA) with 10 parallel I/Os
cellip.ora not found.
但是过一会,又会有相同的报错。
Wed Jan 19 17:38:23 2022
ERROR: ORA-15041 thrown in ARB0 for group number 1
Errors in file /u01/oracle/app/grid/base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_20360.trc:
ORA-15041: diskgroup “DATA” space exhausted
Wed Jan 19 17:38:23 2022
NOTE: stopping process ARB0
NOTE: rebalance interrupted for group 1/0x90482d19 (DATA)
报错信息:rebalance interrupted。rebalance未完成。
此刻查看那两块盘的free_mb又为0了,难怪rebalance不能完成。
此处就有疑问1 ,按理是扩容,现有磁盘的free_mb,应该增大,新增磁盘的free_mb应该较少才对,。但是我们在观察的过程中,那两块盘的free_mb一直在减小,不知道为什么?
3、看来还是空间不够,需要再释放空间。那么还需要释放多少空间那?此处可以参考mos:Exadata Create TableSpace Fails with ORA-01119 ORA-17502 ORA-15041 Diskgroup Space Exhausted Despite Plenitude of Free Space is Present (Doc ID 2372651.1)。计算大概需要的rebalance空间,但是我们当时没有计算。
现在摆在我们面前的就两条路:一是新建磁盘组,进行diskgroup迁移。但是存储已无多余磁盘来创建新的磁盘组。
二是删除数据文件。删除是不允许的,我们可以把磁盘组里面的数据文件移动到其他位置,如文件系统或者另一个磁盘组。此处刚好另一个磁盘组datadg01还有剩余空间,我们就选择把data里面的数据文件移动部分到datadg01里面,释放空间。
4、既然是这两块盘的使用率很高,剩余空间为0,那么我们就可以针对这两块盘进行数据文件移动,即看哪几个数据文件存储在这两块盘里面,然后又针对性的移动,就会很快释放这两块的空间。
参考mos:ORA-15041 DURING REBALANCE OR ADD DISK WHEN PLENTY OF SPACE EXISTS (Doc ID 473271.1)

select group_kfdat "group #",
number_kfdat "disk #",
count(*) "# AU's"
from x$kfdat a
where v_kfdat = 'V'
and not exists (select *
from x$kfdat b
where a.group_kfdat = b.group_kfdat
and a.number_kfdat = b.number_kfdat
and b.v_kfdat = 'F')
group by GROUP_KFDAT, number_kfdat;SQL>   2    3    4    5    6    7    8    9   10   11  

   group #     disk #	  # AU's
---------- ---------- ----------
	 1	   56	   33152
	 1	   3	   33152
SQL> select name, file_number
from v$asm_alias
where group_number in (select group_kffxp
from x$kffxp
where group_kffxp=1
and disk_kffxp in (3,56)
and au_kffxp != 4294967294
and number_kffxp >= 256)
and file_number in (select number_kffxp
from x$kffxp
where group_kffxp=1
and disk_kffxp in (3,56)
and au_kffxp != 4294967294
and number_kffxp >= 256)
and system_created='Y';  2    3    4    5    6    7    8    9   10   11   12   13   14   15 
NAME							     FILE_NUMBER
------------------------------------------------------------ -----------
UNDOTBS2.256.997432009						     256
group_8.257.997354099						     257
group_7.258.997354099						     258
group_6.259.997354087						     259
SYSAUX.260.997301819						     260
UNDOTBS1.261.997301819						     261
group_4.263.997354391						     263
group_1.264.997354227						     264
UNDOTBS2.265.997301983						     265
group_5.267.997354075						     267
TEMP.268.997301931						     268
group_2.269.997301929						     269
group_3.270.997354229						     270
Current.271.997301927						     271
SYSTEM.272.997301819						     272
ZFPT_KF.1291.1036749283 					    1291
ZFPT_KF.1292.1035909701 					    1292
ZFPT_KF.1293.1032861029 					    1293
ZFPT_KF.1294.1032860827 					    1294
ZFPT_KF.1295.1028968735 					    1295
USERS.1296.1027420239						    1296
SYSTEM.1297.1027420191						    1297
SYSAUX.1298.1027420153						    1298
ZFPT_KF.1299.1027420039 					    1299
UNDOTBS2.1300.1027270561					    1300
UNDOTBS2.1301.1027270557					    1301
UNDOTBS2.1302.1027270555					    1302
UNDOTBS2.1303.1027270551					    1303
UNDOTBS1.1304.1027270523					    1304
UNDOTBS1.1305.1027270515					    1305
UNDOTBS1.1306.1027270513					    1306
UNDOTBS1.1307.1027270509					    1307
UNDOTBS1.1308.1027270475					    1308
ZFPT_KF.1309.1027269461 					    1309
ZFPT_KF.1310.1027269459 					    1310
ZFPT_KF.1311.1027269457 					    1311
ZFPT_KF.1312.1027269455 					    1312
ZFPT_KF.1313.1027269451 					    1313
ZFPT_KF.1314.1027269435 					    1314
ZFPT_KF.1315.1027269433 					    1315
ZFPT_KF.1316.1027269431 					    1316
ZFPT_KF.1317.1027269427 					    1317
ZFPT_KF.1318.1027269425 					    1318
ZFPT_KF.1319.1027269421 					    1319
ZFPT_KF.1320.1027269349 					    1320
ZFPT_KF.1321.1022256547 					    1321
ZFPT_KF.1322.1022256531 					    1322
ZFPT_KF.1323.1022256521 					    1323
ZFPT_KF.1324.1022256491 					    1324
ZFPT_KF.1325.1022256477 					    1325
ZFPT_KF.1326.1022256457 					    1326
ZFPT_KF.1327.1022256437 					    1327
ZFPT_KF.1328.1014300783 					    1328
DUBBO.1329.997640731						    1329
ZFPT_CFG.1330.997466985 					    1330
CMS_DATA.1353.997352165 					    1353
ZFPT_KF.1355.997352277						    1355
ZFPT_KF.1356.997352329						    1356
ZFPT_KF.1357.997352367						    1357
ZFPT_KF.1358.997352403						    1358
ZFPT_KF.1359.997352479						    1359
ZFPT_KF.1360.997352653						    1360
ZFPT_KF.1361.997352725						    1361
ZFPT_KF.1362.997352767						    1362
ZFPT_KF.1363.997352803						    1363
ZFPT_KF.1364.997352843						    1364
ZFPT_KF.1365.997352897						    1365
ZFPT_KF.1366.997352933						    1366
ZFPT_KF.1367.997352969						    1367
ZFPT_KF.1368.997353017						    1368
UNDOTBS2.1369.997353267 					    1369
OGG.1370.997353311						    1370

72 rows selected.

排除redo和undo,我们移动ZFPT_KF对应的数据文件。
5、参考mos:How to move ASM database files from one diskgroup to another ? (Doc ID 330103.1)
我们采用rman的方式移动数据文件。此方法需要停相应的业务。
具体步骤如下:
1、数据库打开为归档模式。
2、offline表空间

SQL> alter tablespace ZFPT_KF offline;

Tablespace altered.

3、备份

[oracle@drrac01 ~]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Wed Jan 19 18:45:32 2022

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: HYZF (DBID=3226644262)
RMAN> backup as copy datafile 8 format '+DATADG01';
RMAN> backup as copy datafile 9 format '+DATADG01';
RMAN> backup as copy datafile 10 format '+DATADG01';
RMAN> backup as copy datafile 11 format '+DATADG01';
RMAN> backup as copy datafile 12 format '+DATADG01';
RMAN> backup as copy datafile 13 format '+DATADG01';
RMAN> backup as copy datafile 14 format '+DATADG01';
RMAN> backup as copy datafile 15 format '+DATADG01';
RMAN> backup as copy datafile 16 format '+DATADG01';
RMAN> backup as copy datafile 17 format '+DATADG01';
RMAN> backup as copy datafile 18 format '+DATADG01';
RMAN> backup as copy datafile 19 format '+DATADG01';
RMAN> backup as copy datafile 20 format '+DATADG01';

4、修改控制文件中数据文件路径

RMAN> switch datafile 8,9,10,11,12,13,14,15,16,17,18,19,20 to copy;

datafile 8 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.376.1094410053"
datafile 9 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.377.1094410133"
datafile 10 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.378.1094410155"
datafile 11 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.379.1094410241"
datafile 12 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.380.1094410257"
datafile 13 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.381.1094410315"
datafile 14 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.382.1094410343"
datafile 15 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.383.1094410377"
datafile 16 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.384.1094410417"
datafile 17 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.385.1094410451"
datafile 18 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.386.1094410493"
datafile 19 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.387.1094410541"
datafile 20 switched to datafile copy "+DATADG01/hyzf/datafile/zfpt_kf.388.1094410559"

SQL> select file_name,file_id,tablespace_name,bytes/1024/1024/1024 from dba_data_files where file_id in(8,9,10,11,12,13,14,15,16,17,18,19,20)

FILE_NAME					      FILE_ID TABLESPACE_NAME		     BYTES/1024/1024/1024
-------------------------------------------------- ---------- ------------------------------ --------------------
+DATADG01/hyzf/datafile/zfpt_kf.376.1094410053		    8 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.377.1094410133		    9 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.378.1094410155		   10 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.379.1094410241		   11 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.380.1094410257		   12 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.381.1094410315		   13 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.382.1094410343		   14 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.383.1094410377		   15 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.384.1094410417		   16 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.385.1094410451		   17 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.386.1094410493		   18 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.387.1094410541		   19 ZFPT_KF					       30
+DATADG01/hyzf/datafile/zfpt_kf.388.1094410559		   20 ZFPT_KF					       30

13 rows selected.

5、online回来表空间

SQL> alter tablespace ZFPT_ZW online;

Tablespace altered.

6、此时rebalance可以正常完成。
Wed Jan 19 19:06:31 2022
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Wed Jan 19 19:06:36 2022
GMON updating for reconfiguration, group 1 at 47 for pid 37, osid 18419
NOTE: group 1 PST updated.
Wed Jan 19 19:06:36 2022
NOTE: membership refresh pending for group 1/0x90482d19 (DATA)
GMON querying group 1 at 48 for pid 18, osid 49741
SUCCESS: refreshed membership for 1/0x90482d19 (DATA)
NOTE: Attempting voting file refresh on diskgroup DATA
NOTE: Refresh completed on diskgroup DATA. No voting file found.
Wed Jan 19 19:36:28 2022
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0x90482d19 (DATA)
7、rebalance完。磁盘使用率还是不一致。
4cd829fa2b79b899dee0d5f31db338f.jpg
image.png
参考mos:ORA-15041 Diskgroup Space Exhausted (Doc ID 1367078.1)
元数据的差异导致使用率不均衡,执行:

alter diskgroup <diskgroup name> check all norepair;
This can be run safely on active system. Once this command completes, check the ASM alert.log if it reports any "mismatch" of AT and DD. Something like

NOTE: disk <diskgroup name>, used AU total mismatch: DD=32169, AT=32172
If yes, then run:

alter diskgroup <diskgroup name> check all repair;
This can be run safely on active system and is meant to fix the AT and DD discrepancy. An AT and DD ASM Metadata discrepancy might manifest because of previous failed file allocation in the diskgroup. After the check all repair command, run a manual rebalance

alter diskgroup <diskgroup name> rebalance power <n>;

曾今有过check all repair导致磁盘组挂掉的经历,所以没有执行,让他继续不均衡着,但是也不打算往这个磁盘组里面添加数据文件了。至此,修复完成。
8、还有人说我们可以drop disk 3号和56号磁盘,让其重新触发rebalance。因为此刻add disk的rebalance被hang,所以此刻,drop disk的rebalance根本无法触发,而且即使触发了,3和56号盘上面有数据,他们的数据rebalance到哪里去哪?很有可能会丢数据。drop那块新添加的磁盘也是一样的道理,上面已经有rebalance的数据了,删除会丢数据。
##总结
本次故障中,有几点值得今后的系统注意:
1、asm磁盘组扩容前,应该仔细检查:
1)是否有offline或者hang状态的磁盘;
2)查询每个磁盘的使用率,是否均衡,是否有free_mb为0的情况。

2.ASM磁盘组中尽量使用大小规格统一的磁盘。
3.ASM扩容注意监控rebalance的结果。
4.ASM使用率的监控,建议深化监控到disk级别。参考:Information to gather when diagnosing ASM space issues (Doc ID 351117.1)

遗留问题

虽然问题解决了,但是有2个疑问:
1、一般都是类似alter tablespace add datafile这种,要在asm磁盘中分配空间的时候,检测到某块盘磁盘耗尽,才会报错。但是我们此次情况是扩容,按理说是把空间耗尽盘的数据挪到新添加的磁盘上,以到达空间均衡。是扩容而不是分配空间,按理说是不会报这种错误的,但是不知道为什么?会报这个错误。但是唯一能确定的是确实有磁盘空间耗尽。
2、rebalance的时候,第一次处理的时候,已经挪出了50G的空闲空间,所有磁盘的剩余空间都有,但是在它第二次rebalance的时候,我们可以明显的看到那两块使用率高的磁盘的剩余空间还在一直不断减少,一直到0,最后报错。按理说,原来的盘在rebalance的时候,会把原来磁盘组中的数据挪到新盘,所有原来磁盘组中的磁盘剩余空间应该不会减少,而是会增大。但是相反。所以最后又剩余了0,报错。但是能确定的是加盘之前,asm中没有其他rebalance的操作。

疑问1参考:mos:ORA-15041 Diskgroup Space Exhausted (Doc ID 1367078.1)可以得出答案。
3] If any one or more disk in the diskgroup has very less space ( say < 300 MB ), then manual rebalance/add of new disks will not help. Because rebalance operation needs some amount of free space in each disk. Also add of a new disk will invoke an automatic rebalance that will also fail with ORA-15041. In this case, you should look to remove/move some files from the diskgroup. Check for obsolete RMAN/dump files, temp files etc. Once we remove/move some files, from the diskgroup, query v$asm_disk.free_mb for each disk. If the free_mb is say > 300 MB, run a manual rebalance ( or add disk )

alter diskgroup rebalance power;
也就是说,当磁盘的free_mb<300MB的时候,那么手动重新平衡/添加新磁盘将无济于事。 因为重新平衡操作需要每个磁盘中的一些可用空间。 此外,添加新磁盘将调用自动重新平衡,该重新平衡也会因 ORA-15041 而失败。
也就是说,不管是给磁盘组扩容,增加空间,还是使用空间,给数据库添加数据文件。当磁盘组中有部分磁盘的free_mb<300MB的时候,rebalance就会报错(重新平衡操作需要每个磁盘中的一些可用空间),就会报错ORA-15041。

参考

ORA-15041 IN A DISKGROUP ALTHOUGH FREE_MB REPORTS SUFFICIENT SPACE (Doc ID 460155.1)
ORA-15041 DURING REBALANCE OR ADD DISK WHEN PLENTY OF SPACE EXISTS (Doc ID 473271.1)
Information to gather when diagnosing ASM space issues (Doc ID 351117.1)

  • 2
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值