Demystifying ASM REQUIRED_MIRROR_FREE_MB
and USABLE_FILE_MB
Posted by Harald van Breederode on January 3, 2013
A recurring question in my classes is how Oracle Automatic Storage Management (ASM) calculates theREQUIRED_MIRROR_FREE_MB
and USABLE_FILE_MB
disk group values. As usual the answer is: It depends! ;-)
In short: the REQUIRED_MIRROR_FREE_MB
value in V$ASM_DISKGROUP
indicates how much free space is required in an ASM disk group to restore redundancy after the failure of an ASM disk or ASM failure group. TheUSABLE_FILE_MB
value indicates how much space is available in an ASM disk group considering the redundancy level of the disk group. The question is: How does ASM calculate these values?
The answer can be found by creating ASM disk groups in different configurations and take a look at theREQUIRED_MIRROR_FREE_MB
and USABLE_FILE_MB
values in V$ASM_DISKGROUP
. All we need are a bunch of small disks.
ASM> select label,os_mb from v$asm_disk
2 where label like 'SAN%' order by label;
LABEL OS_MB
------------------------------- ----------
SAN01 255
SAN02 255
SAN03 255
SAN04 255
SAN05 255
SAN06 255
|
REQUIRED_MIRROR_FREE_MB
and USABLE_FILE_MB
in external redundancy disk groups
Let’s get started by creating an external redundancy disk group on top of the above 6 disks:
ASM> create diskgroup demo external redundancy
2 disk 'ORCL:san01'
3 disk 'ORCL:san02'
4 disk 'ORCL:san03'
5 disk 'ORCL:san04'
6 disk 'ORCL:san05'
7 disk 'ORCL:san06'
8 attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
|
Now we can explore this disk group by querying several columns from V$ASM_DISKGROUP
:
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED EXTERN 1530 1468 0 1468
|
The above output shows that this disk group size is 1530MB (TOTAL_MB
) of which 1468MB (FREE_MB
)is free space. Because this is an external redundancy disk group, all free space is available for storing files as indicated by theUSABLE_FILE_MB
column. The REQUIRED_MIRROR_FREE_MB
column is zero because ASM does not mirror user data in external redundancy disk groups.
Note: The “missing” 62MB is used for storing ASM metadata.
REQUIRED_MIRROR_FREE_MB
and USABLE_FILE_MB
in normal redundancy disk groups
Next we create a normal redundancy disk group with six failure groups of one disk each.
ASM> create diskgroup demo normal redundancy
2 failgroup FG1 disk
3 'ORCL:san01'
4 failgroup FG2 disk
5 'ORCL:san02'
6 failgroup FG3 disk
7 'ORCL:san03'
8 failgroup FG4 disk
9 'ORCL:san04'
10 failgroup FG5 disk
11 'ORCL:san05'
12 failgroup FG6 disk
13 'ORCL:san06'
14 attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
|
ASM uses failure groups to provide redundancy inside disk groups. Data stored in one failure group is also stored in another failure group to protect it against failures. In a normal redundancy disk group the loss of one failure group can be tolerated without affecting data availability. ASM will restore redundancy by re-creating the lost data in another failure group. But this requires that a certain amount of disk group space is available within the same disk group to store this data. Because of this, it is useful to know how big a single failure group actually is.
ASM> select failgroup,sum(total_mb) from v$asm_disk
2 where label like 'SAN%' group by failgroup order by failgroup;
FAILGROUP SUM(TOTAL_MB)
------------------------------ -------------
FG1 255
FG2 255
FG3 255
FG4 255
FG5 255
FG6 255
|
The above output tells us that the size of a single failure group is 255MB. Let’s see how this influences the free space requirements and how it affects usable file space:
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED NORMAL 1530 1365 255 555
|
The above output indicates that there should be 255MB free space available in order to allow ASM to restore redundancy after a failure. The value is 255 because it is the size of one failure group. In case failure groups are not equally sized, ASM automatically selects the biggest failure group for this calculation.
The above output also indicates that we can use 555MB for storing files. This value is calculated as follows:
ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable
2 from v$asm_diskgroup where name='DEMO';
USEABLE
----------
555
|
In plain English: The amount of usable file space is equal to the amount of free disk space subtracted by the amount of disk space that is required to restore redundancy after a failure divided by the redundancy level of the disk group which is two in this case.
Thus based on the amount of free space, the amount of required free space and the redundancy level ASM calculates the amount of space available for storing files. If the amount of free space changes, because a file gets created, resized or deleted, this value will also change. This can be demonstrated by creating a data file in this disk group.
SQL> create tablespace foo datafile '+DEMO' size 200m;
Tablespace created.
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED NORMAL 1530 939 255 342
|
Because we created a 200MB data file the amount of usable file space is reduced as shown above. The reduction is slightly more than the file size because ASM allocated a bit more space for its metadata. Let’s add another file to our tablespace:
SQL> alter tablespace foo add datafile '+DEMO' size 200m;
Tablespace altered.
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED NORMAL 1530 534 255 139
|
As expected the amount of usable file space is reduced by slightly more than the actual file size. Can we create another 200MB data file?
SQL> alter tablespace foo add datafile '+DEMO' size 200m;
Tablespace altered.
|
We could indeed create another 200MB data file despite the fact that ASM indicated that there was only 139MB usable file space available. It is important to note that ASM does not enforce the amount of free space indicated byREQUIRED_MIRROR_FREE_MB
. Let’s take a look at the value of USABLE_FILE_MB
now:
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED NORMAL 1530 129 255 -63
|
The value has turned negative! This means that if we encounter a failure ASM will be unable to restore redundancy because there is not enough space to do so. It is very important thatUSABLE_FILE_MB
is monitored! If it becomes negative for a particular disk group, you should either increase the size of the disk group or free up some space in it. For now we will just drop the tablespace ;-)
SQL> drop tablespace foo;
Tablespace dropped.
|
Now let’s create another normal redundancy disk group, but this time with only three failure groups each with two disks instead of six failure groups each with only one disk:
ASM> create diskgroup demo normal redundancy
2 failgroup FG1 disk
3 'ORCL:san01',
4 'ORCL:san02'
5 failgroup FG2 disk
6 'ORCL:san03',
7 'ORCL:san04'
8 failgroup FG3 disk
9 'ORCL:san05',
10 'ORCL:san06'
11 attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
|
Now that the disk group is created we again take a look at the sizes of its failure groups:
ASM> select failgroup,sum(total_mb) from v$asm_disk
2 where label like 'SAN%' group by failgroup order by failgroup;
FAILGROUP SUM(TOTAL_MB)
------------------------------ -------------
FG1 510
FG2 510
FG3 510
|
All three failure groups are equally sized at 510MB. Let’s see how this is reflected in the space administration columns inV$ASM_DISKGROUP
:
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED NORMAL 1530 1365 510 427
|
As expected REQUIRED_MIRROR_FREE_MB
is now 510 which is of course equal to the size of the biggest failure group. This results in 427MB of usable file space. As shown before this value is calculated as follows:
ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable
2 from v$asm_diskgroup where name='DEMO';
USEABLE
----------
427
|
Next we create yet another normal redundancy disk group, but this time with only two failure groups each with three disks instead of three failure groups each with two disks:
ASM> create diskgroup demo normal redundancy
2 failgroup FG1 disk
3 'ORCL:san01',
4 'ORCL:san02',
5 'ORCL:san03'
6 failgroup FG2 disk
7 'ORCL:san04',
8 'ORCL:san05',
9 'ORCL:san06'
10 attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
|
Again we determine the size of all failure groups:
ASM> select failgroup,sum(total_mb) from v$asm_disk
2 where label like 'SAN%' group by failgroup order by failgroup;
FAILGROUP SUM(TOTAL_MB)
------------------------------ -------------
FG1 765
FG2 765
|
Both failure groups are equally sized at 765MB. What will the impact be on the space administration?
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED NORMAL 1530 1416 255 580
|
Hmmm, REQUIRED_MIRROR_FREE_MB
is only 255 (the size of one disk) instead of 765 (the size of one failure group) as one might expect. Apparently ASM only anticipates on the failure of a single disk and no longer anticipates on the failure of a complete failure group.
When you think about this it actually makes sense! As stated at the beginning of this posting, ASM performs mirroring by storing the same data in different failure groups. A normal redundancy disk group requires at least two failure groups and a high redundancy disk group requires at least three failure groups. How can ASM restore redundancy in a normal redundancy disk group when there is only one failure group left? It can’t! Therefore ASM anticipates only on the loss of a single disk.
For completeness, this is how ASM calculated the USABLE_FILE_MB
value of 580:
ASM> select trunc((free_mb - required_mirror_free_mb) / 2) as useable
2 from v$asm_diskgroup where name='DEMO';
USEABLE
----------
580
|
REQUIRED_MIRROR_FREE_MB
and USABLE_FILE_MB
in high redundancy disk groups
Now that we covered external and normal redundancy disk groups, we will take a look at high redundancy disk groups. So, let’s get started by creating a high redundancy disk group with six failure groups of one disk each and determine the size of its failure groups:
ASM> create diskgroup demo high redundancy
2 failgroup FG1 disk
3 'ORCL:san01'
4 failgroup FG2 disk
5 'ORCL:san02'
6 failgroup FG3 disk
7 'ORCL:san03'
8 failgroup FG4 disk
9 'ORCL:san04'
10 failgroup FG5 disk
11 'ORCL:san05'
12 failgroup FG6 disk
13 'ORCL:san06'
14 attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
ASM> select failgroup,sum(total_mb) from v$asm_disk
2 where label like 'SAN%' group by failgroup order by failgroup;
FAILGROUP SUM(TOTAL_MB)
------------------------------ -------------
FG1 255
FG2 255
FG3 255
FG4 255
FG5 255
FG6 255
|
As expected all failure groups are equally sized at 255MB. Could there be a difference between the required free and usable disk space for a high redundancy disk group and a normal redundancy disk group with the same number of disks and failure groups? Let’s have a look:
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED HIGH 1530 1365 510 285
|
REQUIRED_MIRROR_FREE_MB
is 510, which is equal to the sum of the size of two failure groups (or the sum of the sizes of the two biggest failure groups if they are not equally sized). This is because a high redundancy disk group can tolerate the loss of two failure groups without affecting data availability.
USABLE_FILE_MB
is 285 and is calculated as follows:
ASM> select trunc((free_mb - required_mirror_free_mb) / 3) as useable
2 from v$asm_diskgroup where name='DEMO';
USEABLE
----------
285
|
Finally we create another high redundancy disk group but this time with three failure groups each with two disks instead of six failure groups of one disk each, followed by querying the failure group sizes:
ASM> create diskgroup demo high redundancy
2 failgroup FG1 disk
3 'ORCL:san01',
4 'ORCL:san02'
5 failgroup FG2 disk
6 'ORCL:san03',
7 'ORCL:san04'
8 failgroup FG3 disk
9 'ORCL:san05',
10 'ORCL:san06'
11 attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup created.
ASM> select failgroup,sum(total_mb) from v$asm_disk
2 where label like 'SAN%' group by failgroup order by failgroup;
FAILGROUP SUM(TOTAL_MB)
------------------------------ -------------
FG1 510
FG2 510
FG3 510
|
Hopefully the above output isn’t surprising because we have created quite a number of disk groups already on the disks used to write this posting. If you managed to follow and understand everything so far the next output should match your expectations:
ASM> select name, state, type, total_mb, free_mb, required_mirror_free_mb req_free, usable_file_mb use_mb
2 from v$asm_diskgroup where name = 'DEMO';
NAME STATE TYPE TOTAL_MB FREE_MB REQ_FREE USE_MB
---------- ----------- ------ ---------- ------- -------- ------
DEMO MOUNTED HIGH 1530 1365 510 285
|
The 510 listed for REQUIRED_MIRROR_FREE_MB
is the size of two disks and not the size of a single failure group. The reason for this is almost the same as for a normal redundancy disk group with only two failure groups, but this time we have a high redundancy disk group with only three failure groups. If one or two failure groups are lost, ASM cannot restore redundancy as that would require at least three working failure groups.
Below is the calculation for the 285 listed as USABLE_FILE_MB
:
ASM> select trunc((free_mb - required_mirror_free_mb) / 3) as useable
2 from v$asm_diskgroup where name='DEMO';
USEABLE
----------
285
|
In summary
REQUIRED_MIRROR_FREE_MB
is the amount of free space required to restore redundancy after a failure that ASM can tolerate without affecting data availability. This amount depends on the redundancy level and the number of failure groups in the disk group.
Normal redundancy disk groups with at least three failure groups require an amount of free space that is equal to the size of a single failure group. Normal redundancy disk groups with only two failure groups require an amount of free space that is equal to the size of a single disk.
High redundancy disk groups with at least four failure groups require an amount of free space that is equal to the sum of the size of two failure groups. High redundancy disk groups with only three failure groups require an amount of free space that is equal to the sum of the size of two disks.
USABLE_FILE_MB
is the amount of disk space available for storing user data. This amount depends on the total size of the disk group, the disk group redundancy and the amount of required free disk space that ASM needs to restore redundancy after a failure. USABLE_FILE_MB
is calculated as follows:.
USABLE_FILE_MB
= (FREE_MB
– REQUIRED_MIRROR_FREE_MB
) / [2|3]
-Harald