谈谈Oracle ASM的限制

今天在客户现场安装Oracle rac时，创建ASM DISKGROUP时出现问题，以下为解决过程。记录之，以备忘。
客户操作系统版本为AIX 6105，Oracle crs版本为10.2.0.5,rdbms和asm版本为10.2.0.1。
当用dbca工具建立磁盘组时，检索不出备选磁盘。在确定rdisk权限等没问题之后，在启动ASM实例之后，尝试在1号节点手动建立磁盘组，不料建磁盘组时出现以下错误。
[quote]$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Mar 31 13:44:37 2011

Copyright (c) 1982, 2005, Oracle. All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> create diskgroup data1 external redundancy disk '/dev/rhdisk7' size 2047G;
create diskgroup data1 external redundancy disk '/dev/rhdisk7' size 2047G
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
IBM AIX RISC System/6000 Error: 6:[color=red] No such device or address[/color]
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error:[/quote]
红色标注可以看到，Oracle识别不了磁盘rhdisk7，但实际上rhdisk7确实存在。由于时间紧张，上google搜索，在一份otn的帖子中，一老外同行也碰到过类似问题，后来在解决问题描述时指出：
[quote]Hi All,
I have resolved the issue now.
Since we are using SAN disks with raid 5. The disks size visible to ASm was not correct one.
So what i did is that I added SIZE clause during asm diskgroup creation.
create diskgroup oradata external redundany disk '/dev/rhdisk4' size 1000G;
by issuing this statement now I can create diskgroups.
Remember max size of a SAN disk for ASM is 2TB. We can't go beyond 2TB[/quote]
询问客户之后，发现本案例中也是raid5 而且rdisk也超过了2T，于是按照网友指出的方法进行创磁盘组尝试。幸好创建成功！
[quote]SQL> create diskgroup data external redundancy disk '/dev/rhdisk7' size 1990G;

Diskgroup created.[/quote]
创建成功之后，碰到了ASM重启之后，识别不了磁盘组错误：
[quote]SQL> startup
ASM instance started

Total System Global Area 130023424 bytes
Fixed Size 2019136 bytes
Variable Size 102838464 bytes
ASM Cache 25165824 bytes
ORA-15110: no diskgroups mounted[/quote]
或者
[quote]SQL> startup
ASM instance started

Total System Global Area 130023424 bytes
Fixed Size 2019136 bytes
Variable Size 102838464 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup
"DATA,DATA1"[/quote]
解决办法为：
在asm参数文件中显示指明该asm需要mount的磁盘组
[quote]asm_diskgroups='DATA','ARCH'
ASM_DISKSTRING ='/dev/rhdisk7','/dev/rhdisk8'[/quote]
为什么ASM不能创建超过2T的硬盘呢？上metalink一查，Oracle针对这一现象，解释为bug：Bug 6453944 - ORA-15196 with ASM disks larger than 2TB [ID 6453944.8]
[quote]ORA-15196 (ASM block corruption) can occur, if LUNs larger than 2TB are
presented to an ASM diskgroup.

As a result of the fix, ORA-15099 will be raised if larger than 2TB size
disk is specified. This is irrespective of the presence of asmlib.

In the future, this limitation will be removed.

Workaround:
Do not add more than 2 TB size disk to a diskgroup.[/quote]
此外metalink 736891.1对此bug引起的后果有较为详细的描述，其中最直接的后果就是损坏ASM元数据（metadata）：
[quote]There are no clear indications of what can go wrong with ASM LUNs larger than 2TB – but one example is wrapping: when Oracle gets to the end of 2TB, it may wrap back to the start of the LUN and [color=red]overwrite the ASM metadata[/color][/quote]，并导致如下错误发生：
[quote]ORA-15196 : invalid ASM block header [kfc.c:7997] [hard_kfbh] [4] [4] [34 != 130][/quote]
当然这一bug已经在以下版本修复：
[quote] 11.2
10.2.0.4 (Server Patch Set)
11.1.0.7 (Server Patch Set) [/quote]
那么除此之外，ASM还有什么限制呢?(metalink:ID 370921.1).我想除了2TB限制，其他限制达到的可能性还是比较低的。

[quote]63 disk groups in a storage system
10,000 ASM disks in a storage system
[color=red]2 terabyte maximum storage for each ASM disk[/color] (the Bug 6453944 allowed larger sizes, but that led to problems, see Note 736891.1 "ORA-15196 WITH ASM DISKS LARGER THAN 2TB")
40 exabyte maximum storage for each storage system
1 million files for each disk group
2.4 terabyte maximum storage for each file[/quote]