法定容错组，和它失去联系也不影响集群运行
参考：
How to Manually Add NFS voting disk to an Extended Cluster using ASM in 11.2 (Doc ID 1421588.1)
Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)
RAC: Frequently Asked Questions [ID 220970.1]
***********************************************************
什么是Quorum FailGroup
***********************************************************
Quorum FailGroup中，只保存Voting Disk，用于RAC on Extended Distance Clusters，做仲裁盘用
要放在其他存储节点上或者通过NFS共享一个zero-padded文件作为voting disk
对于使用2个阵列提供保护的环境中，也可以使用Quorum FG
假设使用high冗余，一共5个投票盘。那么5个投票盘在2个阵列上的数目肯定不同
假设一个阵列上有3个，另一个阵列上有2个，如果存在3个投票盘的阵列宕机，那么将只有2个投票盘可用
这种情况下，集群的所有节点将宕机
这种情况下，必须在第三个地点放置一个投票盘，在2个阵列上各有2个投票盘。这样任意一个阵列宕机，还可以保证有3个投票盘是可用的
***********************************************************
如何添加Quorum FailGroup
***********************************************************
一般有3个阵列的情况不多，下面测试通过NFS来配置
1.当前情况：
HIGH冗余，5个投票盘
[oracle@database2 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7fc99b8d4dc54f27bf967f23524a19e3 (/dev/asm-crs1) [CRSDG]
2. ONLINE 2ffd39d609bc4f71bf2b19de2e71c7a8 (/dev/asm-crs2) [CRSDG]
3. ONLINE 002eb188e9b14ffbbf4d5a607ade51c2 (/dev/asm-crs3) [CRSDG]
4. ONLINE 2319348cf3cc4f6abf116f973d8fd922 (/dev/asm-crs4) [CRSDG]
5. ONLINE 6c3cb875ba7e4fe2bffe97189e2bae25 (/dev/asm-crs5) [CRSDG]
Located 5 voting disk(s).
SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /dev/asm-crs05 CRSDG_0004 NORMAL
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.配置NFS
2.1 NFS服务器端
[root@dm01db01 /]# cat /etc/exports
/oracle/votedisk 192.168.123.31(rw,sync,no_root_squash)
/oracle/votedisk 192.168.123.32(rw,sync,no_root_squash)
[root@dm01db01 /]# /etc/rc.d/init.d/portmap start
Starting portmap: [ OK ]
[root@dm01db01 /]# /etc/rc.d/init.d/nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
Stopping RPC idmapd: [ OK ]
Starting RPC idmapd: [ OK ]
2.2 数据库服务器端MOUNT文件系统
# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048
# chown -R grid.oinstall /u01/app/oracle/votedisk
2.3 添加 quorum fg disk
SQL> alter system set asm_diskstring="/dev/asm*","/u01/app/oracle/votedisk/vote*" sid='*' scope=both;
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01';
SQL> select GROUP_NUMBER,DISK_NUMBER,OS_MB,NAME,PATH,FAILGROUP,state from v$asm_disk order by 1,2;
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /dev/asm-crs05 CRSDG_0004 NORMAL
1 5 2048 CRSDG_0005 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.4 删除多余的磁盘
在quorum fg添加后，可以看到还没有votedisk切换到其上
这时，我们将删除有3个投票盘阵列上的一个投票盘
[grid@database1 votedisk]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 83fa917a0e844f23bf27238aff51b57a (/dev/asm-crs05) [CRSDG]
5. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
SQL> alter diskgroup crsdg drop disk CRSDG_0004; <==
[grid@database1 votedisk]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
5. ONLINE 6fc616a6923a4fb2bffca18e44a58533 (/u01/app/oracle/votedisk/votedisk01) [CRSDG] <==
***********************************************************
相关配置
***********************************************************
对于数据库DiskGroup，创建时应该为不同阵列的盘，指定好FAILGROUP，以确保2份数据（NORMAL冗余）放在不同的阵列中
修改参数asm_preferred_read_failure_groups，不同的节点，从不同的FG组中读取数据
***********************************************************
测试
***********************************************************
1.NFS服务器宕机
2014-02-13 16:56:38.476:
[cssd(3577)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 99910 milliseconds
2014-02-13 16:57:28.541:
[cssd(3577)]CRS-1614:No I/O has completed after 75% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 49850 milliseconds
2014-02-13 16:57:58.565:
[cssd(3577)]CRS-1613:No I/O has completed after 90% of the maximum interval. Voting file /u01/app/oracle/votedisk/votedisk01 will be considered not functional in 19830 milliseconds
2014-02-13 16:58:18.573:
[cssd(3577)]CRS-1604:CSSD voting file is offline: /u01/app/oracle/votedisk/votedisk01; details at (:CSSNM00058:) in /u01/app/11.2.0/grid/log/database1/cssd/ocssd.log.
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).
集群件运行正常此时 ,互相之间可用沟通
恢复
[root@database1 ~]# umount -f /u01/app/oracle/votedisk
[root@database1 ~]# mount -t nfs -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0 192.168.123.139:/oracle/votedisk /u01/app/oracle/votedisk
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ ------------------------------ ------------------------------ --------
0 5 2048 /u01/app/oracle/votedisk/voted NORMAL <== 文件此时并不属于CRSDG
isk01
0 9 2048 /dev/asm-crs05 NORMAL
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 0 CRSDG_0004 FGQ NORMAL <==
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
[root@database1 votedisk]# dd if=/dev/zero of=/u01/app/oracle/votedisk/votedisk01 bs=1M count=2048 <== 重新格式化Voting disk
SQL> alter diskgroup CRSDG add quorum failgroup FGQ DISK '/u01/app/oracle/votedisk/votedisk01'; <== 重新添加
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
0 4 2048 /dev/asm-crs05 NORMAL
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
1 5 0 CRSDG_0005 FGQ NORMAL <==
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
SQL> alter diskgroup crsdg drop quorum disk CRSDG_0005 force; <== 将旧的删除
GROUP_NUMBER DISK_NUMBER OS_MB NAME PATH FAILGROUP STATE
------------ ----------- ---------- ------------------------------ --------------- ------------------------------ --------
0 4 2048 /dev/asm-crs05 NORMAL <==
1 0 2048 CRSDG_0000 /dev/asm-crs01 CRSDG_0000 NORMAL
1 1 2048 CRSDG_0001 /dev/asm-crs02 CRSDG_0001 NORMAL
1 2 2048 CRSDG_0002 /dev/asm-crs03 CRSDG_0002 NORMAL
1 3 2048 CRSDG_0003 /dev/asm-crs04 CRSDG_0003 NORMAL
1 4 2048 CRSDG_0004 /u01/app/oracle FGQ NORMAL <==
/votedisk/voted
isk01
2 0 20480 DATADG_0000 /dev/asm-data1 DATADG_0000 NORMAL
2 1 20480 DATADG_0001 /dev/asm-data2 DATADG_0001 NORMAL
2.先于NFS服务器启动GRID
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
5. OFFLINE ad0268dcec654f14bfe60497d9490780 () []
Located 5 voting disk(s).
[root@database1 ~]# crsctl stat res -t <== 无法启动
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[root@database2 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE INTERMEDIATE database2 OCR not started <== ASM启动，但是CRSDG无法MOUNT
ora.cluster_interconnect.haip
1 ONLINE ONLINE database2
ora.crf
1 ONLINE ONLINE database2
ora.crsd
1 ONLINE OFFLINE
ora.cssd
1 ONLINE ONLINE database2
ora.cssdmonitor
1 ONLINE ONLINE database2
ora.ctssd
1 ONLINE ONLINE database2 ACTIVE:0
ora.diskmon
1 OFFLINE OFFLINE
ora.drivers.acfs
1 ONLINE ONLINE database2
ora.evmd
1 ONLINE INTERMEDIATE database2
ora.gipcd
1 ONLINE ONLINE database2
ora.gpnpd
1 ONLINE ONLINE database2
ora.mdnsd
1 ONLINE ONLINE database2
SQL> alter diskgroup crsdg mount;
alter diskgroup crsdg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "4" is missing from group number "1"
SQL> alter diskgroup crsdg mount force; <== 强制MOUNT
Diskgroup altered.
[root@database1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.DATADG.dg
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER_1522.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.LISTENER_1523.lsnr
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.asm
ONLINE ONLINE database1 Started
ONLINE ONLINE database2 Started
ora.gsd
OFFLINE OFFLINE database1
OFFLINE OFFLINE database2
ora.net1.network
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.ons
ONLINE ONLINE database1
ONLINE ONLINE database2
ora.registry.acfs
ONLINE ONLINE database1
ONLINE ONLINE database2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE database1
ora.cvu
1 OFFLINE OFFLINE
ora.database1.vip
1 ONLINE ONLINE database1
ora.database2.vip
1 ONLINE ONLINE database2
ora.oc4j
1 ONLINE ONLINE database1
ora.orcl.billa3.svc
1 ONLINE ONLINE database1
ora.orcl.db
1 ONLINE ONLINE database1 Open
2 ONLINE ONLINE database2 Open
ora.scan1.vip
1 ONLINE ONLINE database1
[root@database1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 46bf9f1d43574f5bbfefe1377db152c4 (/dev/asm-crs01) [CRSDG]
2. ONLINE 10892f57e2e84ffabfa4a5e6fa86aee5 (/dev/asm-crs02) [CRSDG]
3. ONLINE 0a66ffb250394f13bfcfdb1946056058 (/dev/asm-crs03) [CRSDG]
4. ONLINE 18e561a6a5ff4fc0bf996f740aff70da (/dev/asm-crs04) [CRSDG]
Located 4 voting disk(s).
恢复和测试1相同
617
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
