恢复votedisk报错CRS-4000: Command Replace failed, or completed with errors.

恢复votedisk报错如下：

[root@rac1 bin]# ./crsctl replace votedisk +griddata

Failed to create voting files on disk group griddata.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.

原因是：
     As per our standard architecture +ASM  Votefile should always have odd numbers other than External redundancy(which requires only one disk).  
A  normal redundancy ASM disk group, which is supposed to hold Voting Files, requires 3 disks in separate failure groups, while a normal redundancy ASM disk group that is not used to store Voting Files requires only 2 disks in separate failure groups.

SOLUTION
Add  one more disk/failgroup to our existing Normal redundancy diskgroup +NRML  as below and then need to replace votedisk:-
SQL> alter diskgroup NRML add disk '/dev/asm-dsikf';
 Diskgroup altered.
[root@rac1 bin]# ./crsctl replace votedisk +griddata
Successful addition of voting disk 49bcc0c86e744f39bfb02fe6599a5bfe.
Successful addition of voting disk 3d82fd0ca6fa4f22bf7af3aacf35c470.
Successful addition of voting disk 32d24628e1a44fffbf0103dcb35a187c.
Successfully replaced voting disk group with +griddata.
CRS-4266: Voting file(s) successfully replaced

[root@rac1 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   49bcc0c86e744f39bfb02fe6599a5bfe (/dev/asm-diskb) [GRIDDATA]
 2. ONLINE   3d82fd0ca6fa4f22bf7af3aacf35c470 (/dev/asm-diske) [GRIDDATA]
 3. ONLINE   32d24628e1a44fffbf0103dcb35a187c (/dev/asm-diskf) [GRIDDATA]

 ASM alert logs有类似的错误
================
NOTE: Voting File refresh pending for group 4/0x555886a9 (NRML)
NOTE: Attempting voting file creation in diskgroup NRML
NOTE: voting file allocation on grp 4 disk NRML_0000
NOTE: voting file allocation on grp 4 disk NRML_0001
ERROR: Voting file allocation failed for group NRML
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_20379.trc:
ORA-15274: Not enough failgroups (3) to create voting files
NOTE: Attempting voting file refresh on diskgroup NRML
NOTE: Refresh completed on diskgroup NRML. No voting file found.
Fri Nov 01 20:54:28 2013