ASM diskheader corruption

概要：

一客户asm diskheader corrution，导致4T数据量的数据库无法启动，无备份，重新构造数据周期太长，无法接受。

最后通过kfed工具修改磁盘头来解决该故障，本文详细的记录了处理过程。

 

故障现象：

 

Tue Jun 25 10:14:59 2013
SQL>  alter diskgroup HXCXDATA mount force 
NOTE: cache registered group HXCXDATA number=12 incarn=0x0d73f50d
NOTE: cache began mount (first) of group HXCXDATA number=12 incarn=0x0d73f50d
NOTE: Assigning number (12,1) to disk (/dev/sddlmbb)
NOTE: Assigning number (12,2) to disk (/dev/sddlmbc)
NOTE: Assigning number (12,9) to disk (/dev/sddlmbj)
NOTE: Assigning number (12,8) to disk (/dev/sddlmbi)
NOTE: Assigning number (12,7) to disk (/dev/sddlmbh)
NOTE: Assigning number (12,10) to disk (/dev/sddlmbk)
NOTE: Assigning number (12,6) to disk (/dev/sddlmbg)
NOTE: Assigning number (12,5) to disk (/dev/sddlmbf)
Tue Jun 25 10:15:00 2013
ERROR: no read quorum in group: required 1, found 0 disks
NOTE: cache dismounting (not clean) group 12/0x0D73F50D (HXCXDATA) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 53483, image: oracle@zjsgdbnfwbsk01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 12/0x0D73F50D (HXCXDATA) 
NOTE: cache ending mount (fail) of group HXCXDATA number=12 incarn=0x0d73f50d
NOTE: cache deleting context for group HXCXDATA 12/0x0d73f50d
GMON dismounting group 12 at 54 for pid 42, osid 53483
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
 

ERROR: diskgroup HXCXDATA was not mounted

ORA-15032: not all alterations performed

ORA-15017: diskgroup "HXCXDATA" cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup "HXCXDATA"

ERROR:  alter diskgroup HXCXDATA mount force

 

故障分析：

检查集群资源状态：

 

zjsgdbnfwbsk01:/home/grid$crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.HXCXDATA.dg
               ONLINE  OFFLINE      zjsgdbnfwbsk01                               
               ONLINE  OFFLINE      zjsgdbnfwbsk02                               
ora.HXCXFRA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.LISTENER.lsnr
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.NFJQDATA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.NFJQFRA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.NFSCDATA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.NFSCFRA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.OCR_VOTE.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.SKYDATA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.WBDATA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.WBDSDATA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.WBDSFRA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE  ONLINE       zjsgdbnfwbsk02                               
ora.WBFRA.dg
               ONLINE  ONLINE       zjsgdbnfwbsk01                               
               ONLINE