概要:
一客户asm diskheader corrution,导致4T数据量的数据库无法启动,无备份,重新构造数据周期太长,无法接受。
最后通过kfed工具修改磁盘头来解决该故障,本文详细的记录了处理过程。
故障现象:
Tue Jun 25 10:14:59 2013
SQL> alter diskgroup HXCXDATA mount force
NOTE: cache registered group HXCXDATA number=12 incarn=0x0d73f50d
NOTE: cache began mount (first) of group HXCXDATA number=12 incarn=0x0d73f50d
NOTE: Assigning number (12,1) to disk (/dev/sddlmbb)
NOTE: Assigning number (12,2) to disk (/dev/sddlmbc)
NOTE: Assigning number (12,9) to disk (/dev/sddlmbj)
NOTE: Assigning number (12,8) to disk (/dev/sddlmbi)
NOTE: Assigning number (12,7) to disk (/dev/sddlmbh)
NOTE: Assigning number (12,10) to disk (/dev/sddlmbk)
NOTE: Assigning number (12,6) to disk (/dev/sddlmbg)
NOTE: Assigning number (12,5) to disk (/dev/sddlmbf)
Tue Jun 25 10:15:00 2013
ERROR: no read quorum in group: required 1, found 0 disks
NOTE: cache dismounting (not clean) group 12/0x0D73F50D (HXCXDATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 53483, image: oracle@zjsgdbnfwbsk01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 12/0x0D73F50D (HXCXDATA)
NOTE: cache ending mount (fail) of group HXCXDATA number=12 incarn=0x0d73f50d
NOTE: cache deleting context for group HXCXDATA 12/0x0d73f50d
GMON dismounting group 12 at 54 for pid 42, osid 53483
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
NOTE: Disk in mode 0x8 marked for de-assignment
ERROR: diskgroup HXCXDATA was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "HXCXDATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "HXCXDATA"
ERROR: alter diskgroup HXCXDATA mount force
故障分析:
检查集群资源状态:
zjsgdbnfwbsk01:/home/grid$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.HXCXDATA.dg
ONLINE OFFLINE zjsgdbnfwbsk01
ONLINE OFFLINE zjsgdbnfwbsk02
ora.HXCXFRA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.LISTENER.lsnr
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.NFJQDATA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.NFJQFRA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.NFSCDATA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.NFSCFRA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.OCR_VOTE.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.SKYDATA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.WBDATA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.WBDSDATA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.WBDSFRA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE ONLINE zjsgdbnfwbsk02
ora.WBFRA.dg
ONLINE ONLINE zjsgdbnfwbsk01
ONLINE