【Oracle ASM数据恢复】ORA-15038: disk ‘XXXXXXX’ mismatch on ‘Time Stamp’ with Target Disk Group错误解析

如果在mount diskgroup时遇到如下错误，则你有必要读一下这篇文章：

 

SQL> alter diskgroup DATA mount;
alter diskgroup DATA mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "17" is missing from group number "1"
ORA-15042: ASM disk "16" is missing from group number "1"
ORA-15042: ASM disk "15" is missing from group number "1"
ORA-15042: ASM disk "14" is missing from group number "1"
ORA-15042: ASM disk "13" is missing from group number "1"
ORA-15038: disk 'ORCL:DATA25' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA24' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA23' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA22' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]
ORA-15038: disk 'ORCL:DATA21' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156]

[oracle@mlab2 ~]$ oerr ora 15042
15042, 00000, "ASM disk \"%s\" is missing from group number \"%s\" "
// *Cause:  The specified disk, which is a necessary part of a diskgroup,
//          could not be found on the system.
// *Action: Check the hardware configuration.
//
[oracle@mlab2 ~]$ oerr ora 15038
15038, 00000, "disk '%s' mismatch on '%s' with target disk group [%s] [%s]"
// *Cause:  An attempt was made to mount into a disk group a disk whose
//          recorded allocation unit size, metadata block size, physical 
//          sector size, or creation time stamp was inconsistent with the other 
//          disk group members.
// *Action: Check if the system configuration has changed. Verify disk 
//          discovery string. 
//

 

 

 

该错误一般是由于OS级别的多路径配置存在问题，导致某个节点上出现大量重复的存储设备而引起的。

对于该类问题，首先从alert.log确认最近一次成功mount时那些磁盘对mount而言是必要的，例如：

 

 

NOTE: cache opening disk 11 of grp 1: DATA2 label:DATA2 <<<<<<<<<<<<<<<<<< 5 NOTE: cache opening disk 12 of grp 1: DATA20 label:DATA20 >
NOTE: cache opening disk 13 of grp 1: DATA3 label:DATA3 <<<<<<<<<<<<<<<<<<<<<< 6
NOTE: cache opening disk 14 of grp 1: DATA4 label:DATA4
NOTE: cache opening disk 15 of grp 1: DATA5 label:DATA5
NOTE: cache opening disk 16 of grp 1: DATA6 label:DATA6
NOTE: cache opening disk 17 of grp 1: DATA7 label:DATA7
NOTE: cache opening disk 18 of grp 1: DATA8 label:DATA8
NOTE: cache opening disk 19 of grp 1: DATA9 label:DATA9

 

 

 

接下来可以通过kfed来验证DISK HEADER，使用如下脚本：

#! /bin/sh
rm /tmp/kfed_DH.out /tmp/kfed_FS.out /tmp/kfed_BK.out /tmp/kfed_FD.out /tmp/kfed_DD.out /tmp/kfed_PST.out
for i in `ls /dev/asm-disk*`
do
echo $i >> /tmp/kfed_DH.out
kfed read $i >> /tmp/kfed_DH.out
echo $i >> /tmp/kfed_FS.out
kfed read $i blkn=1 >> /tmp/kfed_FS.out
echo $i >> /tmp/kfed_BK.out
kfed read $i aun=1 blkn=254 >> /tmp/kfed_BK.out
echo $i >> /tmp/kfed_FD.out
kfed read $i aun=2 blkn=1 >> /tmp/kfed_FD.out
echo $i >> /tmp/kfed_DD.out
kfed read $i aun=2 blkn=2 >> /tmp/kfed_DD.out
echo $i >> /tmp/kfed_PST.out
kfed read $i aun=1 blkn=2 >> /tmp/kfed_PST.out
done

====================================================================================================================================

 
 

  
  Ask_Maclean_liu_Oracle
 
 
2> Then validate the existing disk create timestamp on node 1 and
also check create timestamp of same disk on node 2 from ASMLib Label level.
Check the timestamp at ASMLib label on node 2 .
kfed read /dev/oracleasm/disks/ | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
Note down ,"kfdhdb.crestmp.hi" and "kfdhdb.crestmp.lo" from same Label on node 2 and note it down.
For Node 1 ,it will scan unique one ASMLib label check the same ASMlib label on node 1 too.
kfed read /dev/oracleasm/disks/ | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
And note down the major and minor number of same label from below command on node 1.
ls -ltr /dev/oracleasm/disks/
and get the emcpower device name from /proc/partitions by matching major and minor number .
Then note down the emcpower device name too.
3> Then try to findout which emcpower device is mapped with same ASMLib label using below command from root OS user.
kfed read /dev/emcpower<#>| egrep 'kfbh.type|kfdhdb.driver.provstr|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
Note down ,emcpower device names on node 1,because you will see lot devices with same ASMLib label for string "kfdhdb.driver.provstr".
Also note down,the value of string "kfdhdb.crestmp.hi" and "kfdhdb.crestmp.lo" all same labeled emcpower pseudo devices .
7> Then mount all those diskgroup after starting up the cluster.
Example ::----------
For DATA2 ,
Node 2,
(grid) grid11gr2 houxpsdbrac02:+ASM2:~> kfed read /dev/oracleasm/disks/DATA2 | egrep
'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dsknum: 11 ; 0x024: 0x000b
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.crestmp.hi: 32960211 ; 0x0a8: HOUR=0x13 DAYS=0x16 MNTH=0xb YEAR=0x7db
kfdhdb.crestmp.lo: 2385579008 ; 0x0ac: USEC=0x0 MSEC=0x43 SECS=0x23 MINS=0x23
Node 1,
disknum 11 : /dev/emcpowercf1
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483659 ; 0x008: TYPE=0x8 NUMB=0xb
kfbh.check: 4029604929 ; 0x00c: 0xf02ee441
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDATA2 ; 0x000: length=13 <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved  : 50 ; 0x00c: 0x00000032
kfdhdb.driver.reserved  : 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved  : 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved  : 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved  : 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 11 ; 0x024: 0x000b <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA2 ; 0x068: length=5
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32960399 ; 0x0a8: HOUR=0xf DAYS=0x1c MNTH=0xb YEAR=0x7db
kfdhdb.crestmp.lo: 31373312 ; 0x0ac: USEC=0x0 MSEC=0x3ae SECS=0x1d MINS=0x0
/dev/emcpowerdb1
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483659 ; 0x008: TYPE=0x8 NUMB=0xb
kfbh.check: 2125098320 ; 0x00c: 0x7eaa6d50
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDATA2 ; 0x000: length=13 <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved  : 50 ; 0x00c: 0x00000032
kfdhdb.driver.reserved  : 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved  : 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved  : 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved  : 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 11 ; 0x024: 0x000b <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA2 ; 0x068: length=5
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32960211 ; 0x0a8: HOUR=0x13 DAYS=0x16 MNTH=0xb YEAR=0x7db <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.crestmp.lo: 2385579008 ; 0x0ac: USEC=0x0 MSEC=0x43 SECS=0x23 MINS=0x23 <<<<<<<<<<<<<<<<<<<<<<<<<<
Note:: If you want to change the asm_diskstring as a workaround ,you have to again find out the right device to open write access to exact device
as at mounting time ASM writes mount timestamp.