记一次存储block corrupt 导致数据库大恢复

前几天在vmware装了一次Oracle 10g +rac +asm，详见http://itspace.iteye.com/blog/631585。
停止mcrac1节点vip，注意红色字体标示，也将mcrac1节点的asm实例停止了。
[quote][oracle@mcrac1 dbs]$ crs_stop ora.mcrac1.vip -f
Attempting to stop `ora.mcrac1.LISTENER_MCRAC1.lsnr` on member `mcrac1`
Stop of `ora.mcrac1.LISTENER_MCRAC1.lsnr` on member `mcrac1` succeeded.
[color=red]Attempting to stop `ora.mcrac1.ASM1.asm` on member `mcrac1`[/color]
Stop of `ora.mcrac1.ASM1.asm` on member `mcrac1` succeeded.
Attempting to stop `ora.mcrac1.vip` on member `mcrac1`
Stop of `ora.mcrac1.vip` on member `mcrac1` succeeded.[/quote]
asm实例停止之后，再次在mcrac1节点启动实例的时候意外发生了：
[quote]SQL> startup
ASM instance started

Total System Global Area 92274688 bytes
Fixed Size 1217884 bytes
Variable Size 65890980 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15130: diskgroup "DATADG" is being dismounted[/quote]
进一步后台观察alert日志，发现有Corrupt Block产生，导致ASM实例启动错误：
[quote]Sat Apr 3 16:22:17 2010
SQL> ALTER DISKGROUP ALL MOUNT
Sat Apr 3 16:22:17 2010
NOTE: cache registered group DATADG number=1 incarn=0x811fef38
Sat Apr 3 16:22:17 2010
Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
Sat Apr 3 16:22:17 2010
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Sat Apr 3 16:22:17 2010
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Sat Apr 3 16:22:17 2010
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Sat Apr 3 16:22:17 2010
NOTE: Hbeat: instance not first (grp 1)
NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/raw/raw1
Sat Apr 3 16:22:17 2010
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/raw/raw2
NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/raw/raw3
NOTE: cache mounting (not first) group 1/0x811FEF38 (DATADG)
Sat Apr 3 16:22:17 2010
kjbdomatt send to node 1
Sat Apr 3 16:22:18 2010
NOTE: attached to recovery domain 1
Sat Apr 3 16:22:19 2010
WARNING: cache failed to read fn=3 indblk=0 from disk(s): 2
[color=red]ORA-15196: invalid ASM block header [kfc.c:7910] [endian_kfbh] [3] [2147483648] [6 != 1][/color]
NOTE: a corrupted block was dumped to the trace file
System State dumped to trace file /oracle/app/admin/+ASM/bdump/+asm1_lgwr_11493.trc
NOTE: cache initiating offline of disk 2 group 1
WARNING: offlining disk 2.3956219848 (DATADG_0002) with mask 0x3
NOTE: PST update: grp = 1, dsk = 2, mode = 0x6
Sat Apr 3 16:22:19 2010
ERROR: too many offline disks in PST (grp 1)
Sat Apr 3 16:22:20 2010
NOTE: PST not enabling heartbeating (grp 1): group dismounted
Sat Apr 3 16:22:20 2010
NOTE: halting all I/Os to diskgroup DATADG
NOTE: active pin found: 0x0x2427ccd0
NOTE: active pin found: 0x0x2427cc64
Sat Apr 3 16:22:20 2010
WARNING: instance recovery required
Sat Apr 3 16:22:20 2010
NOTE: Hbeat: instance not first (grp 1)
Sat Apr 3 16:22:20 2010
NOTE: cache mounting (not first) (retry) group 1/0x811FEF38 (DATADG)
ERROR: ORA-15130 signalled during mount of diskgroup DATADG
NOTE: cache dismounting group 1/0x811FEF38 (DATADG)
Sat Apr 3 16:22:21 2010
kjbdomdet send to node 1
detach from dom 1, sending detach message to node 1
Sat Apr 3 16:22:21 2010
Dirty detach reconfiguration started (old inc 6, new inc 6)
List of nodes:
0 1
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 1 invalid = TRUE
35 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Sat Apr 3 16:22:21 2010
WARNING: dirty detached from domain 1
Sat Apr 3 16:22:21 2010
NOTE: PST enabling heartbeating (grp 1)
Sat Apr 3 16:22:21 2010
ERROR: diskgroup DATADG was not mounted
Sat Apr 3 16:22:21 2010
WARNING: PST-initiated MANDATORY DISMOUNT of group DATADG not performed - group not mounted
Sat Apr 3 16:22:21 2010
Errors in file /oracle/app/admin/+ASM/bdump/+asm1_b000_11614.trc:
ORA-15001: diskgroup "DATADG" does not exist or is not mounted
[/quote]
后台trace文件有如下提示，从提示中可以看出kfbh.endian已经为0x06了，正确的值应该为1.
[quote]*** SERVICE NAME:() 2010-04-03 16:44:53.125
*** SESSION ID:(41.1) 2010-04-03 16:44:53.125
OSM metadata block dump:
[color=red]kfbh.endian: 6 ; 0x000: 0x06[/color]
kfbh.hard: 162 ; 0x001: 0xa2
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 12617264 ; 0x004: T=0 NUMB=0xc08630
kfbh.block.obj: 853426 ; 0x008: TYPE=0x0 NUMB=0xd05b2
kfbh.check: 67174400 ; 0x00c: 0x04010000
kfbh.fcn.base: 21267 ; 0x010: 0x00005313
kfbh.fcn.wrap: 1 ; 0x014: 0x00000001
kfbh.spare1: 51428 ; 0x018: 0x0000c8e4
kfbh.spare2: 853426 ; 0x01c: 0x000d05b2[/quote]

从理论上来讲，mcrac1和mcrac2节点用的是共享存储，一旦mcrac1上发生Corrupt Block，mcrac2节点也用该同样启动错误，但奇怪的是mcrac2节点asm实例启动正常，从alert日志中可以看出。
[quote]Sat Apr 3 16:55:19 2010
SQL> ALTER DISKGROUP ALL MOUNT
Sat Apr 3 16:55:19 2010
NOTE: cache registered group DATADG number=1 incarn=0xbb92fa11
Sat Apr 3 16:55:19 2010
Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
Sat Apr 3 16:55:19 2010
NOTE: Hbeat: instance first (grp 1)
Sat Apr 3 16:55:23 2010
NOTE: start heartbeating (grp 1)
NOTE: cache opening disk 0 of grp 1: DATADG_0000 label:VOL1
Sat Apr 3 16:55:23 2010
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: DATADG_0001 label:VOL2
NOTE: cache opening disk 2 of grp 1: DATADG_0002 label:VOL3
NOTE: cache mounting (first) group 1/0xBB92FA11 (DATADG)
* allocate domain 1, invalid = TRUE
kjbdomatt send to node 0
Sat Apr 3 16:55:23 2010
NOTE: attached to recovery domain 1
Sat Apr 3 16:55:24 2010
NOTE: cache recovered group 1 to fcn 0.3049
Sat Apr 3 16:55:24 2010
NOTE: opening chunk 1 at fcn 0.3049 ABA
NOTE: seq=7 blk=214
Sat Apr 3 16:55:24 2010
NOTE: cache mounting group 1/0xBB92FA11 (DATADG) succeeded
[color=red]SUCCESS: diskgroup DATADG was mounted[/color]
Sat Apr 3 16:55:24 2010
NOTE: recovering COD for group 1/0xbb92fa11 (DATADG)
SUCCESS: completed COD recovery for group 1/0xbb92fa11 (DATADG)[/quote]
考虑到vmware存在bug，有时候只需要将出问题的虚拟机重启，即可解决问题，于是将mcrac1节点虚拟机重启，重启之后问题依旧。既然mcrac2节点能启动正常，于是在mcrac2节点数据库实例在mount状态下将数据库做全备。
[quote]SQL> select dbid from v$database;

DBID
----------
676148534[/quote]
[quote]RMAN> CONFIGURE CONTROLFILE AUTOBACKUP on;

new RMAN configuration parameters:
CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored

RMAN> backup database format '/ocfs2/backup/%d__%s_%p_%T.bkp';

Starting backup at 03-APR-10
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00001 name=+DATADG/dbrac/datafile/system.256.715082291
input datafile fno=00003 name=+DATADG/dbrac/datafile/sysaux.257.715082291
input datafile fno=00002 name=+DATADG/dbrac/datafile/undotbs1.258.715082293
input datafile fno=00005 name=+DATADG/dbrac/datafile/undotbs2.264.715082411
input datafile fno=00004 name=+DATADG/dbrac/datafile/users.259.715082293
channel ORA_DISK_1: starting piece 1 at 03-APR-10
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 04/03/2010 17:26:00
[color=red]ORA-19566: exceeded limit of 0 corrupt blocks for file [/color]

+DATADG/dbrac/datafile/sysaux.257.715082291[/quote]
不幸的是，3号数据文件即辅助表空间已经发生Corrupt Block。设置参数，使得能备份成功。
[quote]RUN
{
SET MAXCORRUPT FOR DATAFILE 3 TO 100000;
BACKUP database format '/ocfs2/backup/%d__%s_%p_%T.bkp';
}[/quote]
查看V$DATABASE_BLOCK_CORRUPTION可以看到3好数据文件上已经有201个Corrupt Block
[quote]QL> select * from V$DATABASE_BLOCK_CORRUPTION;

FILE# BLOCK# BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
3 30724 1 0 FRACTURED
3 30725 52 0 CORRUPT
3 30778 3 0 CORRUPT
3 30782 1 0 CORRUPT
3 30785 14 0 CORRUPT
3 30800 1 0 CORRUPT
...


3 37341 3 0 ALL ZERO
3 37401 8 0 ALL ZERO
3 37410 2 0 ALL ZERO
3 37413 20 0 ALL ZERO
3 37437 1 0 ALL ZERO
3 37439 1 0 ALL ZERO
3 37441 40 0 ALL ZERO
3 37489 1 0 ALL ZERO
3 37493 34 0 ALL ZERO
3 37528 2 0 ALL ZERO
3 37531 11 0 ALL ZERO

FILE# BLOCK# BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
3 37543 42 0 ALL ZERO
3 37609 56 0 ALL ZERO
3 37681 720 0 ALL ZERO


201 rows selected.[/quote]
进一步用dbv校验asm中3号数据文件，Corrupt Block依旧！
[quote][oracle@mcrac2 dbs]$ dbv USERID=sys/oracle blocksize=8192

file='+DATADG/dbrac/datafile/sysaux.257.715082291'

...
Page 31730 is marked corrupt
Corrupt block relative dba: 0x00c07bf2 (file 3, block 31730)
Bad header found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00c0931a
last change scn: 0x0000.00117a68 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x7a680601
check value in block header: 0xcd36
computed block checksum: 0x0

Page 31731 is marked corrupt
Corrupt block relative dba: 0x00c07bf3 (file 3, block 31731)
Bad header found during dbv:
Data in bad block:
type: 6 format: 2 rdba: 0x00c0931b
last change scn: 0x0000.00117a79 seq: 0x1 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x7a790601
check value in block header: 0x9425
computed block checksum: 0x0[/quote]
校验1号数据文件，所幸的是没有产生任何Corrupt Block。
[quote][oracle@mcrac2 dbs]$ dbv USERID=sys/oracle blocksize=8192

file='+DATADG/dbrac/datafile/SYSTEM.256.715082291'

DBVERIFY: Release 10.2.0.1.0 - Production on Sat Apr 3 17:47:17 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

DBVERIFY - Verification starting : FILE = +DATADG/dbrac/datafile/SYSTEM.256.715082291


DBVERIFY - Verification complete

Total Pages Examined : 61440
Total Pages Processed (Data) : 36661
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 6827
Total Pages Failing (Index): 0
Total Pages Processed (Other): 1774
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 16178
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Highest block SCN : 0 (0.0)[/quote]

在做好全备的前提下，将两节点的所有实例关闭，重新划盘，重做ASM DISK GROUP.
双节点关闭所有实例，一节点删除asm磁盘

[quote]# oracleasm listdisks
VOL1
VOL2
VOL3

# /etc/init.d/oracleasm deletedisk VOL1
Removing ASM disk "VOL1": [ OK ]
# /etc/init.d/oracleasm deletedisk VOL2
Removing ASM disk "VOL2": [ OK ]
# /etc/init.d/oracleasm deletedisk VOL3
Removing ASM disk "VOL3": [ OK ][/quote]

删除磁盘之后二节点查询
[quote]# oracleasm listdisks
VOL1
VOL2
VOL3

# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Cleaning disk "VOL1"
Cleaning disk "VOL2"
Cleaning disk "VOL3"
Scanning system for ASM disks...
# oracleasm listdisks[/quote]

一节点创建磁盘
[quote]# oracleasm createdisk VOL1 /dev/sdc1
Writing disk header: done
Instantiating disk: done
# oracleasm createdisk VOL2 /dev/sdd1
Writing disk header: done
Instantiating disk: done
# oracleasm createdisk VOL3 /dev/sde1
Writing disk header: done
Instantiating disk: done
# oracleasm listdisks
VOL1
VOL2
VOL3[/quote]
二节点扫描磁盘
[quote]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "VOL1"
Instantiating disk "VOL2"
Instantiating disk "VOL3"
# oracleasm listdisks
VOL1
VOL2
VOL3[/quote]

一节点重新创建磁盘组
[quote][oracle@mcrac1 ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Apr 3 17:55:59 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ASM instance started

Total System Global Area 92274688 bytes
Fixed Size 1217884 bytes
Variable Size 65890980 bytes
ASM Cache 25165824 bytes
SQL> create diskgroup datavg external redundancy disk 'ORCL:VOL1','ORCL:VOL2','ORCL:VOL3';

Diskgroup created.
SQL> select name,PATH,STATE from v$asm_disk;

NAME
------------------------------
PATH
--------------------------------------------------------------------------------
STATE
--------
VOL1
ORCL:VOL1
NORMAL

VOL2
ORCL:VOL2
NORMAL

NAME
------------------------------
PATH
--------------------------------------------------------------------------------
STATE
--------

VOL3
ORCL:VOL3
NORMAL

SQL> select name,STATE from v$asm_diskgroup;

NAME STATE
------------------------------ -----------
DATAVG MOUNTED[/quote]
后台alert日志显示
[quote]NOTE: detached from domain 1
Sat Apr 3 18:01:07 2010
SUCCESS: diskgroup DATAVG was created
NOTE: requesting all-instance PST refresh for group=0
NOTE: cache registered group DATAVG number=1 incarn=0x501d8e37
Sat Apr 3 18:01:07 2010
NOTE: Hbeat: instance first (grp 1)
Sat Apr 3 18:01:12 2010
NOTE: start heartbeating (grp 1)
NOTE: cache opening disk 0 of grp 1: VOL1 label:VOL1
Sat Apr 3 18:01:12 2010
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: VOL2 label:VOL2
NOTE: cache opening disk 2 of grp 1: VOL3 label:VOL3
NOTE: cache mounting (first) group 1/0x501D8E37 (DATAVG)
* allocate domain 1, invalid = TRUE
kjbdomatt send to node 1
Sat Apr 3 18:01:12 2010
NOTE: attached to recovery domain 1
Sat Apr 3 18:01:12 2010
NOTE: cache recovered group 1 to fcn 0.0
Sat Apr 3 18:01:12 2010
NOTE: opening chunk 1 at fcn 0.0 ABA
NOTE: seq=2 blk=0
Sat Apr 3 18:01:12 2010
NOTE: cache mounting group 1/0x501D8E37 (DATAVG) succeeded
SUCCESS: diskgroup DATAVG was mounted
Sat Apr 3 18:01:13 2010
NOTE: recovering COD for group 1/0x501d8e37 (DATAVG)
SUCCESS: completed COD recovery for group 1/0x501d8e37 (DATAVG)[/quote]

但是mount disk时报错
[quote]SQL> startup
ASM instance started

Total System Global Area 92274688 bytes
Fixed Size 1217884 bytes
Variable Size 65890980 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup
"DATADG"[/quote]

后台日志报错
[quote]SQL> ALTER DISKGROUP ALL MOUNT
Sat Apr 3 18:07:07 2010
NOTE: cache registered group DATADG number=1 incarn=0x17ce908a
Sat Apr 3 18:07:07 2010
Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface
Sat Apr 3 18:07:07 2010
ERROR: no PST quorum in group 1: required 2, found 0
Sat Apr 3 18:07:07 2010
NOTE: cache dismounting group 1/0x17CE908A (DATADG)
NOTE: dbwr not being msg'd to dismount
ERROR: diskgroup DATADG was not mounted[/quote]
经查是参数文件中asm_diskgroups='DATADG',而我重建的disk group为DATAVG，一字之差，使我多走了很多弯路，包括再次重划磁盘，重启主机，修改asm_diskstring为'/dev/sdc1','/dev/sdd1','/dev/sde1'，使用dd命令将磁盘分区清空等尝试方法，期间甚至发生了磁盘不可写的情况。
[quote]Warning: invalid flag 0x0000 of partition table[/quote]
将2节点asm实例建好之后，由于没有自动备份至asm磁盘的控制文件和参数文件已经被格式化掉，只能手工编辑参数文件和控制文件。
参数文件内容如下
[quote]processes = 150
sessions = 170
__shared_pool_size = 134217728
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size= 0
#spfile= '+DATAVG/dbrac/spfiledbrac.ora'
sga_target = 436207616
control_files = '+DATAVG/dbrac/controlfile/control01.ctl'
db_block_size = 8192
__db_cache_size = 289406976
compatible= 10.2.0.1.0
db_file_multiblock_read_count= 16
#cluster_database= TRUE
cluster_database_instances= 2
db_create_file_dest = '+DATAVG'
thread= 1
instance_number= 1
undo_management= AUTO
dbrac1.undo_tablespace = 'UNDOTBS1'
dbrac2.undo_tablespace = 'UNDOTBS2'
remote_login_passwordfile= 'EXCLUSIVE'
#db_domain =
#dispatchers= '(PROTOCOL=TCP)' '(SERVICE=dbracXDB)'
#remote_listener= LISTENERS_DBRAC
job_queue_processes = 10
background_dump_dest = '/oracle/app/admin/dbrac/bdump'
user_dump_dest= '/oracle/app/admin/dbrac/udump'
core_dump_dest = '/oracle/app/admin/dbrac/cdump'
audit_file_dest = '/oracle/app/admin/dbrac/adump'
db_name= 'dbrac'
open_cursors= 300
pga_aggregate_target =144703488[/quote]
控制文件内容如下
[quote]CREATE CONTROLFILE REUSE DATABASE "DBRAC" RESETLOGS ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 1968
LOGFILE
GROUP 1 '+DATAVG/dbrac/datafile/redo01.log' SIZE 50M,
GROUP 2 '+DATAVG/dbrac/datafile/redo02.log' SIZE 50M,
GROUP 3 '+DATAVG/dbrac/datafile/redo03.log' SIZE 50M
-- STANDBY LOGFILE
DATAFILE
'/tmp/system.256.715082291',
'/tmp/sysaux.257.715082291',
'/tmp/undotbs1.258.715082293',
'/tmp/undotbs2.264.715082411',
'/tmp/users.259.715082293'
CHARACTER SET ZHS16GBK;[/quote]
将数据库实例启动至nomount。
[quote]oracle@mcrac1 dbs]$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.1.0 - Production on Sun Apr 4 09:50:22 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/oracle/app/product/10.2.0/db_2/dbs/tmp.txt'
ORACLE instance started.

Total System Global Area 436207616 bytes
Fixed Size 1219856 bytes
Variable Size 142607088 bytes
Database Buffers 289406976 bytes
Redo Buffers 2973696 bytes[/quote]

在rman中恢复数据文件
[quote]DECLARE
devtype varchar2(256);
done boolean;
BEGIN
devtype := dbms_backup_restore.DeviceAllocate (type => '',ident => 'testdatafile');
dbms_backup_restore.RestoreSetDatafile;
dbms_backup_restore.RestoreDatafileTo(dfnumber => 1,toname => '/tmp/system.256.715082291');
dbms_backup_restore.RestoreDatafileTo(dfnumber => 2,toname => '/tmp/undotbs1.258.715082293');
dbms_backup_restore.RestoreDatafileTo(dfnumber => 3,toname => '/tmp/sysaux.257.715082291');
dbms_backup_restore.RestoreDatafileTo(dfnumber => 4,toname => '/tmp/users.259.715082293');
dbms_backup_restore.RestoreDatafileTo(dfnumber => 5,toname => '/tmp/undotbs2.264.715082411');
dbms_backup_restore.RestoreBackupPiece(done => done,handle => '/ocfs2/backup/DBRAC__6_1_20100403.bkp', params => null);
dbms_backup_restore.DeviceDeallocate;
END;
/[/quote]

接着创建控制文件，使用resetlogs打开数据库。
[quote]SQL> alter database open resetlogs;

Database altered.[/quote]
接下来，将数据文件转入asm实例中，再次将实例置为mount状态
[quote]RMAN> RUN
2> {
3> SET MAXCORRUPT FOR DATAFILE 3 TO 100000;
BACKUP as copy database format '+DATAVG';
}
4> 5>
executing command: SET MAX CORRUPT

Starting backup at 04-APR-10
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=153 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00001 name=/tmp/system.256.715082291
output filename=+DATAVG/dbrac/datafile/system.262.715427575 tag=TAG20100404T095253 recid=1

stamp=715427610
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:45
channel ORA_DISK_1: starting datafile copy
input datafile fno=00003 name=/tmp/sysaux.257.715082291
output filename=+DATAVG/dbrac/datafile/sysaux.263.715427619 tag=TAG20100404T095253 recid=2

stamp=715427635
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile fno=00002 name=/tmp/undotbs1.258.715082293
output filename=+DATAVG/dbrac/datafile/undotbs1.264.715427645 tag=TAG20100404T095253 recid=3

stamp=715427648
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
channel ORA_DISK_1: starting datafile copy
input datafile fno=00005 name=/tmp/undotbs2.264.715082411
output filename=+DATAVG/dbrac/datafile/undotbs2.265.715427651 tag=TAG20100404T095253 recid=4

stamp=715427653
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting datafile copy
copying current control file
output filename=+DATAVG/dbrac/controlfile/backup.266.715427655 tag=TAG20100404T095253 recid=5

stamp=715427656
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=/tmp/users.259.715082293
output filename=+DATAVG/dbrac/datafile/users.267.715427657 tag=TAG20100404T095253 recid=6

stamp=715427657
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 04-APR-10


RMAN> switch database to copy;

datafile 1 switched to datafile copy "+DATAVG/dbrac/datafile/system.262.715427575"
datafile 2 switched to datafile copy "+DATAVG/dbrac/datafile/undotbs1.264.715427645"
datafile 3 switched to datafile copy "+DATAVG/dbrac/datafile/sysaux.263.715427619"
datafile 4 switched to datafile copy "+DATAVG/dbrac/datafile/users.267.715427657"
datafile 5 switched to datafile copy "+DATAVG/dbrac/datafile/undotbs2.265.715427651"[/quote]
打开的时候报corrupted
[quote]RMAN> alter database open;

database opened
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of alter db command at 04/04/2010 09:55:36
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 59584)
ORA-01110: data file 1: '+DATAVG/dbrac/datafile/system.262.715427575'[/quote]

后台日志显示：
[quote]Sun Apr 4 09:55:35 2010
Hex dump of (file 1, block 59584) in trace file

/oracle/app/admin/dbrac/udump/dbrac1_ora_5410.trc
Corrupt block relative dba: 0x0040e8c0 (file 1, block 59584)
Bad header found during buffer read
Data in bad block:
type: 40 format: 2 rdba: 0x00c03fe0
last change scn: 0x0000.0005b923 seq: 0x2 flg: 0x04
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0xb9232802
check value in block header: 0x4ae3
computed block checksum: 0x0
Reread of rdba: 0x0040e8c0 (file 1, block 59584) found same corrupted data
[/quote]

用dbv校验转asm之前的1号数据文件时，没有存在Corrupt Block
[quote][oracle@mcrac1 tmp]$ dbv file=system.256.715082291 blocksize=8192

DBVERIFY: Release 10.2.0.1.0 - Production on Sun Apr 4 10:59:39 2010

Copyright (c) 1982, 2005, Oracle. All rights reserved.

DBVERIFY - Verification starting : FILE = system.256.715082291


DBVERIFY - Verification complete

Total Pages Examined : 61440
Total Pages Processed (Data) : 36686
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 6827
Total Pages Failing (Index): 0
Total Pages Processed (Other): 1774
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 16153
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Highest block SCN : 1191473 (0.1191473)[/quote]
而使用rman转换之后就有Corrupt Block。
[quote][oracle@mcrac1 tmp]$ dbv USERID=sys/oracle BLOCKSIZE=8192 file='+DATAVG/dbrac/datafile/system.262.715427575'
DBVERIFY - Verification complete

Total Pages Examined : 61440
Total Pages Processed (Data) : 36620
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 6801
Total Pages Failing (Index): 0
Total Pages Processed (Other): 1774
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 16131
[color=red]Total Pages Marked Corrupt : 117[/color]
Total Pages Influx : 3
Highest block SCN : 0 (0.0)[/quote]
从这里可以判定，虚拟机共享存储有问题，初步估计是划盘时没有预分配空间所致。
以上内容，仅作记录，留待以后研究！
ptw：发这篇博文时，系统提示我,这也要河蟹，[color=red]呵呵[/color]
[img]http://dl.iteye.com/upload/attachment/229101/8bc67722-9678-3457-aab5-808025fbb2eb.jpg[/img]