Oracle15001,在使用RMAN恢复使用ASM文件系统的数据库时，遇到ORA-15001错误 | 信春哥，系统稳，闭眼上线不回滚！...

同事在使用RMAN向ASM文件系统中恢复数据库的时候,遇到ORA-15001错误:

channel c1: starting datafile backup set restore

channel c1: restoring control file

channel c1: reading from backup piece /data/ivldb/data/arc_20160405/spfile_c-1357933872-20160405-00

channel c1: ORA-19870: error while restoring backup piece /data/ivldb/data/arc_20160405/spfile_c-1357933872-20160405-00

ORA-19504: failed to create file "+DATA/ivldb34/controlfile/control01.ctl"

ORA-17502: ksfdcre:3 Failed to create file +DATA/ivldb34/controlfile/control01.ctl

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplete

ORA-15040: diskgroup is incomplet

failover to previous backup

告警日志报错信息如下：

Tue Apr 05 15:39:12 2016

NOTE: Loaded library: System

ORA-15025: could not open disk "/dev/mapper/data1"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data10"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data11"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data12"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data2"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data3"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data4"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data5"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

ORA-15025: could not open disk "/dev/mapper/data6"

ORA-27041: unable to open file

Linux-x86_64 Error: 13: Permission denied

Additional information: 9

此时ASM是启动的,而且所有磁盘组均已挂载(MOUNT)，而且asmcmd命令可以对磁盘组进行读写操作，检查磁盘权限也没有问题。

[oracle@SL010A-IVDB03 trace]$ ll /dev/dm-*

brw-rw---- 1 grid asmadmin 252, 0 4月 5 17:26 /dev/dm-0

brw-rw---- 1 grid asmadmin 252, 1 4月 5 17:26 /dev/dm-1

brw-rw---- 1 grid asmadmin 252, 10 4月 5 17:26 /dev/dm-10

brw-rw---- 1 grid asmadmin 252, 11 4月 5 17:26 /dev/dm-11

brw-rw---- 1 grid asmadmin 252, 12 4月 5 17:26 /dev/dm-12

brw-rw---- 1 grid asmadmin 252, 13 4月 5 17:26 /dev/dm-13

brw-rw---- 1 grid asmadmin 252, 14 4月 5 17:26 /dev/dm-14

brw-rw---- 1 grid asmadmin 252, 2 4月 5 17:26 /dev/dm-2

brw-rw---- 1 grid asmadmin 252, 3 4月 5 17:26 /dev/dm-3

brw-rw---- 1 grid asmadmin 252, 4 4月 5 17:26 /dev/dm-4

brw-rw---- 1 grid asmadmin 252, 5 4月 5 17:26 /dev/dm-5

brw-rw---- 1 grid asmadmin 252, 6 4月 5 17:26 /dev/dm-6

brw-rw---- 1 grid asmadmin 252, 7 4月 5 17:26 /dev/dm-7

brw-rw---- 1 grid asmadmin 252, 8 4月 5 17:26 /dev/dm-8

brw-rw---- 1 grid asmadmin 252, 9 4月 5 17:26 /dev/dm-9

同事是在一套原有的RAC数据库服务器上，先卸载了ORACLE和GRID软件，并删除了oracle和grid操作系统用户，然后重新创建，并使用dd命令擦写了使用的磁盘的数据，而且在恢复之前，他成功的使用oracle将spfile创建到ASM磁盘组中，数据库也可以成功启动到nomount状态，在restore的时候就遇到了这个问题。检查环境并没有发现什么问题，猜测可能是他删除操作系统用户的时候，udev绑定的磁盘为的所有者grid被删除，磁盘的属组出了问题，他再次创建用户的时候并没有重启udev服务，也没有重启服务器，猜测可能是这引起的问题，重启服务器后，问题解决，再次恢复未再遇到这个问题。