在新建的RAC上恢复数据库时ASM报错ORA-15001 ORA-15040

涉及版本

目前发现：11g 、12c、19c 都有过这个问题

故障背景

为生产环境RAC，搭建ADG灾备RAC环境。所以在灾备环境中仅需要安装好cluster软件和数据库软件。不用DBCA建库。数据库是生产端通过RMAN备份恢复而来的。这里我就不再赘述备库恢复流程了。

• 现在我们在node1上，通过存放在本地pfile将数据库启动到nomount。
• 然后登陆到RMAN中，通过备份片恢复控制文件时发生报错

报错信息如下：

RMAN> restore controlfile from '/qybackup/tongchengzaibei/rman/rdgdb/ctl_fgvf3u2e_1_1.bak';

Starting restore at 09-DEC-20
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=2323 instance=rdgdb2 device type=DISK

channel ORA_DISK_1: restoring control file
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 12/09/2020 12:38:13
ORA-19870: error while restoring backup piece /qybackup/tongchengzaibei/rman/rdgdb/ctl_fgvf3u2e_1_1.bak
ORA-19504: failed to create file "+DATADG"
ORA-17502: ksfdcre:4 Failed to create file +DATADG
ORA-15001: diskgroup "DATADG" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete

处理流程

1 查看告警日志

• 查看db的告警日志，发现了大量的磁盘没有权限的报错

ORA-15025: could not open disk "/dev/mapper/asm_data08"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk "/dev/mapper/asm_data09"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk "/dev/mapper/asm_data10"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk "/dev/mapper/asm_data11"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
ORA-15025: could not open disk "/dev/mapper/asm_data12"
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
Wed Dec 09 10:58:07 2020
SUCCESS: diskgroup DATADG was dismounted
ERROR: diskgroup DATADG was not mounted

2 查看磁盘权限

既然报错说磁盘权限问题，那么我们就去查看下磁盘权限，以下输出虽然很多，但是可以清楚的看到asm所使用到的磁盘都是grid:asmadmin 和 660权限，没有任何问题。

[root@xxxx01:/root]#ll /dev/mapper/asm*
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_arch01 -> ../dm-9
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_arch02 -> ../dm-19
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_arch03 -> ../dm-21
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data01 -> ../dm-22
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data02 -> ../dm-30
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data03 -> ../dm-16
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data04 -> ../dm-13
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data05 -> ../dm-25
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data06 -> ../dm-10
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data07 -> ../dm-12
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data08 -> ../dm-17
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data09 -> ../dm-4
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data10 -> ../dm-11
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data11 -> ../dm-3
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data12 -> ../dm-18
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data13 -> ../dm-2
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data14 -> ../dm-23
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data15 -> ../dm-20
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data16 -> ../dm-15
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data17 -> ../dm-5
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data18 -> ../dm-7
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data19 -> ../dm-24
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data20 -> ../dm-28
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_data21 -> ../dm-29
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data22 -> ../dm-6
lrwxrwxrwx 1 root root 7 Dec  9 12:44 /dev/mapper/asm_data23 -> ../dm-8
lrwxrwxrwx 1 root root 8 Dec  9 12:49 /dev/mapper/asm_ocr_vot01 -> ../dm-14
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_ocr_vot02 -> ../dm-26
lrwxrwxrwx 1 root root 8 Dec  9 12:44 /dev/mapper/asm_ocr_vot03 -> ../dm-27
[root@dralmdb01:/root]#
[root@dralmdb01:/root]#
[root@dralmdb01:/root]#ll /dev/dm*
brw-rw---- 1 root disk     253,  0 Dec  9 12:43 /dev/dm-0
brw-rw---- 1 root disk     253,  1 Dec  9 12:43 /dev/dm-1
brw-rw---- 1 grid asmadmin 253, 10 Dec  9 12:44 /dev/dm-10
brw-rw---- 1 grid asmadmin 253, 11 Dec  9 12:44 /dev/dm-11
brw-rw---- 1 grid asmadmin 253, 12 Dec  9 12:44 /dev/dm-12
brw-rw---- 1 grid asmadmin 253, 13 Dec  9 12:44 /dev/dm-13
brw-rw---- 1 grid asmadmin 253, 14 Dec  9 12:53 /dev/dm-14
brw-rw---- 1 grid asmadmin 253, 15 Dec  9 12:44 /dev/dm-15
brw-rw---- 1 grid asmadmin 253, 16 Dec  9 12:44 /dev/dm-16
brw-rw---- 1 grid asmadmin 253, 17 Dec  9 12:44 /dev/dm-17
brw-rw---- 1 grid asmadmin 253, 18 Dec  9 12:44 /dev/dm-18
brw-rw---- 1 grid asmadmin 253, 19 Dec  9 12:44 /dev/dm-19
brw-rw---- 1 grid asmadmin 253,  2 Dec  9 12:44 /dev/dm-2
brw-rw---- 1 grid asmadmin 253, 20 Dec  9 12:44 /dev/dm-20
brw-rw---- 1 grid asmadmin 253, 21 Dec  9 12:44 /dev/dm-21
brw-rw---- 1 grid asmadmin 253, 22 Dec  9 12:53 /dev/dm-22
brw-rw---- 1 grid asmadmin 253, 23 Dec  9 12:44 /dev/dm-23
brw-rw---- 1 grid asmadmin 253, 24 Dec  9 12:44 /dev/dm-24
brw-rw---- 1 grid asmadmin 253, 25 Dec  9 12:44 /dev/dm-25
brw-rw---- 1 grid asmadmin 253, 26 Dec  9 12:53 /dev/dm-26
brw-rw---- 1 grid asmadmin 253, 27 Dec  9 12:53 /dev/dm-27
brw-rw---- 1 grid asmadmin 253, 28 Dec  9 12:44 /dev/dm-28
brw-rw---- 1 grid asmadmin 253, 29 Dec  9 12:44 /dev/dm-29
brw-rw---- 1 grid asmadmin 253,  3 Dec  9 12:44 /dev/dm-3
brw-rw---- 1 grid asmadmin 253, 30 Dec  9 12:44 /dev/dm-30
brw-rw---- 1 root disk     253, 31 Dec  9 12:43 /dev/dm-31
brw-rw---- 1 root disk     253, 32 Dec  9 12:43 /dev/dm-32
brw-rw---- 1 root disk     253, 33 Dec  9 12:43 /dev/dm-33
brw-rw---- 1 root disk     253, 34 Dec  9 12:43 /dev/dm-34
brw-rw---- 1 root disk     253, 35 Dec  9 12:43 /dev/dm-35
brw-rw---- 1 root disk     253, 36 Dec  9 12:43 /dev/dm-36
brw-rw---- 1 grid asmadmin 253,  4 Dec  9 12:44 /dev/dm-4
brw-rw---- 1 grid asmadmin 253,  5 Dec  9 12:44 /dev/dm-5
brw-rw---- 1 grid asmadmin 253,  6 Dec  9 12:44 /dev/dm-6
brw-rw---- 1 grid asmadmin 253,  7 Dec  9 12:44 /dev/dm-7
brw-rw---- 1 grid asmadmin 253,  8 Dec  9 12:44 /dev/dm-8
brw-rw---- 1 grid asmadmin 253,  9 Dec  9 12:53 /dev/dm-9

3 查看命令权限

没有权限不仅仅是磁盘本身的权限对不对，还有一种可能是oracle用户没有读取asm磁盘的权限。

[oracle@drfindb01:/home/oracle]$ll $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 240993408 Dec  8 18:30 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle

OK！可以看到oracle命令的权限是 oracle:oinstall ，并没有关于那一大堆磁盘权限asmadin的。所以。问题在这儿了。

解决办法

方法1 （推荐）

切换到grid用户下，执行脚本去修改，oracle的命令权限

[grid@dralmdb01:/home/grid]$cd $ORACLE_HOME
[grid@dralmdb01:/u01/app/11.2.0/grid]$cd bin
[grid@dralmdb01:/u01/app/11.2.0/grid/bin]$
[grid@dralmdb01:/u01/app/11.2.0/grid/bin]$./setasmgidwrap o=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle

方法2

方法二就简单粗暴了些，直接改权限。。

su -

chown oracle:asmadmin /u01/app/oracle/product/12.1/db_1/bin/oracle
chmod 6751 /u01/app/oracle/product/12.1/db_1/bin/oracle

重启服务器

修改完全下之后需要重启服务器 reboot！

如果没有重启服务器的话，会看到以下问题：

[oracle@drfindb02:/home/oracle]$sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Dec 9 12:24:01 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected.
SQL> exit
Disconnected

或者这样的问题：

[oracle@drfindb02:/home/oracle]$rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Wed Dec 9 12:23:21 2020

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00554: initialization of internal recovery manager package failed
RMAN-04005: error from target database: 
ORA-27140: attach to post/wait facility failed
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Operation not permitted
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 1000 (oinstall), current egid = 1003 (asmadmin)

好啦。。ASM可以用了，，继续干活去了。