一次权限修改导致的RAC无法正常启动的问题的解决

关键字RAC，ORA-17503，ORA-12537

测试的Oracle12c RAC数据库出现问题，无法启动了（OEL6.5+ORACLE12C12.1.0.1）

1、RAC中资源状态

[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.DATA.dg
               ONLINE  OFFLINE      rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.FRA.dg
               ONLINE  OFFLINE      rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.OCR_VOTING.dg
               OFFLINE OFFLINE      rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.net1.network
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.ons
               ONLINE  ONLINE       rac1                     STABLE
               ONLINE  ONLINE       rac2                     STABLE
ora.proxy_advm
               ONLINE  OFFLINE      rac1                     STABLE
               ONLINE  OFFLINE      rac2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                     STABLE
ora.asm
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  ONLINE       rac2                     STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       rac2                     STABLE
ora.oc4j
      1        OFFLINE OFFLINE                               STABLE
ora.orarac.db
      1        ONLINE  ONLINE       rac1                     Open,STABLE
      2        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                     STABLE
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                     STABLE
-------------------------------------------------------------------------------

rac1节点ASM为OFFLINE，rac2节点db为OFFLINE。

2、手动启动两个资源报错

[grid@rac1 ~]$ srvctl start asm -node rac1
PRCR-1013 : Failed to start resource ora.asm
PRCR-1064 : Failed to start resource ora.asm on node rac1
CRS-2674: Start of 'ora.asm' on 'rac1' failed


[grid@rac1 ~]$ srvctl start instance -d orarac -i orarac2
PRCR-1013 : Failed to start resource ora.orarac.db
PRCR-1064 : Failed to start resource ora.orarac.db on node rac2
CRS-5017: The resource action "ora.orarac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/orarac/spfileorarac.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orarac/spfileorarac.ora
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 2
Additional information: 12255236
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/12.1.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orarac.db' on 'rac2' failed

3、原因分析

查看权限设置发现：

[root@rac1 u01]# ll
total 36
drwxrwxr-x 4 oracle oinstall  4096 Mar 10 09:39 app
drwxr-xr-x 3 oracle oinstall  4096 Mar 23 15:55 backup
drwxrwxr-x 9 oracle oinstall  4096 Mar 13 08:34 gridbase
drwxrwxr-x 2 oracle oinstall 16384 Mar 10 09:13 lost+found
drwxrwxr-x 6 oracle oinstall  4096 Mar 12 10:42 oraInventory
drwxrwxr-x 3 oracle oinstall  4096 Mar 12 10:01 software

[root@rac2 u01]# ll
total 44
drwxrwxr-x 4 root   oinstall  4096 Mar 10 09:39 app
drwxr-xr-x 3 oracle oinstall  4096 Mar 23 15:57 backup
-rwxrwxr-x 1 oracle oinstall  8827 Mar 10 10:00 cvuqdisk-1.0.9-1.rpm
drwxrwxr-x 6 grid   oinstall  4096 Mar 11 15:59 gridbase
drwxrwxr-x 2 oracle oinstall 16384 Mar 10 09:35 lost+found
drwxrwxr-x 5 grid   oinstall  4096 Mar 12 10:42 oraInventory

由于大意修改了权限，导致问题的发生。

4、解决rac1上ASM无法启动的问题

修改rac1节点的文件夹权限，便可以启动ASM。

5、解决rac2上db无法启动的问题

CRS-5017: The resource action "ora.orarac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/orarac/spfileorarac.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/orarac/spfileorarac.ora
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied

从上面能看出是权限的问题，搜索ora-17503，找到解决办法（重设$GRID_HOME/bin/oracle文件的权限），分别在两个节点上执行：

[root@rac1 grid]# cd bin
[root@rac1 bin]# ls -la oracle
-rwxrwxrwx 1 oracle oinstall 263068997 Mar 11 15:21 oracle
[root@rac1 bin]# chown grid:oinstall oracle
[root@rac1 bin]# chmod 6751 oracle

[root@rac2 u01]# cd /u01/app/grid/12.1.0/grid
[root@rac2 grid]# cd bin
[root@rac2 bin]# ls -la oracle
-rwxrwxrwx 1 grid oinstall 263068997 Mar 11 15:22 oracle
[root@rac2 bin]# chmod 6751 oracle

之后启动正常。

最后还需要在两个节点确认权限(否则通过tns连接时可能出现ora-12537):

chmod 6751 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

6、总结

在操作系统中修改文件夹权限一定要谨慎！