oracle 12c rac Linux-x86_64 Error: 13: Permission denied

情况描述：

oracle 12c rac 4节点，每次启动集群只能启动三个节点，rac1上的数据库实例总是起不来。

2017-11-08 15:02:54.198 [ORAAGENT(24327)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 24327 开头
2017-11-08 15:04:36.473 [ORAAGENT(24441)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 24441 开头
2017-11-08 15:04:39.063 [ORAAGENT(24441)]CRS-5017: 资源操作 "ora.tar.db start" 遇到以下错误: 
2017-11-08 15:04:39.063+ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+data/tar/PARAMETERFILE/spfile.306.958783071'
ORA-17503: ksfdopn:2 Failed to open file +data/tar/PARAMETERFILE/spfile.306.958783071
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 2667
Additional information: 327614466
。有关详细信息, 请参阅 "(:CLSN00107:)" (位于 "/taryartar/12c/grid_base/diag/crs/rac1/crs/trace/crsd_oraagent_oracle.trc" 中)。
2017-11-08 15:04:39.110 [ORAAGENT(24475)]CRS-8500: Oracle Clusterware ORAAGENT 进程以操作系统进程 ID 24475 开头

资源情况如下：

主要是数据库实例在节点1上起不来，这里我有4个节点，只起了2个节点。

查看它指定的文件/taryartar/12c/grid_base/diag/crs/rac1/crs/trace/crsd_oraagent_oracle.trc

一堆看不懂的东西：

2017-11-08 15:04:40.232382 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstConnection::disassociateEdition OCI error 1034
2017-11-08 15:04:40.232430 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection connection count 1
2017-11-08 15:04:40.232435 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection sid  tar1, InstConnection 880597f0
2017-11-08 15:04:40.232441 : USRTHRD:2565469952: {1:20442:1250} InstConnection::breakCall pConnxn:880597f0  DetachLock:00f31fe8 m_pSrvH:880d1d10, m_pSvcH:88108958
2017-11-08 15:04:40.232459 : USRTHRD:2565469952: {1:20442:1250} InstConnection:~InstConnection: this 880597f0
2017-11-08 15:04:40.232787 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection delete InstConnection 880597f0
2017-11-08 15:04:40.232796 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection freed 1
2017-11-08 15:04:40.232804 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::stopConnection
2017-11-08 15:04:40.232813 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection connection count 0
2017-11-08 15:04:40.232817 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection freed 0
2017-11-08 15:04:40.232821 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::stopConnection sid tar1 status  1
2017-11-08 15:04:40.232828 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstAgent::stop db/asm
2017-11-08 15:04:40.232832 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::stopConnection
2017-11-08 15:04:40.232839 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection connection count 0
2017-11-08 15:04:40.232843 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::removeConnection freed 0
2017-11-08 15:04:40.232847 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] ConnectionPool::stopConnection sid tar1 status  1
2017-11-08 15:04:40.232880 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] getResAttrib: attrib name USR_ORA_OPI value false len 5
2017-11-08 15:04:40.232895 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstAgent::stop Container->stop, isShutdown = 0
2017-11-08 15:04:40.232916 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstAgent::stop calling initCssData
2017-11-08 15:04:40.232930 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] CssData Constructor groupName:DBTAR cssRegistrationType:0 memno:-1
2017-11-08 15:04:40.232936 : USRTHRD:2565469952: {1:20442:1250} CssData::initMutex &m_cssDataLock:880724a8
2017-11-08 15:04:40.232960 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] CssGroup::regis registration of groupName:DBTAR mbrid:-1 m_grpNum:-1 regType:0 succeeded
2017-11-08 15:04:40.232967 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] CssData::Constructor regis succeeded groupName:DBTAR memno:-1
2017-11-08 15:04:40.232971 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] CssData::Constructor m_pCssGroup:8807b160 group DBTAR
2017-11-08 15:04:40.232980 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstAgent::init m_groupName:DBTAR m_pCssData 880703e0
2017-11-08 15:04:40.232985 : USRTHRD:2565469952: {1:20442:1250} CssData Destructor groupName:DBTAR
2017-11-08 15:04:40.232991 : USRTHRD:2565469952: {1:20442:1250} CssGroup::dereg default regType:0
2017-11-08 15:04:40.233002 : USRTHRD:2565469952: {1:20442:1250} CssData::destroyMutex &m_cssDataLock:880724a8
2017-11-08 15:04:40.233006 : USRTHRD:2565469952: {1:20442:1250} CssData::Destructor
2017-11-08 15:04:40.233012 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstAgent::stop prev clsagfw_res_status 2 current clsagfw_res_status 2
2017-11-08 15:04:40.233016 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] InstAgent::stop m_prevState:2 }
2017-11-08 15:04:40.233024 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] clean  }
2017-11-08 15:04:40.233037 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [clean] (:CLSN00106:) clsn_agent::clean }
2017-11-08 15:04:40.233045 :    AGFW:2565469952: {1:20442:1250} Command: clean for resource: ora.tar.db 1 1 completed with status: SUCCESS
2017-11-08 15:04:40.233556 :CLSFRAME:2582431328:  TM [MultiThread] is changing desired thread # to 3. Current # is 2
2017-11-08 15:04:40.233604 :CLSFRAME:2582431328:  TM [MultiThread] is changing desired thread # to 4. Current # is 3
2017-11-08 15:04:40.233857 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] InstAgent::checkState gimh oracleSid:tar1
2017-11-08 15:04:40.233864 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] Gimh::constructor ohome:/taryartar/12c/db_base/db_home sid:tar1
2017-11-08 15:04:40.233892 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] ConnectionPool::resetConnection  s_statusOfConnectionMap 00f3aed8
2017-11-08 15:04:40.233897 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] ConnectionPool::resetConnection sid tar1 status  2
2017-11-08 15:04:40.233915 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] Gimh::check OH /taryartar/12c/db_base/db_home SID tar1
2017-11-08 15:04:40.233952 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] Gimh::check condition changes to (GIMH_NEXT_NUM) 1(Instance Shutdown) exists
2017-11-08 15:04:40.233959 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] clsnDbAgent:checkCbk 9 state shutdown shutdownBegin:1 GIMH 1
2017-11-08 15:04:40.233969 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] DbAgent:checkCbk planned offline or failed ret 2
2017-11-08 15:04:40.233998 : USRTHRD:2565469952: {1:20442:1250} DbAgent::DedicatedThread::stop entry
2017-11-08 15:04:40.234002 : USRTHRD:2565469952: {1:20442:1250} DbAgent::DedicatedThread::abortConnxn entry
2017-11-08 15:04:40.234006 : USRTHRD:2565469952: {1:20442:1250} DbAgent::DedicatedThread::abortConnxn exit
2017-11-08 15:04:40.234010 : USRTHRD:2565469952: {1:20442:1250} Thread:DedicatedThread stop {
2017-11-08 15:04:40.234014 : USRTHRD:2565469952: {1:20442:1250} Thread:DedicatedThread Thread stop:thread acquire lock:88023348
2017-11-08 15:04:40.234018 : USRTHRD:2565469952: {1:20442:1250} Thread:DedicatedThread stop release thndMx:88023348 }
2017-11-08 15:04:40.234021 : USRTHRD:2565469952: {1:20442:1250} DbAgent::DedicatedThread::stop exit
2017-11-08 15:04:40.234049 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] DbAgent:checkCbk shutdown reset s_PDBStatusMap
2017-11-08 15:04:40.234427 :    AGFW:2563368704: {1:20442:1250} Agent sending reply for: RESOURCE_CLEAN[ora.tar.db 1 1] ID 4100:6408
2017-11-08 15:04:40.235685 : USRTHRD:2565469952: {1:20442:1250} Gimh::destructor gimh_dest_query_ctx rc=0
2017-11-08 15:04:40.235710 : USRTHRD:2565469952: {1:20442:1250} Gimh::destructor gimh_dest_inst_ctx rc=0
2017-11-08 15:04:40.235727 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] ConnectionPool::stopConnection
2017-11-08 15:04:40.235738 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] ConnectionPool::removeConnection connection count 0
2017-11-08 15:04:40.235744 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] ConnectionPool::removeConnection freed 0
2017-11-08 15:04:40.235748 :CLSDYNAM:2565469952: [ora.tar.db]{1:20442:1250} [check] ConnectionPool::stopConnection sid tar1 status  1
2017-11-08 15:04:40.236176 :    AGFW:2563368704: {1:20442:1250} ora.tar.db 1 1 state changed from: CLEANING to: PLANNED_OFFLINE
2017-11-08 15:04:40.236196 :    AGFW:2563368704: {1:20442:1250} ora.tar.db 1 1 state details has changed from:  to: Instance Shutdown
2017-11-08 15:04:40.236237 :    AGFW:2563368704: {1:20442:1250} Agent sending last reply for: RESOURCE_CLEAN[ora.tar.db 1 1] ID 4100:6408
2017-11-08 15:04:40.237375 :    AGFW:2565469952: {1:20442:1250} Agent has no resources to be monitored, Shutting down ..
2017-11-08 15:04:40.237399 :    AGFW:2565469952: {1:20442:1250} Agent sending message to PE: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:43
2017-11-08 15:04:40.238196 :    AGFW:2563368704: {1:20442:1250} Agent is shutting down.
2017-11-08 15:04:40.238205 :   AGENT:2563368704: {1:20442:1250} Agfw calling user exitCB, will exit on return
2017-11-08 15:04:40.238209 :   AGENT:2563368704: {1:20442:1250} returned from user exitCB, exiting
2017-11-08 15:04:40.238228 :    AGFW:2563368704: {1:20442:1250} Agent is exiting with exit code: 1

我们就不纠结这个了，从最后一个报错信息Linux-x86_64 Error: 13: Permission denied来看，应该是操作系统权限的问题。

起初本能的检查了节点1跟节点2的oracle用户下的$ORACLE_HOME/bin/oracle文件的权限：

节点1：

节点2：

重点看s位，发现权限都一样，宿主也相同。没问题。

再来检查grid用户的$ORACLE_HOME/bin/oracle

节点1：

节点2：

发现节点1的oracle文件比节点2少了两个s位权限，增加之：

然后重新启动节点1的数据库实例：

启动成功，查看资源状态：