ORACLE集群管理-CRS故障

The start up of CRSD fails with ORA-01017: invalid username/password; logon denied (文档 ID2141978.1)
1. The crsd and evmd are in INTERMEDIATE/OFFLINE status

$crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE node2 Started
ora.cluster_interconnect.haip
1 ONLINE ONLINE node2
ora.crf
1 ONLINE OFFLINE node2
ora.crsd
1 ONLINE INTERMEDIATE node2
ora.cssd
1 ONLINE ONLINE node2
ora.cssdmonitor
1 ONLINE ONLINE node2
ora.ctssd
1 ONLINE ONLINE node2 OBSERVER
ora.diskmon
1 ONLINE ONLINE node2
ora.evmd
1 ONLINE INTERMEDIATE node2
ora.gipcd
1 ONLINE ONLINE node2
ora.gpnpd
1 ONLINE ONLINE node2
ora.mdnsd
1 ONLINE ONLINE node2
-----------
2. All DISKGROUPs are in UNMOUNTED status.
3. <grid_home>/log/<node>/alert<node>.log :
------------
2016-05-11 04:52:02.896:
[cssd(39230)]CRS-1713:CSSD daemon is started in clustered mode
2016-05-11 04:52:23.746:
[cssd(39230)]CRS-1707:Lease acquisition for node node2 number 2 completed
.
2016-05-11 04:52:52.838:
[ohasd(38960)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2016-05-11 04:52:56.125:
[crsd(40040)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in
<grid_home>/log/<node>/crsd/crsd.log.
2016-05-11 04:52:56.131:
[crsd(40040)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while
accessing the physical storage
ORA-01017: invalid username/password; logon denied
]. Details at (:CRSD00111:) in <grid_home>/log/<node>/crsd/crsd.log.
-------------
4. <grid_home>/log/<node>/crsd.log of the problematic node :
-------------------------------------
2016-05-11 04:52:50.433: [ CRSMAIN][2876733168] Initializing OCR
[ CLWAL][2876733168]clsw_Initialize: OLR initlevel [70000]
2016-05-11 04:52:51.709: [ OCRASM][2876733168]proprasmo: Error in open/create file in dg [DBFS_DG]
[ OCRASM][2876733168]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge
2016-05-11 04:52:51.709: [ OCRASM][2876733168]ASM Error Stack : ORA-01017: invalid
username/password; logon denied
2016-05-11 04:52:52.726: [ OCRASM][2876733168]proprasmo: kgfoCheckMount returned [7]
2016-05-11 04:52:52.726: [ OCRASM][2876733168]proprasmo: The ASM instance is down
2016-05-11 04:52:52.726: [ OCRRAW][2876733168]proprioo: Failed to open [+DBFS_DG]. Returned proprasmo()
with [26]. Marking location as UNAVAILABLE.
-------------
5. Connecting to ASM instance of node2 using "sqlplus / as sysasm" as the grid owner also failed with
"ORA-01017: invalid username/password; logon denied".
CHANGES
The Grid has been upgraded from 11.2.0.3 to 11.2.0.4 recently.
CAUSE
The ownership and permission of "<GRID_HOME>/bin/oracle" was not correct, it was owned by root and set-
user/set-group identification permissions was not set.
-rw-r----x 1 root oinstall 291315984 Feb 25 14:10 <GRID_HOME>/bin/oracle
SOLUTION
1. Stop CRS on problematic node.
#crsctl stop crs -f
2. Change the permission and ownership of "<GRID_HOME>/bin/oracle" as below.
#chmod 6751 <GRID_HOME>/bin/oracle
$ ls -lrt <grid_home>/bin/oracle
-rwsr-s--x 1 oracle oinstall 291315984 Feb 25 14:10 <grid_home>/bin/oracle
3. Start CRS
#crsctl start crs