rac1启动报错CRS-2674: Start of 'ora.rac.db' on 'rac1' failed

登录rac1，发现实例已经down了，重新启动也报错，如下
+++alter：
Fri Jan 17 01:00:34 2014
ERROR: slave communication error with ASM; terminating process 6443
Errors in file /opt/app/oracle/diag/rdbms/rac/rac1/trace/rac1_lgwr_6443.trc:
+++alterlog
Fri Jan 17 01:00:38 2014
PMON (ospid: 6395): terminating the instance due to error 470
Fri Jan 17 01:00:38 2014
System state dump is made for local instance
System State dumped to trace file /opt/app/oracle/diag/rdbms/rac/rac1/trace/rac1_diag_6405.trc
Fri Jan 17 01:00:41 2014
ORA-1092 : opitsk aborting process
Fri Jan 17 01:00:41 2014
ORA-1092 : opitsk aborting process
Trace dumping is performing id=[cdmp_20140117010038]
Fri Jan 17 01:00:41 2014
License high water mark = 10
Instance terminated by PMON, pid = 6395
USER (ospid: 6619): terminating the instance
Instance terminated by USER, pid = 6619

2014-01-17 01:00:39.523: [    AGFW][2585734032] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:21534
76
2014-01-17 01:00:40.046: [    AGFW][2585734032] Received state change for ora.rac.db 1 1 [old state = ONLINE, new state = FAI
LED]
2014-01-17 01:00:40.046: [    AGFW][2585734032] Received state LABEL change for ora.rac.db 1 1 [old label  = Open, new label
= Abnormal Termination]
2014-01-17 01:00:40.089: [    AGFW][2585734032] Agfw Proxy Server sending message to PE, Contents = [MIDTo:2|OpID:3|FromA:{In
valid|Node:0|Process:0|Type:0}|ToA:{Invalid|Node:-1|Process:-1|Type:-1}|MIDFrom:0|Type:4|Pri2|Id:199745]
2014-01-17 01:00:40.089: [    AGFW][2585734032] Agfw Proxy Server replying to the message: RESOURCE_STATUS[Proxy] ID 20481:21
53476
2014-01-17 01:00:40.183: [   CRSPE][2575227792] State change received from rac1 for ora.rac.db 1 1
2014-01-17 01:00:40.227: [   CRSPE][2575227792] Processing PE command id=10064. Description: [Resource State Change (ora.rac.
db 1 1) : 0x98ef1e10]
2014-01-17 01:00:40.250: [   CRSPE][2575227792] RI [ora.rac.db 1 1] new external state [UNKNOWN] old value: [ONLINE] on rac1
label = [Abnormal Termination] override:OFFLINE
2014-01-17 01:00:40.250: [   CRSPE][2575227792] Set State Details to [Abnormal Termination] from [ Open] for [ora.rac.db 1 1]
2014-01-17 01:00:40.266: [  CRSRPT][2573126544] Publishing event: Cluster Resource State Change Event for ora.rac.db:rac1 : 0
x98e9cc58
2014-01-17 01:00:40.266: [  CRSRPT][2573126544] Publish to eons buffered event : 0x98e9cc58
2014-01-17 01:00:40.333: [   CRSPE][2575227792] Resource Resource Instance ID[ora.rac.db 1 1]. Values:
STATE=UNKNOWN
TARGET=ONLINE
LAST_SERVER=rac1
CURRENT_RCOUNT=0
LAST_RESTART=1389350208
FAILURE_COUNT=0
FAILURE_HISTORY=
STATE_DETAILS=Abnormal Termination
INCARNATION=1
STATE_CHANGE_VERS=10
LAST_FAULT=0
DEGREE_ID=1
ID=ora.rac.db 1 1
CARDINALITY_ID=1
Lock Info:
Write Locks:none
ReadLocks:External state override: OFFLINE|STATE INITED| has failed!
2014-01-17 01:00:40.359: [   CRSPE][2575227792] Processing unplanned state change for [ora.rac.db 1 1]
2014-01-17 01:00:40.363: [   CRSPE][2575227792] ora.rac.db 1 1: uptime exceeds uptime threshold , resetting restart count
2014-01-17 01:00:40.379: [   CRSPE][2575227792] Local Recovery picked for [ora.rac.db 1 1]
2014-01-17 01:00:40.440: [   CRSPE][2575227792] Sending message to agfw: id = 199749
2014-01-17 01:00:40.440: [    AGFW][2585734032] Agfw Proxy Server received the message: RESOURCE_CLEAN[ora.rac.db 1 1] ID 410
0:199749
2014-01-17 01:00:40.450: [   CRSPE][2575227792] CRS-2679: Attempting to clean 'ora.rac.db' on 'rac1'

2014-01-17 01:00:40.567: [    AGFW][2585734032] Agfw Proxy Server forwarding the message: RESOURCE_CLEAN[ora.rac.db 1 1] ID 4
100:199749 to the agent /opt/app/grid/product/11.2.0/bin/oraagent_oracle
2014-01-17 01:00:40.568: [    AGFW][2585734032] Agfw Proxy Server received the message: RESOURCE_PROBE[ora.rac.racs1.svc 1 1]
 ID 4097:199751
2014-01-17 01:00:40.569: [    AGFW][2585734032] Agfw Proxy Server forwarding the message: RESOURCE_PROBE[ora.rac.racs1.svc 1 
1] ID 4097:199751 to the agent /opt/app/grid/product/11.2.0/bin/oraagent_oracle
2014-01-17 01:00:40.769: [    AGFW][2585734032] Agfw Proxy Server received the message: RESOURCE_STATUS[Proxy] ID 20481:21534
92
2014-01-17 01:00:40.769: [    AGFW][2585734032] Received state change for ora.rac.racs1.svc 1 1 [old state = ONLINE, new stat
e = OFFLINE]
2014-01-17 01:00:40.854: [  CRSRPT][2573126544] Publishing event: Cluster Resource Not Restarting Event : 0x98f08db8
2014-01-17 01:00:40.854: [  CRSRPT][2573126544] Publish to eons buffered event : 0x98f08db8
2014-01-17 01:00:41.146: [  CRSCCL][2615118736]clscsend completed:msgTag= 0xcccccccc version= 0 msgType= 0 msgId= 137 msglen
= 5381 clschdr.size_clscmsgh= 5469 src= (1, 4294121890) dest= (2, 4294126300)
2014-01-17 01:00:41.299: [    AGFW][2585734032] Received the reply to the message: RESOURCE_CLEAN[ora.rac.db 1 1] ID 4100:199
752 from the agent /opt/app/grid/product/11.2.0/bin/oraagent_oracle
2014-01-17 01:00:41.300: [    AGFW][2585734032] Agfw Proxy Server sending the reply to PE for message:RESOURCE_CLEAN[ora.rac.
db 1 1] ID 4100:199749
2014-01-17 01:00:41.300: [   CRSPE][2575227792] Received reply to action [Clean] message ID: 199749
2014-01-17 01:00:41.300: [   CRSPE][2575227792] Got agent-specific msg: ORA-01092: ORACLE instance terminated. Disconnection
forced
Process ID: 0
Session ID: 0 Serial number: 0

[grid@rac1 ~]$ srvctl start instance -d rac -i rac1
PRCR-1013 : Failed to start resource ora.rac.db
PRCR-1064 : Failed to start resource ora.rac.db on node rac1
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.rac.db' on 'rac1' failed

[grid@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora....N2.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.eons       ora.eons.type  ONLINE    ONLINE    rac1        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora.rac.db     ora....se.type ONLINE    ONLINE    rac2        
ora....cs1.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....cs2.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac1        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac2        
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    rac1  

2014-01-17 01:00:40.296
[crsd(4403)]CRS-2765:Resource 'ora.rac.db' has failed on server 'rac1'.
2014-01-17 01:00:40.772
[crsd(4403)]CRS-2765:Resource 'ora.rac.racs1.svc' has failed on server 'rac1'.
2014-01-17 01:00:40.801
[crsd(4403)]CRS-2771:Maximum restart attempts reached for resource 'ora.rac.racs1.svc'; will not restart.

[root@rac1 ~]# /opt/app/grid/product/11.2.0/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac2'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac1' succeeded
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'rac1'
CRS-2673: Attempting to stop 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1'
CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded
CRS-2677: Stop of 'ora.eons' on 'rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac1'
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[grid@rac2 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac2        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac2        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora....N2.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac2        
ora.eons       ora.eons.type  ONLINE    ONLINE    rac2        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac2        
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   ONLINE    ONLINE    rac2        
ora.rac.db     ora....se.type ONLINE    ONLINE    rac2        
ora....cs1.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....cs2.svc ora....ce.type ONLINE    ONLINE    rac2        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac2        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac2        
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    rac2     

[root@rac1 ~]# /opt/app/grid/product/11.2.0/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[grid@rac1 crsd]$ srvctl start instance -d rac -i rac1
PRCD-1027 : Failed to retrieve database rac
PRCR-1070 : Failed to check if resource ora.rac.db is registered
Cannot communicate with crsd

[grid@rac1 crsd]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N2.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.eons       ora.eons.type  ONLINE    ONLINE    rac1        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora.rac.db     ora....se.type ONLINE    ONLINE    rac2        
ora....cs1.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....cs2.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac1        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1        
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    rac2        


[grid@rac1 crsd]$ srvctl start instance -d rac -i rac1
PRCR-1013 : Failed to start resource ora.rac.db
PRCR-1064 : Failed to start resource ora.rac.db on node rac1
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.rac.db' on 'rac1' failed

+++突然想起来，前天为了编译kfed工具，改过/opt/app/grid/product/11.2.0/bin目录的权限
可能就是这个导致的
对比rac2的权限，把rac1的权限改回去
[grid@rac2 bin]$ ll -h | grep -v "grid oinstall"
total 1.1G
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 crsd
-rwxr----x 1 root oinstall 101M Nov 30  2012 crsd.bin
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 diskmon
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 diskmonagent
-rwxr----x 1 root oinstall 5.5M Nov 30  2012 diskmonagent.bin
-rwxr----x 1 root oinstall 403K Nov 30  2012 diskmon.bin
-rwsr-x--- 1 root oinstall 1.1M Nov 30  2012 extjob
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 gnsd
-rwxr----x 1 root oinstall 5.5M Nov 30  2012 gnsd.bin
-rwsr-x--- 1 root oinstall  32K Nov 30  2012 jssu
-rwxr-xr-x 1 root oinstall 4.2K Nov 30  2012 oclskd
-rwxr-x--x 1 root oinstall 5.6M Nov 30  2012 oclskd.bin
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 octssd
-rwxr----x 1 root oinstall 5.6M Nov 30  2012 octssd.bin
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 ohasd
-rwxr----x 1 root oinstall 102M Nov 30  2012 ohasd.bin
-rw-r--r-- 1 root root     4.2K Nov 30  2012 oradaemonagent
-rwsr-x--- 1 root oinstall  70K Nov 30  2012 oradism
-rw-r--r-- 1 root root     4.2K Nov 30  2012 oradnssd
-rwxr----x 1 root oinstall 140K Nov 30  2012 oradnssd.bin
-rwxr----x 1 root oinstall 4.2K Nov 30  2012 orarootagent
-rwxr----x 1 root oinstall  34M Nov 30  2012 orarootagent.bin


[root@rac1 ~]# cd /opt/app/grid/product/11.2.0/
[root@rac1 11.2.0]# ll -d bin/
drwxr-xr-x 2 grid oinstall 12288 Jan 16 00:52 bin/
[root@rac1 11.2.0]# chown root:oinstall bin
[root@rac1 11.2.0]# ll -d bin/             
drwxr-xr-x 2 root oinstall 12288 Jan 16 00:52 bin/
[root@rac1 11.2.0]# cd bin/
[root@rac1 bin]# chown root:oinstall  crsd
[root@rac1 bin]# chown root:oinstall  crsd.bin
[root@rac1 bin]# chown root:oinstall  diskmon
[root@rac1 bin]# chown root:oinstall  diskmonagent
[root@rac1 bin]# chown root:oinstall  diskmonagent.bin
[root@rac1 bin]# chown root:oinstall  diskmon.bin
[root@rac1 bin]# chown root:oinstall  extjob
[root@rac1 bin]# chown root:oinstall  gnsd
[root@rac1 bin]# chown root:oinstall  gnsd.bin
[root@rac1 bin]# chown root:oinstall  jssu
[root@rac1 bin]# chown root:oinstall  oclskd
[root@rac1 bin]# chown root:oinstall  oclskd.bin
[root@rac1 bin]# chown root:oinstall  octssd
[root@rac1 bin]# chown root:oinstall  octssd.bin
[root@rac1 bin]# chown root:oinstall  ohasd
[root@rac1 bin]# chown root:oinstall  ohasd.bin
[root@rac1 bin]# chown root:root      oradaemonagent
[root@rac1 bin]# chown root:oinstall  oradism
[root@rac1 bin]# chown root:root      oradnssd
[root@rac1 bin]# chown root:oinstall  oradnssd.bin
[root@rac1 bin]# chown root:oinstall  orarootagent
[root@rac1 bin]# chown root:oinstall  orarootagent.bin


改完权限以后重启crs，再次启动rac2还是报错
[grid@rac1 ~]$ srvctl start instance -d rac -i rac1
PRCR-1013 : Failed to start resource ora.rac.db
PRCR-1064 : Failed to start resource ora.rac.db on node rac1
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.rac.db' on 'rac1' failed

对比rac1和rac2的权限，发现还有一个文件权限不对，而且这个文件比较重要
[grid@rac2 bin]$ ll oracle
-rwsr-s--x 1 grid oinstall 152462728 Nov 30  2012 oracle

[root@rac1 bin]# ll oracle
-rwxr-x--x 1 grid oinstall 152462728 Nov 30  2012 oracle
[root@rac1 bin]# 
[root@rac1 bin]# 
[root@rac1 bin]#  chmod 6755 oracle
[root@rac1 bin]# ll oracle         
-rwsr-sr-x 1 grid oinstall 152462728 Nov 30  2012 oracle

再次启动rac2，已经OK
[grid@rac1 ~]$ srvctl start instance -d rac -i rac1
[grid@rac1 ~]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    rac1        
ora.DATA.dg    ora....up.type ONLINE    ONLINE    rac1        
ora....ER.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N1.lsnr ora....er.type ONLINE    ONLINE    rac1        
ora....N2.lsnr ora....er.type ONLINE    ONLINE    rac2        
ora.asm        ora.asm.type   ONLINE    ONLINE    rac1        
ora.eons       ora.eons.type  ONLINE    ONLINE    rac1        
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type ONLINE    ONLINE    rac1        
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   ONLINE    ONLINE    rac1        
ora.rac.db     ora....se.type ONLINE    ONLINE    rac1        
ora....cs1.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....cs2.svc ora....ce.type ONLINE    ONLINE    rac2        
ora....SM1.asm application    ONLINE    ONLINE    rac1        
ora....C1.lsnr application    ONLINE    ONLINE    rac1        
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    ONLINE    ONLINE    rac1        
ora.rac1.vip   ora....t1.type ONLINE    ONLINE    rac1        
ora....SM2.asm application    ONLINE    ONLINE    rac2        
ora....C2.lsnr application    ONLINE    ONLINE    rac2        
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    ONLINE    ONLINE    rac2        
ora.rac2.vip   ora....t1.type ONLINE    ONLINE    rac2        
ora....ry.acfs ora....fs.type ONLINE    ONLINE    rac1        
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    rac1        

ora.scan2.vip  ora....ip.type ONLINE    ONLINE    rac2      

  

[grid@rac1 ~]$ srvctl status database -d rac
Instance rac1 is running on node rac1

Instance rac2 is running on node rac2