ASM down, OCR error for 2 nodes RAC.

asm instance cannot start up. crs, evm service down.

2015-04-25 18:05:03.935: [  OCRMSG][1994333200]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2015-04-25 18:05:03.935: [  OCRMSG][1994333200]GIPC error [29] msg [gipcretConnectionRefused]
2015-04-25 18:05:03.935: [  OCRMSG][1994333200]prom_connect: error while waiting for connection complete [24]
2015-04-25 18:05:03.936: [  CRSOCR][1994333200] OCR context init failure.  Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]
2015-04-25 18:05:04.937: [  OCRMSG][1994333200]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2015-04-25 18:05:04.937: [  OCRMSG][1994333200]GIPC error [29] msg [gipcretConnectionRefused]
2015-04-25 18:05:04.937: [  OCRMSG][1994333200]prom_connect: error while waiting for connection complete [24]
2015-04-25 18:05:04.937: [  CRSOCR][1994333200] OCR context init failure.  Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]
2015-04-25 18:05:05.939: [  OCRMSG][1994333200]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2015-04-25 18:05:05.939: [  OCRMSG][1994333200]GIPC error [29] msg [gipcretConnectionRefused]
2015-04-25 18:05:05.939: [  OCRMSG][1994333200]prom_connect: error while waiting for connection complete [24]
2015-04-25 18:05:05.939: [  CRSOCR][1994333200] OCR context init failure.  Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]
2015-04-25 18:05:06.940: [    EVMD][1994333200] Waiting for OCR. Seconds elapsed: 6780
2015-04-25 18:05:06.941: [  OCRMSG][1994333200]prom_waitconnect: CONN NOT ESTABLISHED (0,29,1,2)
2015-04-25 18:05:06.941: [  OCRMSG][1994333200]GIPC error [29] msg [gipcretConnectionRefused]
2015-04-25 18:05:06.941: [  OCRMSG][1994333200]prom_connect: error while waiting for connection complete [24]
2015-04-25 18:05:06.941: [  CRSOCR][1994333200] OCR context init failure.  Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]
2015-04-25 18:05:07.188: [    EVMD][1100499264] EVMD exiting on stop request from default
2015-04-25 18:05:07.188: [    EVMD][1100499264] Done.














root@bss2pdb1 # crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
root@bss2pdb1 # crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
root@bss2pdb1 # crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssd' on 'bss2pdb1'




2015-04-25 08:19:10.407: [ CRSMAIN][3279110640] Getting local node number
2015-04-25 08:19:10.408: [ CRSMAIN][3279110640] Initializing OCR
[   CLWAL][3279110640]clsw_Initialize: OLR initlevel [70000]
2015-04-25 08:19:10.458: [  OCRAPI][3279110640]clsu_get_private_ip_addresses: no ip addresses found.
[  OCRAPI][3279110640]a_init_clsss: failed to call clsu_get_private_ip_addr (7)
2015-04-25 08:19:10.663: [  OCRAPI][3279110640]a_init:13!: Clusterware init unsuccessful : [44]
2015-04-25 08:19:10.664: [  CRSOCR][3279110640] OCR context init failure.  Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2015-04-25 08:19:10.664: [  CRSOCR][3279110640][PANIC] OCR Context is NULL(File: caaocr.cpp, line: 145)


2015-04-25 08:19:10.664: [    CRSD][3279110640][PANIC] CRSD Exiting. OCR Failed
2015-04-25 08:19:10.664: [    CRSD][3279110640] Done.






root@bss2pdb1 # ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
root@bss2pdb1 # ./crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

搞了半天...发现是interconnection IP没了................