ORA-15081: failed to submit an I/O operation to a disk-存储挂载异常

今天驻地运维的兄弟在QQ上找我，说报告系统的RAC使用不了，Oracle RAC服务无法启动，

现像：

 [oracle@rac1 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon. 

 [oracle@rac2 ~]$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon. 

分析 数据库日志：

日志大量报I/O错误，根据经验这是存储层面出现问题，让驻地运维的兄弟去机房看看存储磁盘是不是挂了，10分钟后告诉我，存储电源线松了，我晕，电源竟然不搞冗余，省几千来块钱没必要了！还好这不是生产环境，不然肯定又被批了！

解决：

我让运维的兄弟先把服务器关掉，存储也关掉，先开存储-->（5分钟左右）开启一台DB服务器-(观察RAC服务器起来后)--->再开启另外一台DB服务

数据库RAC服务正常应用 。

2014-10-23 08:00:04.026000 +08:00

Thread 1 advanced to log sequence 273 (LGWR switch)
  Current log# 1 seq# 273 mem# 0: +DATA/reportdb/onlinelog/group_1.297.861203105
2014-10-23 11:30:15.832000 +08:00
WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:1, diskname:ORCL:DATA disk:0x0.0xe968bd99 au:835804
 iop:0x2b81f05f1000 bufp:0x2b81ef908a00 offset(bytes):876404064256 iosz:16384 operation:1(Read) synchronous:1
 result: 4 osderr:0x3 osderr1:0x2e pid:757
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 300 in group 1 from disk DATA  allocation unit 835804 reason error; if possible,will try another mirror side 
WARNING: IO Failed. subsys:/opt/oracle/extapi/64/asm/orcl/1/libasm.so dg:1, diskname:ORCL:DATA disk:0x0.0xe968bd99 au:835800
 iop:0x2b93fecec000 bufp:0x2b93ff189e00 offset(bytes):876399837184 iosz:16384 operation:1(Read) synchronous:1
 result: 4 osderr:0x3 osderr1:0x2e pid:15697
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 300 in group 1 from disk DATA  a