exadata的紧张一刻-ORA-27626: Exadata error: 2201 (IO cancelled due to slow/hung disk)

生产环境alert日志持续报错，顿时一惊-IO报错？赶集看下数据库没crash，进行一轮番检查，当前数据库运行正常，长出一口气，分析一下报错。

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_2422.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.10.1;192.168.10.2/RECOC1_CD_10_xxxx03 at offset 1931402543104 for data length 1040384
ORA-27626: Exadata error: 2201 (IO cancelled due to slow/hung disk)
NOTE: ASM has redirected some slow reads to mirror sides to improve performance.
WARNING: failed to read mirror side 1 of virtual extent 7235 logical extent 0 of file 2509 in group [1.1162375872] from disk DATAC1_CD_08_xxxx04  allocation unit 460482 reason timed-out; if possible, will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 7235 logical extent 1 of file 2509 in group [1.1162375872] from disk DATAC1_CD_07_xxxx03 allocation unit 529539

从字面意思很简单IO出现了问题，从WAENING一行可以看到从DATAC1_CD_08_xxxx04寻找空间超时（time out）并尝试从镜像组获取，并且在DATAC1_CD_07_xxxx03磁盘中找到，好像事情没有那么严重，安全起见还是分析一下。

此报错的原因是由于exadata自11.2.3.31后有延迟写入的特性，延迟写入避免由于io慢引发错误信息，此报错是延迟写入超时后关闭io，尝试其他的磁盘。

从mos（Doc ID 2186724.1）中我们也确定了分析结果：

If you are getting following errors on 11.2.0.4 Database even when the Patch for Bug 19014915 is in place. Make a note fix for Bug 19014915 is already included in QUARTERLY
DATABASE PATCH FOR EXADATA (JAN2015 - 11.2.0.4.15) - Patch 20059387 as per
Bug Fix List: the 11.2.0.4 Patch Bundles for Oracle Exadata Database Machine Note 1601749.1
 

SOLUTION
1) You can IGNORE these errors , they are NOT Harmful if Exadata CELL (storage) nodes do NOT have any physical/logical disk errors .
2) You can apply the Patch 22468034 in case you do NOT want to see the errors in Alert log.
Make a note this issue is applicable ONLY for 11.2.0.4 database after the Patch for Bug 19014915 is in place
检查数据库补丁：

[oracle@node1 OPatch]$ ./opatch lsinv| grep 19014915 
     16804061, 18783969, 17571039, 14263190, 18798414, 21067387, 19014915
[oracle@node1 OPatch]$ 

 

总结：

对于11.2.0.4 数据库如果已经打了补丁19014915，如果在出现这个报错可以忽略（IGNORE ），也可以通过patch22468034解决。

Oracle exadata也不是那么完善，也是在生产中不断补丁和迭代的。