生产环境alert日志持续报错,顿时一惊-IO报错?赶集看下数据库没crash,进行一轮番检查,当前数据库运行正常,长出一口气,分析一下报错。
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_2422.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/192.168.10.1;192.168.10.2/RECOC1_CD_10_xxxx03 at offset 1931402543104 for data length 1040384
ORA-27626: Exadata error: 2201 (IO cancelled due to slow/hung disk)
NOTE: ASM has redirected some slow reads to mirror sides to improve performance.
WARNING: failed to read mirror side 1 of virtual extent 7235 logical extent 0 of file 2509 in group [1.1162375872] from disk DATAC1_CD_08_xxxx04 allocation unit 460482 reason timed-out; if possible, will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 7235 logical extent 1 of file 2509 in group [1.1162375872] from disk DATAC1_CD_07_xxxx03 allocation unit 529539
从字面意思很简单IO出现了问题,从WAENING一行可以看到从DATAC1_CD_08_xxxx04寻找空间超时(time out)并尝试从镜像组获取,并且在DATAC1_CD_07_xxxx03磁盘中找到,好像事情没有那么严重,安全起见还是分析一下。
此报错的原因是由于exadata自11.2.3.31后有延迟写入的特性,延迟写入避免由于io慢引发错误信息,此报错是延迟写入超时后关闭io,尝试其他的磁盘。
从mos(Doc ID 2186724.1)中我们也确定了分析结果:
If you are getting following errors on 11.2.0.4 Database even when the Patch for Bug 19014915 is in place. Make a note fix for Bug 19014915 is already included in QUARTERLY
DATABASE PATCH FOR EXADATA (JAN2015 - 11.2.0.4.15) - Patch 20059387 as per
Bug Fix List: the 11.2.0.4 Patch Bundles for Oracle Exadata Database Machine Note 1601749.1
SOLUTION
1) You can IGNORE these errors , they are NOT Harmful if Exadata CELL (storage) nodes do NOT have any physical/logical disk errors .
2) You can apply the Patch 22468034 in case you do NOT want to see the errors in Alert log.
Make a note this issue is applicable ONLY for 11.2.0.4 database after the Patch for Bug 19014915 is in place
检查数据库补丁:
[oracle@node1 OPatch]$ ./opatch lsinv| grep 19014915
16804061, 18783969, 17571039, 14263190, 18798414, 21067387, 19014915
[oracle@node1 OPatch]$
总结:
对于11.2.0.4 数据库如果已经打了补丁19014915,如果在出现这个报错可以忽略(IGNORE ),也可以通过patch22468034解决。
Oracle exadata也不是那么完善,也是在生产中不断补丁和迭代的。