1 问题概述
生产环境中,有一套Oracle RAC,突然反映说IO 错误: The Network Adapter could not establish the connection
,并且节点掉了一个。同时备库、单实例库均无法访问。
于是查看trace log进行分析,发现以下报错且不断循环:
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05 WARNING: Write Failed. group:1 disk:0 AU:2112 offset:0 size:173056
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
WARNING: Write Failed. group:1 disk:0 AU:905 offset:721408 size:512Errors in file /u01/app/oracle/diag/rdbms/racdb_stan
b/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05
WARNING: failed to write mirror side 1 of virtual extent 36 logical extent 0 of file 266 in group 1 on disk 0 allocatio
111
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05 NOTE: Suppress further IO Write errors on group:1 disk:0
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_pr00_31669.trc:
ORA-00333: redo log read error block 74851 count 1
ORA-00312: online log 6 thread 2: '+FRA/racdb_standby/onlinelog/redo06.log'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
NOTE: Suppress further IO Write errors on group:1 disk:0
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_34796.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: Read Failed. group:1 disk:0 AU:2111 offset:574976 size:512
Errors in file /u01/app/oracle/diag/rdbms/racdb_standby/racdb/trace/racdb_rfs_101669.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 38 logical extent 0 of file 262 in group 1 on disk 0 allocatio
05
WARNING: failed to write mirror side 1 of virtual extent 37 logical extent 0 of file 266 in group 1 on disk 0 allocatio
112
.......
.......
2 问题解决
经过排查,是由于存储掉电引起的故障,有存储的交换机出现异常导致。
rac环境在存储掉电后,两个节点的服务器都会自动重启。之前的rac环境掉了一个节点是由于服务器光驱内有安装光盘卡住,导致未重启成功。
服务器重启后,rac环境自动重启,单实例环境手动重启。
重启后需要注意:
- 无论单实例还是rac都要检查并重新开启监听
- 如果EM上有shell作业的也需要重新开启EM
- 受影响未执行备份作业的,为确保数据安全需要手动执行全备
- 逐一检查各项状态是否正常,比如存储是否重新挂载、备库是否有gap等