本文描述了一起Oracle RAC 11.2.0.4在Red Hat 6.7上遇到的数据库集群宕机事件。在alert.log中，发现与I/O错误相关的问题，如写入失败、读取失败、输入/输出错误等，涉及文件包括redo logs、control files和disk group DATA。故障发生后，通过重启服务器暂时恢复了正常。需要进一步调查以确定故障的具体原因，例如磁盘问题、存储故障或软件配置错误。
数据库版本：
oracle rac 11.2.0.4
系统版本：
red hat 6.7
症状：数据库集群宕机，重启服务器后恢复正常。
alert.Log内容如下：
Auto-tuning: Starting background process GTXa
Starting background process GTXa
Sat Mar 28 21:54:46 2020
GTXa started with pid=241, OS id=45993
Sat Mar 28 22:04:53 2020
Auto-tuning: Shutting down background process GTXa
Sat Mar 28 22:32:17 2020
WARNING: Write Failed. group:1 disk:0 AU:462813 offset:808448 size:1024
Sat Mar 28 22:32:17 2020
WARNING: Read Failed. group:1 disk:0 AU:102 offset:16384 size:16384
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [1.894009738] from disk DATA_0000 allocation unit 102 reason error; if possible, will try another mirror side
Sat Mar 28 22:32:17 2020
WARNING: Read Failed. group:1 disk:0 AU:102 offset:65536 size:16384
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group [1.894009738] from disk DATA_0000 allocation unit 102 reason error; if possible, will try another mirror side
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_lgwr_13581.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 1024
WARNING: failed to write mirror side 1 of virtual extent 150 logical extent 0 of file 324 in group 1 on disk 0 allocation unit 462813
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_lgwr_13581.trc:
ORA-00345: redo log write error block 308779 count 2
ORA-00312: online log 35 thread 1: ‘+DATA/his/redo35.log’
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_ms00_51243.trc:
ORA-00202: control file: ‘+DATA/his/control01.ctl’
ORA-15081: failed to submit an I/O operation to a disk
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 209024
Additional information: -1
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_cs01_50942.trc:
ORA-00202: control file: ‘+DATA/his/control01.ctl’
ORA-15081: failed to submit an I/O operation to a disk
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 208928
Additional information: -1
Sat Mar 28 22:32:17 2020
WARNING: Read Failed. group:1 disk:0 AU:106 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 256 in group [1.894009738] from disk DATA_0000 allocation unit 106 reason error; if possible, will try another mirror side
Sat Mar 28 22:32:17 2020
WARNING: Write Failed. group:1 disk:0 AU:102 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_ckpt_13583.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 256 in group 1 on disk 0 allocation unit 102
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_ckpt_13583.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: ‘+DATA/his/control01.ctl’
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_ckpt_13583.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: ‘+DATA/his/control01.ctl’
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/his/his1/trace/his1_lmon_13539.trc:
ORA-00202: control file: ‘+DATA/his/control01.ctl’
ORA-15081: failed to submit an I/O operation to a disk
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 217184
Additional information: -1
CKPT (ospid: 13583): terminating the instance due to error 221
asm日志如下：
Sat Mar 28 22:32:17 2020
WARNING: Write Failed. group:1 disk:0 AU:15 offset:942080 size:4096
NOTE: unable to write any mirror side for diskgroup DATA
NOTE: cache initiating offline of disk 0 group DATA
NOTE: process lgwr+asm1 (12727) initiating offline of disk 0.3916001663 (DATA_0000) with mask 0x7e in group 1
NOTE: initiating PST update: grp = 1, dsk = 0/0xe969717f, mask = 0x6a, op = clear
Sat Mar 28 22:32:17 2020
NOTE: repairing group 2 file 288 extent 0
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_r000_52616.trc:
ORA-15025: could not open disk “/dev/oracleasm/disks/FRA”
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_r000_52616.trc:
ORA-15025: could not open disk “/dev/oracleasm/disks/FRA”
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
请问有人碰到过这种问题吗？ 是什么原因造成的呢？
扫一扫
872
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
