有一套Oracle RAC,突然反映说不能对部分表格进行读写操作了,但是数据库正常,大部分数据读写也正常,先将服务切至另一台Server使Oracle对外正常服务,然后开始troubleshooting。
HP-UX系统层面没有问题,存储3par没有预警,确认链路正常,以上步骤是为了确认不是个别共享存储出问题导致的数据无法访问。然后查看trace log进行分析(/oracle/app/diag/rdbms/mesdb/mesdb2/trace/alter_mesdb2.log),发现以下报错且不断循环:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 713 logical extent 0 of file 532 in group 9 on disk 0 allocation unit 595268
Fri Apr 11 12:13:00 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 66 logical extent 0 of file 310 in group 9 on disk 1 allocation unit 171803
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 66 logical extent 0 of file 310 in group 9 on disk 1 allocation unit 171803
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors i