存储故障时的ORA-7445错误

今天由于存储的磁盘故障，导致了一台服务器的DOWN机，没想到Oracle在出现硬件错误的时候也会报ORA-7445错误。

 

 

在alert文件中，可以找到下面的错误信息：

Mon Jun  2 16:05:40 2008
Errors in file /u1/oracle/admin/repdb01/udump/repdb01_ora_8648.trc:
ORA-07445: exception encountered: core dump [00000001004530E8] [SIGBUS] [Object specific hardware error] [0x103319670] [] []
Mon Jun  2 16:05:40 2008
KCF: write/open error block=0x1557 nline=1
     file=32 /data/oradata/repdb01/undo/UNDOTBS2_1.dbf
     error=27063 txt: 'SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192'
Mon Jun  2 16:05:40 2008
KCF: write/open error block=0x1558 nline=1
     file=32 /data/oradata/repdb01/undo/UNDOTBS2_1.dbf
     error=27063 txt: 'SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192'
Mon Jun  2 16:05:41 2008
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_dbw2_540.trc:
ORA-07445: exception encountered: core dump [0000000101262540] [SIGBUS] [Object specific hardware error] [0x101262540] [] []
ORA-27063: Message 27063 not found;  product=RDBMS; facility=ORA
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192
Mon Jun  2 16:05:42 2008
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_dbw0_536.trc:
ORA-07445: exception encountered: core dump [0000000101262540] [SIGBUS] [Object specific hardware error] [0x101262540] [] []
ORA-27063: Message 27063 not found;  product=RDBMS; facility=ORA
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192
Mon Jun  2 16:05:42 2008
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_lgwr_544.trc:
ORA-00345: Message 345 not found;  product=RDBMS; facility=ORA
; arguments: [840190] [2]
ORA-00312: Message 312 not found;  product=RDBMS; facility=ORA
; arguments: [3] [1] [/data/oradata/repdb01/log/log3.ora]
ORA-27063: Message 27063 not found;  product=RDBMS; facility=ORA
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 1024
Mon Jun  2 16:05:51 2008
Errors in file /u1/oracle/admin/repdb01/udump/repdb01_ora_8668.trc:
ORA-00449: Message 449 not found;  product=RDBMS; facility=ORA
; arguments: [LGWR] [340]
ORA-00340: Message 340 not found;  product=RDBMS; facility=ORA
Mon Jun  2 16:05:51 2008
Errors in file /u1/oracle/admin/repdb01/bdump/repdb01_lgwr_544.trc:
ORA-00340: Message 340 not found;  product=RDBMS; facility=ORA
; arguments: [3] [1]
ORA-00345: Message 345 not found;  product=RDBMS; facility=ORA
; arguments: [840190] [2]
ORA-00312: Message 312 not found;  product=RDBMS; facility=ORA
; arguments: [3] [1] [/data/oradata/repdb01/log/log3.ora]
ORA-27063: Message 27063 not found;  product=RDBMS; facility=ORA
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 1024
Mon Jun  2 16:05:51 2008
LGWR: terminating instance due to error 340
Instance terminated by LGWR, pid = 544

从错误信息不难发现，问题出现在IO系统上，由于Oracle无法执行正常的IO操作，导致数据库最终被LGWR进程中止。

先来看看集中错误信息：

ORA-27063 skgfospo: number of bytes read/written is incorrect
Cause: The number of bytes read/written as returned by AIOWAIT does not match the original number, additional information indicates both these numbers.
Action: Check errno.

ORA-00345 redo log write error block string count string
Cause: An I/O error occurred while writing the log.
Action: Correct the cause of the error, and then restart the system. If the log is lost, apply media/incomplete recovery.

ORA-00340 IO error processing online log string of thread string
Cause: An I/O error occurred on the named online log.
Action: Restore accessibility to the file or restore the file from backup.

ORA-00312 online log string thread string: 'string'
Cause: This message reports the file name for details of another message.
Action: Other messages will accompany this message. See the associated messages for the appropriate action to take.

ORA-00449 background process 'string' unexpectedly terminated with error string
Cause: A foreground process needing service from a background process has discovered the process died.
Action: Refer to the message code given in the message and the trace file for the foreground and the background processes.

可以看到，这些错误都是和IO错误相关，更具体一点，基本上都是由于LGWR进程无法正确的写入数据造成的。

而上面的7445错误就比较奇怪了，检查一下ORA-7445对应的TRACE文件：

bash-2.03$ more /u1/oracle/admin/repdb01/udump/repdb01_ora_8648.trc
/u1/oracle/admin/repdb01/udump/repdb01_ora_8648.trc
Oracle9i Enterprise Edition Release 9.2.0.4.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.4.0 - Production
ORACLE_HOME = /data/oracle/product/920
System name:    SunOS
Node name:      newreport
Release:        5.8
Version:        Generic_117350-26
Machine:        sun4u
Instance name: repdb01
Redo thread mounted by this instance: 1
Oracle process number: 16
Unix process pid: 8648, image: oracle@newreport (TNS V1-V3)

*** 2008-06-02 16:05:40.569
*** SESSION ID:(19.5476) 2008-06-02 16:05:40.303
Exception signal: 10 (SIGBUS), code: 3 (Object specific hardware error), addr: 0x103319670, PC: [0x1004530e8, 00000001004530E8]
*** 2008-06-02 16:05:40.572
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [00000001004530E8] [SIGBUS] [Object specific hardware error] [0x103319670] [] []
Current SQL statement for this session:
SELECT DECODE(null,'','Total System Global Area','') NAME_COL_PLUS_SHOW_SGA,    SUM(VALUE), DECODE (null,'', 'bytes','')  FROM V$SGA
    UNION ALL    SELECT NAME NAME_COL_PLUS_SHOW_SGA , VALUE,    DECODE (null,'', 'bytes','') FROM V$SGA
----- Call Stack Trace -----
calling              call     entry                argument values in hex     
location             type     point                (? means dubious value)    
-------------------- -------- -------------------- ----------------------------
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
ksedmp()+328         CALL     ksedst()+0           FFFFFFFF7FFF4B40 ?
                                                   000000000 ? 000000000 ?
                                                   00000003E ?
                                                   FFFFFFFF7FFF53D8 ?
                                                   1031D56C8 ?
ssexhd()+604         CALL     ksedmp()+0           000000000 ? 000103400 ?
                                                   0001035D9 ? 000102C00 ?
                                                   1035D9000 ? 1035D9C28 ?
sigacthandler()+44   PTR_CALL 0000000000000000     1035E1000 ?
                                                   FFFFFFFF7FFF6370 ?
                                                   000000000 ? 000000001 ?
                                                   1035DEDD8 ? 00000000A ?
00000001004530E8     PTR_CALL 0000000000000000     00000000A ?
                                                   FFFFFFFF7FFF6370 ?
                                                   FFFFFFFF7FFF6090 ?
                                                   00000000A ? 000000000 ?
                                                   004000000 ?
0000000000102C00     FRM_LESS 0000000000000000     000000000 ? 103319670 ?
                                                   000000008 ? 000000000 ?
                                                   000000000 ? 004000000 ?
prsgnt()+36          CALL     0000000100452520     1035D9E88 ? 1035E4398 ?
                                                   000000001 ? 000000003 ?
                                                   692FFEF58 ? 692FF0100 ?
00000001009C2488     CALL     prsgnt()+0           6A1BA0538 ? 103319670 ?
                                                   0010F8000 ? 1035E12A8 ?
                                                   000000000 ? 000000043 ?
kkmfcbfx()+136       CALL     00000001009C21C0     000001200 ? 000001000 ?
                                                   000000000 ? 1035E4398 ?
                                                   1035D9FC8 ?
                                                   FFFFFFFF7CF5EDE8 ?
kkmpfcbk()+236       CALL     kkmfcbfx()+0         FFFFFFFF7FFF80F0 ?
                                                   FFFFFFFF7CF5EDE8 ?
                                                   000000056 ? 000001200 ?
                                                   000001000 ? 000001000 ?
qcsprfro()+544       PTR_CALL 0000000000000000     FFFFFFFF7FFF80F0 ?
                                                   FFFFFFFF7CF5EDE8 ?
                                                   000000006 ? 000000000 ?
                                                   6A1BA0568 ? 000000000 ?
qcspafq()+140        CALL     qcsprfro()+0         000000000 ? 000000043 ?
                                                   000000006 ? 1009C1D40 ?
                                                   000000000 ? 000000000 ?
qcspqb()+252         CALL     qcspafq()+0          000000000 ? 000000000 ?
                                                   000000000 ?
                                                   FFFFFFFF7CF5EA80 ?
                                                   FFFFFFFF7FFF80F0 ?
                                                   FFFFFFFF7CF5EDE8 ?
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
qcspqb()+180         CALL     qcspqb()+0           102DB1000 ? 102DB18C8 ?
                                                   FFFFFFFF7CF5EA80 ?
                                                   FFFFFFFF7FFF80F0 ?
                                                   FFFFFFFF7FFFA400 ?
                                                   6A1BA01E0 ?
qcspqb()+1080        CALL     qcspqb()+0           102DB1000 ? 102DB18C8 ?
                                                   FFFFFFFF7CF5F140 ?
                                                   FFFFFFFF7FFF80F0 ?
                                                   000040040 ? 000000003 ?
00000001009BF1F0     CALL     qcspqb()+0           000040000 ? 000000040 ?
                                                   FFFFFFFF7CF5DA18 ?
                                                   FFFFFFFF7FFF80F0 ?
                                                   000040000 ? 000000000 ?
opiSem()+1024        CALL     00000001009BF1C0     FFFFFFFF7FFFA400 ?
                                                   FFFFFFFF7CF5DA18 ?
                                                   000000000 ? 000000000 ?
                                                   00000000C ? 102DB0000 ?
0000000100FD6D80     CALL     opiSem()+0           1035D9C28 ? 000000000 ?
                                                   000000000 ? 000000000 ?
                                                   FFFFFFFF7FFF8650 ?
                                                   6A1BA01E0 ?
0000000100ACC9E4     CALL     0000000100FD6CC0     FFFFFFFF7CF60490 ?
                                                   000000000 ? 6A1BA01E0 ?
                                                   000000000 ? 000000000 ?
                                                   000000000 ?
0000000100221BE0     PTR_CALL 0000000000000000     102DB0E08 ?
                                                   FFFFFFFF7FFFB330 ?
                                                   1035D9FC8 ? 000000000 ?
                                                   FFFFFFFF7CF690F0 ?
                                                   FFFFFFFF7FFFB330 ?
0000000100ACB3E4     CALL     0000000100221A60     69127B1E8 ? 0001035D9 ?
                                                   1035D9E50 ? 1035D9E88 ?
                                                   000000000 ? 103195000 ?
0000000101BC04CC     PTR_CALL 0000000000000000     6B5DCCDA4 ? 6B5DCCD30 ?
                                                   FFFFFFFF7CF690F0 ?
                                                   6A1BA01E0 ? 1035E1EB4 ?
                                                   1035D9E88 ?
0000000101BBFB2C     CALL     0000000101BC0220     102BE6B38 ? 69DB480A0 ?
                                                   69DB48060 ? 1035D9FC8 ?
                                                   000000001 ? 6A59570E8 ?
kglpim()+236         CALL     0000000101BBF640     008E7F1F7 ? 6B53D22D8 ?
                                                   000000001 ? 000000001 ?
                                                   1036EFF70 ? 000001098 ?
kglpin()+752         CALL     kglpim()+0           000410000 ? 1035D9FC8 ?
                                                   102BE69E0 ? 000000006 ?
                                                   010010000 ? 6B53D22D8 ?

上面的信息是这个ORA-7445错误的完整信息，可以看到，由于IO出现了问题，导致ORA-7445错误很多的内容没有记录完全，连出错的函数信息都没有记录下来。

不过这里还可以出错的SQL语句，这个SQL语句其实是由于我当时正在SQLPLUS中执行SHOW SGA，而这个正好发生了硬件的故障，从而导致了这个7445错误的产生。

而错误信息中的随后的两个ORA-7445错误，是Oracle的DBWR进程尝试写UNDO表空间的数据文件时，出现了IO错误所引发的，其错误信息摘要如下：

*** 2008-06-02 16:05:40.687
*** SESSION ID:(2.1) 2008-06-02 16:05:40.411
KCF: write/open error block=0x1558 nline=1
     file=32 /data/oradata/repdb01/undo/UNDOTBS2_1.dbf
     error=27063 txt: 'SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192'
Exception signal: 10 (SIGBUS), code: 3 (Object specific hardware error), addr: 0x101262540, PC: [0x101262540, 0000000101262540]
*** 2008-06-02 16:05:41.333
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [0000000101262540] [SIGBUS] [Object specific hardware error] [0x101262540] [] []
ORA-27063: Message 27063 not found;  product=RDBMS; facility=ORA
SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192
----- Call Stack Trace -----
calling              call     entry                argument values in hex     
location             type     point                (? means dubious value)    
-------------------- -------- -------------------- ----------------------------
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
Cannot read symbol name in /data/oracle/product/920/bin/oracle.
ksedmp()+328         CALL     ksedst()+0           FFFFFFFF7FFFC970 ?
                                                   000000000 ? 000000000 ?
                                                   00000003E ?
                                                   FFFFFFFF7FFFD208 ?
                                                   1031D56C8 ?
ssexhd()+604         CALL     ksedmp()+0           000000000 ? 000103400 ?
                                                   0001035D9 ? 000102C00 ?
                                                   1035D9000 ? 1035D9C28 ?
sigacthandler()+44   PTR_CALL 0000000000000000     1035E1000 ?
                                                   FFFFFFFF7FFFE1A0 ?
                                                   000000000 ? 000000001 ?
                                                   1035DEDD8 ? 00000000A ?
kcfrfn()+0           PTR_CALL 0000000000000000     00000000A ?
                                                   FFFFFFFF7FFFE1A0 ?
                                                   FFFFFFFF7FFFDEC0 ?
                                                   00000000A ? 000000020 ?
                                                   000000004 ?
000000010126B7F8     FRM_LESS kcfrfn()+0           000000020 ?
                                                   FFFFFFFF7FFFE4C8 ?
                                                   000000020 ? 000000000 ?
                                                   000000020 ? 000000004 ?
kcbbiop()+824        CALL     000000010126B6C0     102DB0E08 ? 1035D9E88 ?
                                                   000000000 ? 000001558 ?
                                                   000000140 ? 1035D9FC8 ?

由于错误本身的特殊性，这个ORA-7445并没有什么参考性，只是记录一下错误情况。

 

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/4227/viewspace-330848/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/4227/viewspace-330848/