oracle io lost,磁盘IO故障

测试工作正在如火如荼的进行，突然数据库就连接不上了。我连接上主机发现数据库alert_sid日志中有如下信息：

KCF: write/open error block=0x9a6 online=1

file=2 /oracle_data1/UNDOTBS3.dbf

error=27072 txt: 'Linux Error: 5: Input/output error

Additional information: 2469'

Thu Dec  4 12:56:39 2008

Errors in file /opt/ora9/admin/tax/bdump/orcl_dbw0_9605.trc:

ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode

ORA-01114: IO error writing block to file 2 (block # 2470)

ORA-01110: data file 2: '/oracle_data1/UNDOTBS3.dbf'

ORA-27072: skgfdisp: I/O error

Linux Error: 5: Input/output error

Additional information: 2469

DBW0: terminating instance due to error 1242

Instance terminated by DBW0, pid = 9605

数据库已经down了。初步看是因为磁盘的IO错误。看看主机的日志吧。/var/log/message

Dec  4 12:52:10 tax smartd[2924]: Device: /dev/sdb, 2 Currently unreadable (pending) sectors

Dec  4 12:52:10 tax smartd[2924]: Device: /dev/sdb, 2 Offline uncorrectable sectors

Dec  4 12:56:39 tax kernel: ata1: command 0xca timeout, stat 0xd0 host_stat 0x61

Dec  4 12:56:39 tax kernel: ata1: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00

Dec  4 12:56:39 tax kernel: ata1: status=0xd0 { Busy }

Dec  4 12:56:39 tax kernel: SCSI error : <0 0 1 0> return code = 0x8000002

Dec  4 12:56:39 tax kernel: Info fld=0x5b4b38b, Current sdb: sense key Aborted Command

Dec  4 12:56:39 tax kernel: Additional sense: Scsi parity error

Dec  4 12:56:39 tax kernel: end_request: I/O error, dev sdb, sector 95728523

Dec  4 12:56:39 tax kernel: Buffer I/O error on device sdb6, logical block 1483645

Dec  4 12:56:39 tax kernel: lost page write due to I/O error on sdb6

Dec  4 12:56:39 tax kernel: Aborting journal on device sdb6.

Dec  4 12:56:39 tax kernel: ext3_abort called.

Dec  4 12:56:39 tax kernel: EXT3-fs error (device sdb6): ext3_journal_start_sb: Detected aborted journal

Dec  4 12:56:39 tax kernel: Remounting filesystem read-only

Dec  4 12:57:09 tax kernel: ata1: command 0xca timeout, stat 0xd0 host_stat 0x61

Dec  4 12:57:09 tax kernel: ata1: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00

Dec  4 12:57:09 tax kernel: ata1: status=0xd0 { Busy }

Dec  4 12:57:09 tax kernel: SCSI error : <0 0 1 0> return code = 0x8000002

Dec  4 12:57:09 tax kernel: Info fld=0x5b4b38b, Current sdb: sense key Aborted Command

Dec  4 12:57:09 tax kernel: Additional sense: Scsi parity error

Dec  4 12:57:09 tax kernel: end_request: I/O error, dev sdb, sector 41934794

Dec  4 12:57:09 tax kernel: Buffer I/O error on device sdb3, logical block 643

Dec  4 12:57:09 tax kernel: lost page write due to I/O error on sdb3

Dec  4 12:57:44 tax kernel: ata1: command 0xca timeout, stat 0xd0 host_stat 0x61

Dec  4 12:57:44 tax kernel: ata1: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00

Dec  4 12:57:44 tax kernel: ata1: status=0xd0 { Busy }

Dec  4 12:57:44 tax kernel: SCSI error : <0 0 1 0> return code = 0x8000002

Dec  4 12:57:44 tax kernel: Info fld=0x5b4b38b, Current sdb: sense key Aborted Command

Dec  4 12:57:44 tax kernel: Additional sense: Scsi parity error

Dec  4 12:57:44 tax kernel: end_request: I/O error, dev sdb, sector 83864507

Dec  4 12:57:44 tax kernel: Buffer I/O error on device sdb6, logical block 643

Dec  4 12:57:44 tax kernel: lost page write due to I/O error on sdb6

Dec  4 12:57:44 tax sshd(pam_unix)[11222]: session opened for user oracle by (uid=0)

Dec  4 12:58:03 tax sshd(pam_unix)[11276]: session opened for user oracle by (uid=0)

Dec  4 12:59:25 tax kernel: ata1: command 0xc8 timeout, stat 0xd0 host_stat 0x61

Dec  4 12:59:25 tax kernel: ata1: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00

Dec  4 12:59:25 tax kernel: ata1: status=0xd0 { Busy }

Dec  4 12:59:25 tax kernel: SCSI error : <0 0 1 0> return code = 0x8000002

Dec  4 12:59:25 tax kernel: Info fld=0x5b4b38b, Current sdb: sense key Aborted Command

Dec  4 12:59:25 tax kernel: Additional sense: Scsi parity error

Dec  4 12:59:25 tax kernel: end_request: I/O error, dev sdb, sector 41934794

Dec  4 12:59:25 tax kernel: EXT3-fs error (device sdb3): ext3_get_inode_loc: unable to read inode block - inode=12, block=643

Dec  4 12:59:25 tax kernel: Aborting journal on device sdb3.

Dec  4 12:59:55 tax kernel: ata1: command 0xca timeout, stat 0xd0 host_stat 0x61

Dec  4 12:59:55 tax kernel: ata1: translated ATA stat/err 0xd0/00 to SCSI SK/ASC/ASCQ 0xb/47/00

Dec  4 12:59:55 tax kernel: ata1: status=0xd0 { Busy }

操作系统后台出现严重的IO错误。

但是当进入到某一个分区后，竟然无法创建文件，报错误为只读的文件系统。

[oracle@tax oracle_data2]$ touch aa

touch: cannot touch `aa': Read-only file system

操作系统加载的磁盘方式为rw，全部为读写的方式加载的。

oracle_data1]# mount

/dev/sda5 on / type ext3 (rw)

none on /proc type proc (rw)

none on /sys type sysfs (rw)

none on /dev/pts type devpts (rw,gid=5,mode=620)

usbfs on /proc/bus/usb type usbfs (rw)

/dev/sda1 on /boot type ext3 (rw)

none on /dev/shm type tmpfs (rw)

/dev/sda9 on /opt type ext2 (rw)

/dev/sdb6 on /oracle_data1 type ext3 (rw)

/dev/sdb5 on /oracle_data2 type ext3 (rw)

/dev/sdb3 on /oracle_data3 type ext3 (rw)

/dev/sdb2 on /oracle_data4 type ext3 (rw)

/dev/sdb1 on /oracle_data5 type ext3 (rw)

/dev/sda8 on /oracle_index type ext3 (rw)

/dev/sda7 on /oracle_iot type ext3 (rw)

/dev/sda6 on /oracle_tmp type ext3 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

You have new mail in /var/spool/mail/root

既然是文件系统有问题，那么就修复文件系统吧。使用单用户模式进入系统,单用户就是在系统启动的时候启动项加入single选项。

然后使用fsck修故操作系统，修复完毕后，进入系统正常，因为数据库是自动启动的，只能启动都mount状态，说数据库文件需要恢复，于是recover database，修复完成。直接打开数据库了。

最近怎么磁盘总是出现问题呢？