Remounting filesystem read-only硬盘只读，悲剧

Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813149
Dec  6 00:06:36 backup kernel: Aborting journal on device sda1.
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813150
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813151
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813152
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813153
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813154
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813155
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813156
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813157
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813158
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813159
Dec  6 00:06:36 backup kernel: EXT3-fs error (device sda1): ext3_free_blocks_sb: bit already cleared for block 70813160
Dec  6 00:06:39 backup kernel: EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted
Dec  6 00:06:39 backup kernel: EXT3-fs error (device sda1) in ext3_truncate: Journal has aborted
Dec  6 00:06:39 backup kernel: EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted
Dec  6 00:06:40 backup kernel: EXT3-fs error (device sda1) in ext3_orphan_del: Journal has aborted
Dec  6 00:06:40 backup kernel: EXT3-fs error (device sda1) in ext3_reserve_inode_write: Journal has aborted
Dec  6 00:06:40 backup kernel: EXT3-fs error (device sda1) in ext3_delete_inode: Journal has aborted
Dec  6 00:06:40 backup kernel: ext3_abort called.
Dec  6 00:06:41 backup kernel: EXT3-fs error (device sda1): ext3_journal_start_sb: Detected aborted journal
Dec  6 00:06:41 backup kernel: Remounting filesystem read-only

周六 晚上11点50突然收到报警，一台备份服务器连接不上，这是一台组装的pc机1T*6的硬盘和8G内存，负载几个数据库的备份。
内网，外网都连不上，排除网络问题，估计是机子死机或者其他原因，电话机房帮忙重启下。
服务器重启后， mysql不能同步，查看相关日志(mysqld.log message) 发现系统日志中重启前一切正常，判断应该是掉电或者机房人员操作不当，或者电源问题导致掉电.message总提示有一款sda1的硬盘以只读模式挂载， 导致不能同步master的日志。郁闷中。。。。。。。。
备份该硬盘上的数据库，修改相关路径。删除相关relay.log，再启动start slave 。一切正常。备份数据库已经运行，现在看能不能修复硬盘了。硬盘是使用时间不长，读写也不是很频繁，希望能使用fsck修复。
先切换到root
卸载   umount  /dev/sda1
修复   fsck /dev/sda1
[root@backup-db4 tmp]# fsck /dev/sda1
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/sda1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
STOPPING server from pid file /disk5/data/user_46_3312_2/var/mysqld.pid
091206 01:15:45  mysqld ended





Deleted inode 35127814 has zero dtime.  Fix<y>? yes


Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(70814186--70815212) -(70815895--70816236) -(7249719  4--72498220) -(72680194--72712030) -(72712962--72715015) -(74237465--74238156)
Fix<y>? yes

Free blocks count wrong for group #2161 (0, counted=31259).
Fix<y>? yes

Free blocks count wrong for group #2162 (0, counted=22314).
Fix<y>? yes

Free blocks count wrong for group #2190 (7, counted=27802).
Fix<y>? yes

Free blocks count wrong for group #2191 (0, counted=32051).
Fix<y>? yes

Free blocks count wrong for group #2205 (0, counted=17935).
Fix<y>? yes

Free blocks count wrong for group #2206 (5, counted=32252).
Fix<y>? yes

Free blocks count wrong for group #2207 (6, counted=10377).
Fix<y>? yes

Free blocks count wrong for group #2208 (0, counted=22025).
Fix<y>? yes

Free blocks count wrong for group #2209 (0, counted=177).
Fix<y>? yes

Free blocks count wrong for group #2212 (17076, counted=18103).
Fix<y>? yes

Free blocks count wrong for group #2217 (1194, counted=31812).
Fix<y>? yes

Free blocks count wrong for group #2218 (249, counted=32086).
Fix<y>? yes

Free blocks count wrong for group #2219 (244, counted=2298).
Fix<y>? yes

Free blocks count wrong for group #2265 (14131, counted=14823).
Fix<y>? yes

Free blocks count wrong (176438474, counted=176700876).
Fix<y>? yes

Inode bitmap differences:  -35127814
Fix<y>? yes

Free inodes count wrong for group #2144 (15859, counted=15860).
Fix<y>? yes

Free inodes count wrong (121760134, counted=121760135).
Fix<y>? yes


/dev/sda1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda1: 349817/122109952 files (0.8% non-contiguous), 67489124/244190000 bloc  ks

很庆幸，修复成功，如果fsck修复不成功，则可能是硬盘硬件问题，所以操作之前必须备份。可参考文章最后参考资料

中间遇到的问题：
1 我卸载分区后，再挂载，似乎可以写了文件了，message中出现下面日志
Dec  6 01:07:22 backup kernel: kjournald starting.  Commit interval 5 seconds
Dec  6 01:07:22 backup kernel: EXT3-fs warning (device sda1): ext3_clear_journal_err: Filesystem error recorded from previous
mount: IO failure
Dec  6 01:07:22 backup kernel: EXT3-fs warning (device sda1): ext3_clear_journal_err: Marking fs in need of filesystem check.
Dec  6 01:07:22 backup kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Dec  6 01:07:22 backup kernel: EXT3 FS on sda1, internal journal
Dec  6 01:07:22 backup kernel: EXT3-fs: recovery complete.
Dec  6 01:07:22 backup kernel: EXT3-fs: mounted filesystem with ordered data mode.
2 虽然正常，但是硬盘应该还是有错误，所以建议用e2fsck修复。
3 1T的硬盘fsck花掉20分钟，时间比较长，耐心等待。
4 挂载成功 /dev/sda1             917G  244G  628G  28% /disk3
5 挂载成功显示的message
Dec  6 01:45:13 backup kernel: kjournald starting.  Commit interval 5 seconds
Dec  6 01:45:13 backup kernel: EXT3 FS on sda1, internal journal
Dec  6 01:45:13 backup kernel: EXT3-fs: mounted filesystem with ordered data mode.
6掉电真的很危险，备份很重要（即使是备份库，也很麻烦，心惊胆战）