2013年2月28日 星期四




磁盘阵列



一、准备工作



1、创建磁盘



[root@desktop7 ~]# fdisk /dev/sda



The number of cylinders for this disk is set to 60801.


There is nothing wrong with that, but this is larger than 1024,


and could in certain setups cause problems with:


1) software that runs at boot time (e.g., old versions of LILO)


2) booting and partitioning software from other OSs


   (e.g., DOS FDISK, OS/2 FDISK)



Command (m for help): p



Disk /dev/sda: 500.1 GB, 500107862016 bytes


255 heads, 63 sectors/track, 60801 cylinders


Units = cylinders of 16065 * 512 = 8225280 bytes



   Device Boot      Start         End      Blocks   Id  System


/dev/sda1   *           1          13      104391   83  Linux


/dev/sda2              14        6387    51199155   8e  Linux LVM


/dev/sda3            6388       12761    51199155   8e  Linux LVM



Command (m for help): n


Command action


   e   extended


   p   primary partition (1-4)


e


Selected partition 4


First cylinder (12762-60801, default 12762):


Using default value 12762


Last cylinder or +size or +sizeM or +sizeK (12762-60801, default 60801):


Using default value 60801



Command (m for help): n


First cylinder (12762-60801, default 12762):


Using default value 12762


Last cylinder or +size or +sizeM or +sizeK (12762-60801, default 60801): +100M



Command (m for help): n


First cylinder (12775-60801, default 12775):


Using default value 12775


Last cylinder or +size or +sizeM or +sizeK (12775-60801, default 60801): +100M



Command (m for help): n


First cylinder (12788-60801, default 12788):


Using default value 12788


Last cylinder or +size or +sizeM or +sizeK (12788-60801, default 60801): +100M



Command (m for help): n


First cylinder (12801-60801, default 12801):


Using default value 12801


Last cylinder or +size or +sizeM or +sizeK (12801-60801, default 60801): +100M



Command (m for help): n


First cylinder (12814-60801, default 12814):


Using default value 12814


Last cylinder or +size or +sizeM or +sizeK (12814-60801, default 60801): +100M



Command (m for help): p



Disk /dev/sda: 500.1 GB, 500107862016 bytes


255 heads, 63 sectors/track, 60801 cylinders


Units = cylinders of 16065 * 512 = 8225280 bytes



   Device Boot      Start         End      Blocks   Id  System


/dev/sda1   *           1          13      104391   83  Linux


/dev/sda2              14        6387    51199155   8e  Linux LVM


/dev/sda3            6388       12761    51199155   8e  Linux LVM


/dev/sda4           12762       60801   385881300    5  Extended


/dev/sda5           12762       12774      104391   83  Linux


/dev/sda6           12775       12787      104391   83  Linux


/dev/sda7           12788       12800      104391   83  Linux


/dev/sda8           12801       12813      104391   83  Linux


/dev/sda9           12814       12826      104391   83  Linux



2、修改标识



Command (m for help): t


Partition number (1-9): 5


Hex code (type L to list codes): fd


Changed system type of partition 5 to fd (Linux raid autodetect)



Command (m for help): t


Partition number (1-9): 6


Hex code (type L to list codes): fd


Changed system type of partition 6 to fd (Linux raid autodetect)



Command (m for help): t


Partition number (1-9): 7


Hex code (type L to list codes): fd


Changed system type of partition 7 to fd (Linux raid autodetect)



Command (m for help): t


Partition number (1-9): 8


Hex code (type L to list codes): fd


Changed system type of partition 8 to fd (Linux raid autodetect)



Command (m for help): t


Partition number (1-9): 9


Hex code (type L to list codes): fd


Changed system type of partition 9 to fd (Linux raid autodetect)



Command (m for help): p



Disk /dev/sda: 500.1 GB, 500107862016 bytes


255 heads, 63 sectors/track, 60801 cylinders


Units = cylinders of 16065 * 512 = 8225280 bytes



   Device Boot      Start         End      Blocks   Id  System


/dev/sda1   *           1          13      104391   83  Linux


/dev/sda2              14        6387    51199155   8e  Linux LVM


/dev/sda3            6388       12761    51199155   8e  Linux LVM


/dev/sda4           12762       60801   385881300    5  Extended


/dev/sda5           12762       12774      104391   fd  Linux raid autodetect


/dev/sda6           12775       12787      104391   fd  Linux raid autodetect


/dev/sda7           12788       12800      104391   fd  Linux raid autodetect


/dev/sda8           12801       12813      104391   fd  Linux raid autodetect


/dev/sda9           12814       12826      104391   fd  Linux raid autodetect



3、保存退出



Command (m for help): w


The partition table has been altered!



Calling ioctl() to re-read partition table.



WARNING: Re-reading the partition table failed with error 16: Device or resource busy.


The kernel still uses the old table.


The new table will be used at the next reboot.


Syncing disks.



4、同步



[root@desktop7 ～]# partprobe


[root@desktop7 ～]#



二、创建多重磁盘设备



[root@desktop7 ～]# mdadm -C /dev/md0 -l5 -n3 -x1 /dev/sda{5,6,7,8}


mdadm: /dev/sda5 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 04:56:51 2013


mdadm: /dev/sda6 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 04:56:51 2013


mdadm: /dev/sda7 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 04:56:51 2013


mdadm: /dev/sda8 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 04:56:51 2013


Continue creating array? y


mdadm: array /dev/md0 started.


[root@desktop7 ～]# mdadm --detail /dev/md0


/dev/md0:


        Version : 0.90


  Creation Time : Thu Feb 28 06:04:37 2013


     Raid Level : raid5


     Array Size : 208640 (203.78 MiB 213.65 MB)


  Used Dev Size : 104320 (101.89 MiB 106.82 MB)


   Raid Devices : 3


  Total Devices : 4


Preferred Minor : 0


    Persistence : Superblock is persistent



    Update Time : Thu Feb 28 06:04:42 2013


          State : clean


 Active Devices : 3


Working Devices : 4


 Failed Devices : 0


  Spare Devices : 1



         Layout : left-symmetric


     Chunk Size : 64K



           UUID : 275977ae:d8cfe94b:a71f95e4:bbdacf12


         Events : 0.4



    Number   Major   Minor   RaidDevice State


       0       8        5        0      active sync   /dev/sda5


       1       8        6        1      active sync   /dev/sda6


       2       8        7        2      active sync   /dev/sda7



       3       8        8        -      spare   /dev/sda8


[root@desktop7 ～]#



三、管理RAID卷



1、格式化磁盘



[root@desktop7 ~]# mkfs.ext3 /dev/md0


mke2fs 1.39 (29-May-2006)


Filesystem label=


OS type: Linux


Block size=1024 (log=0)


Fragment size=1024 (log=0)


52208 inodes, 208640 blocks


10432 blocks (5.00%) reserved for the super user


First data block=1


Maximum filesystem blocks=67371008


26 block groups


8192 blocks per group, 8192 fragments per group


2008 inodes per group


Superblock backups stored on blocks:


        8193, 24577, 40961, 57345, 73729, 204801



Writing inode tables: done                            


Creating journal (4096 blocks): done


Writing superblocks and filesystem accounting information: done



This filesystem will be automatically checked every 24 mounts or


180 days, whichever comes first.  Use tune2fs -c or -i to override.


[root@desktop7 ~]#



2、挂载RAID磁盘



[root@desktop7 ~]# mkdir /raid5


[root@desktop7 ~]# mount /dev/md0 /raid5


[root@desktop7 ~]# mount


/dev/mapper/Vol0-vol_root on / type ext3 (rw)


proc on /proc type proc (rw)


sysfs on /sys type sysfs (rw)


devpts on /dev/pts type devpts (rw,gid=5,mode=620)


/dev/mapper/virt-station on /var/lib/station type ext3 (rw)


/dev/mapper/virt-windows on /var/lib/windows type ext3 (rw)


/dev/mapper/virt-server on /var/lib/server type ext3 (rw)


/dev/sda1 on /boot type ext3 (rw)


tmpfs on /dev/shm type tmpfs (rw)


none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)


sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)


/dev/md0 on /raid5 type ext3 (rw)


[root@desktop7 ~]#



3、在RAID卷中写入数据查看状态



[root@desktop7 ~]# cp -a /etc/ /raid5


[root@desktop7 ~]# df /raid5


Filesystem           1K-blocks      Used Available Use% Mounted on


/dev/md0                202047    123907     67708  65% /raid5


[root@desktop7 ~]# du -sh /raid5


118M    /raid5


[root@desktop7 ~]#



4、模拟磁盘损坏



[root@desktop7 ~]# mdadm --manage /dev/md0 --fail /dev/sda5


mdadm: set /dev/sda5 faulty in /dev/md0



[root@desktop7 ~]# mdadm --detail /dev/md0


/dev/md0:


        Version : 0.90


  Creation Time : Thu Feb 28 06:04:37 2013


     Raid Level : raid5


     Array Size : 208640 (203.78 MiB 213.65 MB)


  Used Dev Size : 104320 (101.89 MiB 106.82 MB)


   Raid Devices : 3


  Total Devices : 4


Preferred Minor : 0


    Persistence : Superblock is persistent



    Update Time : Thu Feb 28 06:43:15 2013


          State : clean


 Active Devices : 3


Working Devices : 3


 Failed Devices : 1


  Spare Devices : 0



         Layout : left-symmetric


     Chunk Size : 64K



           UUID : 275977ae:d8cfe94b:a71f95e4:bbdacf12


         Events : 0.10



    Number   Major   Minor   RaidDevice State


       0       8        8        0      active sync   /dev/sda8


       1       8        6        1      active sync   /dev/sda6


       2       8        7        2      active sync   /dev/sda7



       3       8        5        -      faulty spare   /dev/sda5


[root@desktop7 ~]#



5、增加新磁盘，移除损坏磁盘



[root@desktop7 ~]# mdadm --manage /dev/md0 --add /dev/sda9


mdadm: added /dev/sda9



[root@desktop7 ~]# mdadm --manage /dev/md0 --remove /dev/sda5


mdadm: hot removed /dev/sda5



查看状态



[root@desktop7 ~]# mdadm --detail /dev/md0


/dev/md0:


        Version : 0.90


  Creation Time : Thu Feb 28 06:04:37 2013


     Raid Level : raid5


     Array Size : 208640 (203.78 MiB 213.65 MB)


  Used Dev Size : 104320 (101.89 MiB 106.82 MB)


   Raid Devices : 3


  Total Devices : 4


Preferred Minor : 0


    Persistence : Superblock is persistent



    Update Time : Thu Feb 28 06:48:29 2013


          State : clean


 Active Devices : 3


Working Devices : 4


 Failed Devices : 0


  Spare Devices : 1



         Layout : left-symmetric


     Chunk Size : 64K



           UUID : 275977ae:d8cfe94b:a71f95e4:bbdacf12


         Events : 0.14



    Number   Major   Minor   RaidDevice State


       0       8        8        0      active sync   /dev/sda8


       1       8        6        1      active sync   /dev/sda6


       2       8        7        2      active sync   /dev/sda7



       3       8        9        -      spare   /dev/sda9


[root@desktop7 ~]#



6、停用与启用RAID磁盘



（1）停用磁盘



[root@desktop7 ~]# mdadm --stop /dev/md0


mdadm: fail to stop array /dev/md0: Device or resource busy


Perhaps a running process, mounted filesystem or active volume group?



（2）先卸载磁盘



[root@desktop7 ~]# umount /raid5



（3）再停用磁盘



[root@desktop7 ~]# mdadm --stop /dev/md0


mdadm: stopped /dev/md0



（4）查看状态



[root@desktop7 ~]# mdadm --detail /dev/md0


mdadm: md device /dev/md0 does not appear to be active.


[root@desktop7 ~]#



（5）启用磁盘



[root@desktop7 ~]# mdadm --assemble --scan /dev/md0


mdadm: /dev/md0 not identified in config file.



未配置初始配置文件/etc/mdadm.conf,所以文件信息全部丢失。



重做RAID卷



[root@desktop7 ~]# mdadm -C /dev/md0 -l5 -n3 -x1 /dev/sda{5,6,7,8}


mdadm: /dev/sda5 appears to contain an ext2fs file system


    size=208640K  mtime=Thu Feb 28 06:20:21 2013


mdadm: /dev/sda5 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 06:04:37 2013


mdadm: /dev/sda6 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 06:04:37 2013


mdadm: /dev/sda7 appears to contain an ext2fs file system


    size=216897K  mtime=Thu Feb 28 06:20:21 2013


mdadm: /dev/sda7 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 06:04:37 2013


mdadm: /dev/sda8 appears to contain an ext2fs file system


    size=208640K  mtime=Thu Feb 28 06:20:21 2013


mdadm: /dev/sda8 appears to be part of a raid array:


    level=raid5 devices=3 ctime=Thu Feb 28 06:04:37 2013


Continue creating array? y


mdadm: array /dev/md0 started.


[root@desktop7 ~]# mdadm --detail /dev/md0


/dev/md0:


        Version : 0.90


  Creation Time : Thu Feb 28 07:22:59 2013


     Raid Level : raid5


     Array Size : 208640 (203.78 MiB 213.65 MB)


  Used Dev Size : 104320 (101.89 MiB 106.82 MB)


   Raid Devices : 3


  Total Devices : 4


Preferred Minor : 0


    Persistence : Superblock is persistent



    Update Time : Thu Feb 28 07:23:04 2013


          State : clean


 Active Devices : 3


Working Devices : 4


 Failed Devices : 0


  Spare Devices : 1



         Layout : left-symmetric


     Chunk Size : 64K



           UUID : 0a03e130:1c27d665:866c94c8:a5f5b144


         Events : 0.4



    Number   Major   Minor   RaidDevice State


       0       8        5        0      active sync   /dev/sda5


       1       8        6        1      active sync   /dev/sda6


       2       8        7        2      active sync   /dev/sda7



       3       8        8        -      spare   /dev/sda8


[root@desktop7 ~]#



[root@desktop7 ~]# mkfs -t ext3 /dev/md0


mke2fs 1.39 (29-May-2006)


Filesystem label=


OS type: Linux


Block size=1024 (log=0)


Fragment size=1024 (log=0)


52208 inodes, 208640 blocks


10432 blocks (5.00%) reserved for the super user


First data block=1


Maximum filesystem blocks=67371008


26 block groups


8192 blocks per group, 8192 fragments per group


2008 inodes per group


Superblock backups stored on blocks:


        8193, 24577, 40961, 57345, 73729, 204801



Writing inode tables: done                            


Creating journal (4096 blocks): done


Writing superblocks and filesystem accounting information: done



This filesystem will be automatically checked every 36 mounts or


180 days, whichever comes first.  Use tune2fs -c or -i to override.


[root@desktop7 ~]# mount /dev/md0 /raid5


[root@desktop7 ~]# mount


/dev/mapper/Vol0-vol_root on / type ext3 (rw)


proc on /proc type proc (rw)


sysfs on /sys type sysfs (rw)


devpts on /dev/pts type devpts (rw,gid=5,mode=620)


/dev/mapper/virt-station on /var/lib/station type ext3 (rw)


/dev/mapper/virt-windows on /var/lib/windows type ext3 (rw)


/dev/mapper/virt-server on /var/lib/server type ext3 (rw)


/dev/sda1 on /boot type ext3 (rw)


tmpfs on /dev/shm type tmpfs (rw)


none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)


sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)


/dev/md0 on /raid5 type ext3 (rw)


[root@desktop7 ~]#


[root@desktop7 ~]# mdadm --detail /dev/md0


/dev/md0:


        Version : 0.90


  Creation Time : Thu Feb 28 07:22:59 2013


     Raid Level : raid5


     Array Size : 208640 (203.78 MiB 213.65 MB)


  Used Dev Size : 104320 (101.89 MiB 106.82 MB)


   Raid Devices : 3


  Total Devices : 4


Preferred Minor : 0


    Persistence : Superblock is persistent



    Update Time : Thu Feb 28 07:27:59 2013


          State : clean


 Active Devices : 3


Working Devices : 4


 Failed Devices : 0


  Spare Devices : 1



         Layout : left-symmetric


     Chunk Size : 64K



           UUID : 0a03e130:1c27d665:866c94c8:a5f5b144


         Events : 0.4



    Number   Major   Minor   RaidDevice State


       0       8        5        0      active sync   /dev/sda5


       1       8        6        1      active sync   /dev/sda6


       2       8        7        2      active sync   /dev/sda7



       3       8        8        -      spare   /dev/sda8


[root@desktop7 ~]#


[root@desktop7 ~]#



7、监控RAID磁盘



[root@desktop7 ~]# service mdmonitor status


mdadm is stopped


[root@desktop7 ~]# service mdmonitor start


[root@desktop7 ~]# service mdmonitor status


mdadm is stopped


[root@desktop7 ~]#