1.添加硬盘
这里我添加了6块1G的硬盘做实验,具体作用如下:
4块做RAID5,1块做RAID5的备用盘(如果RAID5其中一个硬盘坏了,就备用盘会自动替换坏的硬盘),1块实验备用盘
添加好硬盘后,需要重启服务器才能在/dev中看到相关硬盘的信息,如果是热插,需要用下面命令刷新硬盘信息:
[root@localhost a]# ls /dev/sd* --查看硬盘信息,可以看到6个硬盘没有显示出来
/dev/sda /dev/sda1 /dev/sda2
[root@localhost a]# echo "- - -" > /sys/class/scsi_host/host --此时用tab键可以看到目录下有3个host目录
host0/ host1/ host2/
[root@localhost a]# echo "- - -" > /sys/class/scsi_host/host0/scan
[root@localhost a]# echo "- - -" > /sys/class/scsi_host/host1/scan
[root@localhost a]# echo "- - -" > /sys/class/scsi_host/host2/scan
[root@localhost a]# ls /dev/sd* --可以看到6个硬盘已经显示出来了
sda sda1 sda2 sdb sdc sdd sde sdf sdg
2.创建RAID5
安装RAID管理命令
[root@localhost a]# yum -y install mdadm
Loaded plugins: fastestmirror
Determining fastest mirrors
centos7 | 3.6 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package mdadm.x86_64 0:4.0-13.el7 will be installed
--> Processing Dependency: libreport-filesystem for package: mdadm-4.0-13.el7.x86_64
--> Running transaction check
---> Package libreport-filesystem.x86_64 0:2.1.11-40.el7.centos will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===============================================================================================================================================
Package Arch Version Repository Size
===============================================================================================================================================
Installing:
mdadm x86_64 4.0-13.el7 centos7 431 k
Installing for dependencies:
libreport-filesystem x86_64 2.1.11-40.el7.centos centos7 39 k
Transaction Summary
===============================================================================================================================================
Install 1 Package (+1 Dependent package)
Total download size: 470 k
Installed size: 1.0 M
Downloading packages:
-----------------------------------------------------------------------------------------------------------------------------------------------
Total 2.3 MB/s | 470 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libreport-filesystem-2.1.11-40.el7.centos.x86_64 1/2
Installing : mdadm-4.0-13.el7.x86_64 2/2
Verifying : libreport-filesystem-2.1.11-40.el7.centos.x86_64 1/2
Verifying : mdadm-4.0-13.el7.x86_64 2/2
Installed:
mdadm.x86_64 0:4.0-13.el7
Dependency Installed:
libreport-filesystem.x86_64 0:2.1.11-40.el7.centos
Complete!
[root@localhost a]#
执行创建命令:
mdadm --create --auto=yes /dev/md9 --level=5 --raid-devices=4 --spare-devices=1 /dev/sd[b-f]
[root@localhost a]# mdadm --create --auto=yes /dev/md9 --level=5 --raid-devices=4 --spare-devices=1 /dev/sd[b-f] --此命令可缩写为:mdamd -C /dev/md9 -l5 -n4 -x1 /dev/sd[b-f] ,其中md9可以自定义mdX(1-9),sd[b-f]指的是sdb,sdc,sdd,sde,sdf
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md9 started.
显示命令执行成功
如果是硬盘大的话,这个过程可能会很长,一般100G以上一个硬盘就需要看执行进度,命令如下:
[root@localhost a]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md9 : active raid5 sde[5] sdf[4](S) sdd[2] sdc[1] sdb[0]
3139584 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
unused devices: <none>
这个命令是静态显示,需要动态显示进度可以用while循环具体如下:
[root@localhost ~]# while true;do cat /proc/mdstat ;sleep 1 ;done
其效果就是每1秒执行一次 cat /proc/mdstat
两个命令查看创建结果
mdadm -Q /dev/md9
mdadm --detail /dev/md9
[root@localhost ~]# mdadm -Q /dev/md9 --结果简单
/dev/md9: 2.99GiB raid5 4 devices, 1 spare. Use mdadm --detail for more detail.
[root@localhost ~]# mdadm --detail /dev/md9 --结果详细
/dev/md9:
Version : 1.2
Creation Time : Wed Jul 28 10:37:19 2021
Raid Level : raid5
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Wed Jul 28 10:37:25 2021
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Name : localhost.localdomain:9 (local to host localhost.localdomain)
UUID : 850aead9:49d1879a:551d055e:8e68d53c
Events : 18
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
5 8 64 3 active sync /dev/sde
4 8 80 - spare /dev/sdf
创建文件系统
mkfs.xfs /dev/md9
[root@localhost ~]# mkfs.xfs /dev/md9
meta-data=/dev/md9 isize=512 agcount=8, agsize=98176 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=784896, imaxpct=25
= sunit=128 swidth=384 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
创建挂载
临时:
[root@localhost ~]# mkdir /test #创建挂载目录
[root@localhost ~]# mount /dev/md9 /test #把创建好的DAID5磁盘阵列挂在/test下(test的存储空间就是RAID5的容量)
[root@localhost ~]# df -hT #显示挂载信息
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 48G 1.2G 47G 3% /
devtmpfs devtmpfs 476M 0 476M 0% /dev
tmpfs tmpfs 488M 0 488M 0% /dev/shm
tmpfs tmpfs 488M 7.7M 480M 2% /run
tmpfs tmpfs 488M 0 488M 0% /sys/fs/cgroup
/dev/sr0 iso9660 4.2G 4.2G 0 100% /mnt/cdrom
/dev/sda1 xfs 497M 123M 375M 25% /boot
tmpfs tmpfs 98M 0 98M 0% /run/user/0
/dev/md9 xfs 3.0G 33M 3.0G 2% /test
自动(开机自动挂载):
修改挂载文件:
[root@localhost ~]# vim /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Jul 7 20:23:11 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=ab5e5af0-f0af-47b2-a0cc-47b8631e4e61 /boot xfs defaults 0 0
/dev/mapper/centos-swap swap swap defaults 0 0
/dev/sr0 /mnt/cdrom iso9660 defaults 0 0
/dev/md9 /test xfs defaults 0 0 #添加此行
创建并修改RAID开机启动文件
[root@localhost ~]# mdadm --detail /dev/md9 |grep UUID > /etc/mdadm.conf
[root@localhost ~]# vim /etc/mdadm.conf
ARRAY /dev/md9 UUID=850aead9:49d1879a:551d055e:8e68d53c #添加内容
然后模拟损坏一个硬盘,查看RAID5工作原理
mdadm --manage /dev/md9 --fail /dev/sdb
[root@localhost ~]# mdadm --manage /dev/md9 --fail /dev/sdb # 模拟损坏sdb硬盘
mdadm: set /dev/sdb faulty in /dev/md9
[root@localhost ~]# mdadm --detail /dev/md9 #查看重建过程
/dev/md9:
Version : 1.2
Creation Time : Wed Jul 28 10:37:19 2021
Raid Level : raid5
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Wed Jul 28 12:28:26 2021
State : clean, degraded, recovering
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Rebuild Status : 69% complete
Name : localhost.localdomain:9 (local to host localhost.localdomain)
UUID : 850aead9:49d1879a:551d055e:8e68d53c
Events : 31
Number Major Minor RaidDevice State
4 8 80 0 spare rebuilding /dev/sdf #这里就是备用盘/dev/sdf去替补损坏的sdb硬盘
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
5 8 64 3 active sync /dev/sde
0 8 16 - faulty /dev/sdb #可以看到sdb已经坏了
等待重建完毕
[root@localhost ~]# mdadm --detail /dev/md9
/dev/md9:
Version : 1.2
Creation Time : Wed Jul 28 10:37:19 2021
Raid Level : raid5
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Wed Jul 28 12:35:25 2021
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Name : localhost.localdomain:9 (local to host localhost.localdomain)
UUID : 850aead9:49d1879a:551d055e:8e68d53c
Events : 37
Number Major Minor RaidDevice State
4 8 80 0 active sync /dev/sdf
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
5 8 64 3 active sync /dev/sde
0 8 16 - faulty /dev/sdb
重建完成后,基本已经观察到RADI5的工作原理:
3块以上的存储盘(本实验用的4块,实际存储空间为3块的容量),再用一块作为备用磁盘,当存储盘一个出现问题后,备用盘会自动顶替坏磁盘.
下面,开始做RAID的扩容操作,把我们之间添加的/dev/sdg添加到RAID中,并使之容量能正常显示出来
把/dev/sdg添加到RAID md9中
[root@localhost ~]# mdadm --manage /dev/md9 --add /dev/sdg
mdadm: added /dev/sdg
可以看到sdg已经在"替补席"等待了
[root@localhost ~]# mdadm --detail /dev/md9
/dev/md9:
Version : 1.2
Creation Time : Wed Jul 28 10:37:19 2021
Raid Level : raid5
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 4
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Wed Jul 28 12:51:43 2021
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Name : localhost.localdomain:9 (local to host localhost.localdomain)
UUID : 850aead9:49d1879a:551d055e:8e68d53c
Events : 38
Number Major Minor RaidDevice State
4 8 80 0 active sync /dev/sdf
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
5 8 64 3 active sync /dev/sde
0 8 16 - faulty /dev/sdb
6 8 96 - spare /dev/sdg
将sdg的容量扩容到/dev/md9
mdadm --grow /dev/md9 -n5
[root@localhost ~]# mdadm --grow /dev/md9 -n5 #-n5对应之前创建的-4,意思是5个磁盘作为存储盘
mdadm: Need to backup 6144K of critical section..
[root@localhost ~]# mdadm --detail /dev/md9
/dev/md9:
Version : 1.2
Creation Time : Wed Jul 28 10:37:19 2021
Raid Level : raid5
Array Size : 3139584 (2.99 GiB 3.21 GB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 5
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Wed Jul 28 12:58:24 2021
State : clean, reshaping
Active Devices : 5
Working Devices : 5
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Reshape Status : 19% complete
Delta Devices : 1, (4->5)
Name : localhost.localdomain:9 (local to host localhost.localdomain)
UUID : 850aead9:49d1879a:551d055e:8e68d53c
Events : 62
Number Major Minor RaidDevice State
4 8 80 0 active sync /dev/sdf
1 8 32 1 active sync /dev/sdc
2 8 48 2 active sync /dev/sdd
5 8 64 3 active sync /dev/sde
6 8 96 4 active sync /dev/sdg
0 8 16 - faulty /dev/sdb
[root@localhost ~]# mdadm -Q /dev/md9
/dev/md9: 3.99GiB raid5 5 devices, 0 spares. Use mdadm --detail for more detail. 可以看到空间是4个磁盘的空间(RAID5特性就磁盘总数-1的容量),说明RAID容量已经扩展成功
现在需要把文件系统扩容
[root@localhost ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 48G 1.2G 47G 3% /
devtmpfs devtmpfs 476M 0 476M 0% /dev
tmpfs tmpfs 488M 0 488M 0% /dev/shm
tmpfs tmpfs 488M 7.7M 480M 2% /run
tmpfs tmpfs 488M 0 488M 0% /sys/fs/cgroup
/dev/sr0 iso9660 4.2G 4.2G 0 100% /mnt/cdrom
/dev/sda1 xfs 497M 123M 375M 25% /boot
tmpfs tmpfs 98M 0 98M 0% /run/user/0
/dev/md9 xfs 3.0G 33M 3.0G 2% /test
[root@localhost ~]# xfs_growfs /test
meta-data=/dev/md9 isize=512 agcount=8, agsize=98176 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=784896, imaxpct=25
= sunit=128 swidth=384 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 784896 to 1046528
[root@localhost ~]# xfs_growfs /test
meta-data=/dev/md9 isize=512 agcount=8, agsize=98176 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=784896, imaxpct=25
= sunit=128 swidth=384 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 784896 to 1046528
[root@localhost ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 48G 1.2G 47G 3% /
devtmpfs devtmpfs 476M 0 476M 0% /dev
tmpfs tmpfs 488M 0 488M 0% /dev/shm
tmpfs tmpfs 488M 7.7M 480M 2% /run
tmpfs tmpfs 488M 0 488M 0% /sys/fs/cgroup
/dev/sr0 iso9660 4.2G 4.2G 0 100% /mnt/cdrom
/dev/sda1 xfs 497M 123M 375M 25% /boot
tmpfs tmpfs 98M 0 98M 0% /run/user/0
/dev/md9 xfs 4.0G 33M 4.0G 1% /test
可以看到扩容已成功
一般会把坏的硬盘取消掉,换一个新硬盘当备用盘,这里把sdb从RAID中删除,然后又添加了一个sdb(主要是没硬盘了:P)
[root@localhost ~]# mdadm --manage /dev/md9 --remove /dev/sdb
mdadm: hot removed /dev/sdb from /dev/md9
[root@localhost ~]# mdadm --manage /dev/md9 --add /dev/sdb
mdadm: added /dev/sdb
[root@localhost ~]# mdadm --detail /dev/md9
/dev/md9:
Version : 1.2
Creation Time : Wed Jul 28 10:37:19 2021
Raid Level : raid5
Array Size : 4186112 (3.99 GiB 4.29 GB)
Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB)
Raid Devices : 5
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Wed Jul 28 13:10:11 2021
State : clean