1.当前系统信息
当前有2块硬盘挂载,都是10T,系统安装在/dev/sda,/u01,/u02,/u03都还没有分区,待raid做完后再分区,现在的主要目的是将系统盘做成raid1.
分区和挂载情况如下:
/dev/sda1 BIOS boot 此分区只有2M大小,不打算做raid1(由于硬盘大于2T,所以需要这个分区存放分区信息)
/dev/sda2 /boot
/dev/sda3 /
/dev/sda4 swap
[root@qht116 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@qht116 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 99G 3.9G 90G 5% /
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 63G 12M 63G 1% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sda2 190M 133M 44M 76% /boot
tmpfs 13G 48K 13G 1% /run/user/1000
tmpfs 13G 0 13G 0% /run/user/0
[root@qht116 ~]# swapon -s
Filename Type Size Used Priority
/dev/sda4 partition 68157436 0 -1
[root@qht116 ~]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sda: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 09C53634-3391-47BC-9974-11A055DDDBF2
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Microsoft basic
3 415744 210130943 100G Microsoft basic
4 210130944 346445823 65G Linux swap
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sdb: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 37D28917-48E6-488D-8BB5-CB768B2C0E61
2.复制分区信息到/dev/sdb
用sgdisk来复制GPT分区格式,这里千不能将sda和sdb写反了。因为和sfdisk的写法不同,很容易写反。
sgdisk的手册如下:
Man page of SGDISKhttp://www.rodsbooks.com/gdisk/sgdisk.html
[root@qht116 ~]# sgdisk --replicate=/dev/sdb /dev/sda
The operation has completed successfully.
[root@qht116 ~]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sda: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 09C53634-3391-47BC-9974-11A055DDDBF2
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Microsoft basic
3 415744 210130943 100G Microsoft basic
4 210130944 346445823 65G Linux swap
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sdb: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 09C53634-3391-47BC-9974-11A055DDDBF2
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Microsoft basic
3 415744 210130943 100G Microsoft basic
4 210130944 346445823 65G Linux swap
sgdisk将两块硬盘的信息完成同步了,/dev/sdb的UUID需要修改一下:
修改前UUID是一样的
[root@qht116 ~]# blkid /dev/sd*
/dev/sda: PTTYPE="gpt"
/dev/sda1: PARTUUID="17487062-0664-46d6-a2e8-d68c9ae39f7e"
/dev/sda2: UUID="a3fe54f8-b9af-4006-ba84-02fbd88c6cc6" TYPE="ext4" PARTUUID="8f3dc541-2e97-492f-b917-897f7fbaf9d5"
/dev/sda3: UUID="0bdc4335-efe8-4c9d-9858-7a1a1e8915f5" TYPE="ext4" PARTUUID="90703004-1a1c-4644-bda8-455914326130"
/dev/sda4: UUID="81a8e94a-1b58-4d57-9819-4592625c4a71" TYPE="swap" PARTUUID="69126115-c0c5-41a5-8f46-21c005bea37c"
/dev/sdb: PTTYPE="gpt"
/dev/sdb1: PARTUUID="17487062-0664-46d6-a2e8-d68c9ae39f7e"
/dev/sdb2: UUID="b09710d4-ce10-4cb2-a34c-bb1595b06bd6" TYPE="ext4" PARTUUID="8f3dc541-2e97-492f-b917-897f7fbaf9d5"
/dev/sdb3: PARTUUID="90703004-1a1c-4644-bda8-455914326130"
/dev/sdb4: PARTUUID="69126115-c0c5-41a5-8f46-21c005bea37c"
[root@qht116 ~]# sgdisk -G /dev/sdb
The operation has completed successfully.
修改后UUID变了
[root@qht116 ~]# blkid /dev/sd*
/dev/sda: PTTYPE="gpt"
/dev/sda1: PARTUUID="17487062-0664-46d6-a2e8-d68c9ae39f7e"
/dev/sda2: UUID="a3fe54f8-b9af-4006-ba84-02fbd88c6cc6" TYPE="ext4" PARTUUID="8f3dc541-2e97-492f-b917-897f7fbaf9d5"
/dev/sda3: UUID="0bdc4335-efe8-4c9d-9858-7a1a1e8915f5" TYPE="ext4" PARTUUID="90703004-1a1c-4644-bda8-455914326130"
/dev/sda4: UUID="81a8e94a-1b58-4d57-9819-4592625c4a71" TYPE="swap" PARTUUID="69126115-c0c5-41a5-8f46-21c005bea37c"
/dev/sdb: PTTYPE="gpt"
/dev/sdb1: PARTUUID="4c88134a-532b-4627-8b67-4a01dc10141e"
/dev/sdb2: UUID="b09710d4-ce10-4cb2-a34c-bb1595b06bd6" TYPE="ext4" PARTUUID="7401bee8-b0b7-4301-b94f-440938990ea5"
/dev/sdb3: PARTUUID="22aba114-23cf-4629-81dd-05ccfdfa45e0"
/dev/sdb4: PARTUUID="54024bb9-1dd4-42e9-a8e6-2043fa1f5aab"
3.通过mdadm建立raid
先修改/dev/sdb的分区type,由于是大硬盘,无法通过fdisk来修改,需要用parted修改分区type
[root@qht116 ~]# parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) set
Partition number? 2
Flag to Invert? raid
New state? [on]/off?
(parted) set
Partition number? 3
Flag to Invert? raid
New state? [on]/off?
(parted) set
Partition number? 4
Flag to Invert? raid
New state? [on]/off?
(parted) p
Model: TOSHIBA MG06SCA10TE (scsi)
Disk /dev/sdb: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot
Number Start End Size File system Name Flags
1 1049kB 3146kB 2097kB bios_grub
2 3146kB 213MB 210MB ext4 raid
3 213MB 108GB 107GB raid
4 108GB 177GB 69.8GB raid
[root@qht116 ~]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sda: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 09C53634-3391-47BC-9974-11A055DDDBF2
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Microsoft basic
3 415744 210130943 100G Microsoft basic
4 210130944 346445823 65G Linux swap
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sdb: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: AC29124E-B08E-4BD7-B241-A1E9E0ADFFC0
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Linux RAID
3 415744 210130943 100G Linux RAID
4 210130944 346445823 65G Linux RAID
建立md分区
最好指定homehost,否则默认为当前的Hostname,当hostname修改后mdadm信息不会跟着修改。
# mdadm --create /dev/md/foobar --homehost=barfoo
[root@qht116 ~]# mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb2
[root@qht116 ~]# mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb3
[root@qht116 ~]# mdadm --create /dev/md2 --level=1 --raid-disks=2 missing /dev/sdb4
格式化分区
[root@qht116 ~]# mkfs.ext4 /dev/md0
[root@qht116 ~]# mkfs.ext4 /dev/md1
[root@qht116 ~]# mkswap /dev/md2
建立/etc/mdadm.conf
[root@qht116 ~]# mdadm --detail --scan > /etc/mdadm.conf
[root@qht116 ~]# cat /etc/mdadm.conf
ARRAY /dev/md0 metadata=1.2 name=qht116:0 UUID=da370539:60a5f3ae:ccc2028a:7807d731
ARRAY /dev/md1 metadata=1.2 name=qht116:1 UUID=f331e21f:94200535:0c24db2b:e1a40274
ARRAY /dev/md2 metadata=1.2 name=qht116:2 UUID=52bbfa94:0751ffca:ae8070cb:221b5abe
4.传输数据到/dev/sdb
先挂载分区
[root@qht116 /]# mount /dev/md1 /mnt/
[root@qht116 /]# mkdir /mnt/boot
[root@qht116 /]# mount /dev/md0 /mnt/boot/
开始拷贝
[root@qht116 ~]# rsync -auxHAXSv --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/mnt/* /* /mnt
5. grub2 and initramfs
先挂载系统分区到/mnt,以便chroot操作/dev/sdb
[root@qht116 ~]# mount --bind /proc /mnt/proc
[root@qht116 ~]# mount --bind /dev /mnt/dev
[root@qht116 ~]# mount --bind /sys /mnt/sys
[root@qht116 ~]# mount --bind /run /mnt/run
更改当前的操作路径到/mnt,这样就不影响/dev/sda的数据了
chroot到/mnt目录
[root@qht26 ~]# chroot /mnt/
修改/etc/fstab,将三个分区的UUID改成md所以应的值
[root@qht116 /]# blkid /dev/md*
/dev/md0: UUID="758630e5-351b-4e35-9605-712763c0c490" TYPE="ext4"
/dev/md1: UUID="f8c12bef-2801-4ff2-8f4f-2e44390831e2" TYPE="ext4"
/dev/md2: UUID="7fe97dd7-938f-4af3-ad3d-c5ebef7a10cd" TYPE="swap"
[root@qht116 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Feb 25 00:55:09 2022
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
#UUID=0bdc4335-efe8-4c9d-9858-7a1a1e8915f5 / ext4 defaults 1 1
#UUID=a3fe54f8-b9af-4006-ba84-02fbd88c6cc6 /boot ext4 defaults 1 2
#UUID=81a8e94a-1b58-4d57-9819-4592625c4a71 swap swap defaults 0 0
UUID=f8c12bef-2801-4ff2-8f4f-2e44390831e2 / ext4 defaults 1 1
UUID=758630e5-351b-4e35-9605-712763c0c490 /boot ext4 defaults 1 2
UUID=7fe97dd7-938f-4af3-ad3d-c5ebef7a10cd swap swap defaults 0 0
重新生成initramfs
[root@qht116 /]# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bck
[root@qht116 /]# dracut --mdadmconf --fstab --add="mdraid" --filesystems "xfs ext4 ext3" --add-drivers="raid1" --force /boot/initramfs-$(uname -r).img $(uname -r) -M
bash
nss-softokn
i18n
network
ifcfg
drm
plymouth
kernel-modules
mdraid
rootfs-block
terminfo
udev-rules
biosdevname
systemd
usrmount
base
fs-lib
shutdown
[root@qht116 /]#
修改grub 参数,重新生成启动引导文件
增加这两行:
GRUB_CMDLINE_LINUX="rd.auto rd.auto=1 rhgb quiet"
GRUB_PRELOAD_MODULES="mdraid1x"
[root@qht116 /]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.auto rd.auto=1 rhgb quiet"
GRUB_PRELOAD_MODULES="mdraid1x"
GRUB_DISABLE_RECOVERY="true"
生成新的grub引导文件
[root@qht116 /]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-3.10.0-862.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-862.el7.x86_64.img
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Found linux image: /boot/vmlinuz-0-rescue-29b8c8410e55464392ac50c7f519a065
Found initrd image: /boot/initramfs-0-rescue-29b8c8410e55464392ac50c7f519a065.img
/usr/sbin/grub2-probe: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
done
安装grub到/dev/sdb(此步骤千万不能在外面执行,否则安装的还是/dev/sda的grub启动信息)
[root@qht116 /]# grub2-install /dev/sdb
Installing for i386-pc platform.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
好了,现在可以用/dev/sdb来启动系统了。
将/dev/sda拔掉,或者bios里面设置用/dev/sdb的硬盘启动。
我是sas硬盘,支持热插拔,所以直接把/dev/sda拔了。
6.如果一切顺利重启后raid分区都应该是挂载了的
[root@qht116 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 99G 3.9G 90G 5% /
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 4.0K 63G 1% /dev/shm
tmpfs 63G 11M 63G 1% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/md0 189M 162M 14M 93% /boot
tmpfs 13G 64K 13G 1% /run/user/0
[root@qht116 ~]# swapon -s
Filename Type Size Used Priority
/dev/md2 partition 68090876 0 -1
7.将之前的/sda添加到raid
发现重启后sda和sdb的顺序换了,现在md下面挂载了/sda的分区,现在需要将之前的sda(即现在的sdb)添加到md。
[root@qht116 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda3[1]
104791040 blocks super 1.2 [2/1] [_U]
md2 : active raid1 sda4[1]
68090880 blocks super 1.2 [2/1] [_U]
md0 : active raid1 sda2[1]
203776 blocks super 1.2 [2/1] [_U]
unused devices: <none>
[root@qht116 ~]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sda: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: AC29124E-B08E-4BD7-B241-A1E9E0ADFFC0
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Linux RAID
3 415744 210130943 100G Linux RAID
4 210130944 346445823 65G Linux RAID
Disk /dev/md0: 208 MB, 208666624 bytes, 407552 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/md2: 69.7 GB, 69725061120 bytes, 136181760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/md1: 107.3 GB, 107306024960 bytes, 209582080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/sdb: 10000.8 GB, 10000831348736 bytes, 19532873728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt
Disk identifier: 09C53634-3391-47BC-9974-11A055DDDBF2
# Start End Size Type Name
1 2048 6143 2M BIOS boot
2 6144 415743 200M Microsoft basic
3 415744 210130943 100G Microsoft basic
4 210130944 346445823 65G Linux swap
修改/dev/sdb的分区type,方法和上面的一样
[root@qht116 ~]# parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: TOSHIBA MG06SCA10TE (scsi)
Disk /dev/sdb: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot
Number Start End Size File system Name Flags
1 1049kB 3146kB 2097kB bios_grub
2 3146kB 213MB 210MB ext4
3 213MB 108GB 107GB ext4
4 108GB 177GB 69.8GB linux-swap(v1)
(parted) set
Partition number? 2
Flag to Invert? raid
New state? [on]/off?
(parted) set
Partition number? 3
Flag to Invert? raid
New state? [on]/off?
(parted) set
Partition number? 4
Flag to Invert? raid
New state? [on]/off?
(parted) p
Model: TOSHIBA MG06SCA10TE (scsi)
Disk /dev/sdb: 10.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: pmbr_boot
Number Start End Size File system Name Flags
1 1049kB 3146kB 2097kB bios_grub
2 3146kB 213MB 210MB ext4 raid
3 213MB 108GB 107GB ext4 raid
4 108GB 177GB 69.8GB linux-swap(v1) raid
添加分区到md
[root@qht116 ~]# mdadm --manage /dev/md0 --add /dev/sdb2
mdadm: added /dev/sdb2
[root@qht116 ~]# mdadm --manage /dev/md1 --add /dev/sdb3
mdadm: added /dev/sdb3
[root@qht116 ~]# mdadm --manage /dev/md2 --add /dev/sdb4
mdadm: added /dev/sdb4
[root@qht116 ~]# watch -n1 "cat /proc/mdstat"
Personalities : [raid1]
md1 : active raid1 sdb3[2] sda3[1]
104791040 blocks super 1.2 [2/1] [_U]
[>....................] recovery = 3.8% (4086528/104791040) finish=8.2min speed=204326K/sec
md2 : active raid1 sdb4[2] sda4[1]
68090880 blocks super 1.2 [2/1] [_U]
resync=DELAYED
md0 : active raid1 sdb2[2] sda2[1]
203776 blocks super 1.2 [2/2] [UU]
unused devices: <none>
8.待同步完成安装grub到这块需要同步的硬盘
[root@qht116 ~]# grub2-install /dev/sdb
Installing for i386-pc platform.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
参考:
Convert to Raid1 Setup on Existing Centos7 InstallConvert to raid 1 CentOS 7 - Mhc wiki - gnu linux and rfid fun