iSCSI磁盘存储多路径问题解决

前提准备：已经添加好iSCSI磁盘，iSCSI为存储多路径

 

一、安装存储多路径软件

[root@desktop ~]# yum install device-mapper-multipath

 

二、开启mutipath相关模块

[root@desktop ~]# modprobe dm-multipath

[root@desktop ~]# modprobe dm-round-robin

 

三、设置开机自动启动

[root@desktop ~]# systemctl start multipathd.service

[root@desktop ~]# systemctl enable multipathd.service

查看启动效果

[root@desktop ~]# multipath -v2

Jan 02 18:16:45 | /etc/multipath.conf does not exist, blacklisting all devices.

Jan 02 18:16:45 | A default multipath.conf file is located at

Jan 02 18:16:45 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf

Jan 02 18:16:45 | You can run /sbin/mpathconf --enable to create

Jan 02 18:16:45 | /etc/multipath.conf. See man mpathconf(8) for more details

 

四、将multipath默认配置cp到/etc/目录下

这一步经过测试，放在/etc/mutipath 目录下配置不生效，放在/etc/目录下配置才生效

[root@desktop ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/

 

五、编辑配置文件，将要排除的硬盘的wwid号添加到multipath.conf配置文件中

关于wwid介绍参考http://blog.itpub.net/30126024/viewspace-2137508/

 

查看要排除的/dev/sda 的wwid

[root@desktop ~]# /lib/udev/scsi_id -gud /dev/sda

1ATA_VBOX_HARDDISK_VBb04c8d2c-3746644a

无论是实体硬盘还是虚拟磁盘都有wwid，wwid号不随硬盘是否格式化而改变，而uuid会随着硬盘格式化改变

 

将要排除的硬盘的wwid添加进multipath.conf

[root@desktop ~]# vi /etc/multipath.conf

blacklist {

        wwid 1ATA_VBOX_HARDDISK_VBb04c8d2c-3746644a

        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"

        devnode "^hd[a-z]"

}

 

六、添加完毕重启multipathd服务查看效果

[root@desktop ~]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 35.1G 0 disk

├─sda1 8:1 0 1G 0 part /boot

└─sda2 8:2 0 34.1G 0 part

├─centos-root 253:0 0 32.1G 0 lvm /

└─centos-swap 253:1 0 2G 0 lvm [SWAP]

sdb 8:16 0 8G 0 disk

└─mpatha 253:2 0 8G 0 mpath

sdc 8:32 0 8G 0 disk

└─mpatha 253:2 0 8G 0 mpath

sr0 11:0 1 1024M 0 rom

这里看到sdb 和sdc都已经出现了mpath

 

[root@desktop ~]# fdisk -l

Disk /dev/sda: 37.7 GB, 37722587136 bytes, 73676928 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x000b7dd8

 

Device Boot Start End Blocks Id System

/dev/sda1 * 2048 2099199 1048576 83 Linux

/dev/sda2 2099200 73676799 35788800 8e Linux LVM

 

Disk /dev/mapper/centos-root: 34.5 GB, 34493956096 bytes, 67371008 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

 

Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

 

Disk /dev/sdb: 8589 MB, 8589934592 bytes, 16777216 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 4194304 bytes

 

Disk /dev/sdc: 8589 MB, 8589934592 bytes, 16777216 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 4194304 bytes

 

Disk /dev/mapper/mpatha: 8589 MB, 8589934592 bytes, 16777216 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 4194304 bytes

 

七、格式化存储多路径盘/dev/mapper/mpatha

[root@desktop ~]# mkfs.xfs /dev/mapper/mpatha

meta-data=/dev/mapper/mpatha isize=512 agcount=4, agsize=524288 blks

= sectsz=512 attr=2, projid32bit=1

= crc=1 finobt=0, sparse=0

data = bsize=4096 blocks=2097152, imaxpct=25

= sunit=0 swidth=0 blks

naming =version 2 bsize=4096 ascii-ci=0 ftype=1

log =internal log bsize=4096 blocks=2560, version=2

= sectsz=512 sunit=0 blks, lazy-count=1

realtime =none extsz=4096 blocks=0, rtextents=0

 

八、挂载格式化后的存储多路径硬盘

[root@desktop ~]# mount /dev/mapper/mpatha /data

 

[root@desktop ~]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/centos-root 33G 5.0G 28G 16% /

devtmpfs 905M 0 905M 0% /dev

tmpfs 920M 0 920M 0% /dev/shm

tmpfs 920M 9.0M 911M 1% /run

tmpfs 920M 0 920M 0% /sys/fs/cgroup

/dev/sda1 1014M 178M 837M 18% /boot

tmpfs 184M 12K 184M 1% /run/user/42

tmpfs 184M 0 184M 0% /run/user/1000

/dev/mapper/mpatha 8.0G 33M 8.0G 1% /data

 

九、设置开机自动挂载存储多路径iSCSI硬盘

[root@desktop data]# vi /etc/fstab

#

# /etc/fstab

# Created by anaconda on Thu Apr 5 16:28:31 2018

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/centos-root / xfs defaults 0 0

UUID=be5846d5-6002-4088-8f2d-00bfb0a317b8 /boot xfs defaults 0 0

/dev/mapper/centos-swap swap swap defaults 0 0

/dev/mapper/mpatha     /data    xfs     defaults,_netdev     0 0

设置好重启测试

[root@desktop ~]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sda 8:0 0 35.1G 0 disk

├─sda1 8:1 0 1G 0 part /boot

└─sda2 8:2 0 34.1G 0 part

├─centos-root 253:0 0 32.1G 0 lvm /

└─centos-swap 253:1 0 2G 0 lvm [SWAP]

sdb 8:16 0 8G 0 disk

└─mpatha 253:2 0 8G 0 mpath /data

sdc 8:32 0 8G 0 disk

└─mpatha 253:2 0 8G 0 mpath /data

sr0 11:0 1 1024M 0 rom

 

十、测试存储多路径设置

创建一个测试文件

[root@desktop ~]# cd /data/

[root@desktop data]# ls

[root@desktop data]# touch test

 

在服务器端down一个网卡

[root@localhost ~]# ifdown enp0s8

Device 'enp0s8' successfully disconnected.

 

在客户端测试创建文件

[root@desktop data]# touch test1

[root@desktop data]# ls

test test1

总结1：down掉一个网卡还可以继续写入

 

服务器端再down第二个网卡

[root@localhost ~]# ifdown enp0s9

 

[root@desktop data]# touch test2

[root@desktop data]# ls

test test1 test2

注意：这里由于iscsi配置文件/etc/iscsi/iscsi.conf 配置文件中timeout 默认120秒延迟，所以看上去写入了实际没有写入，我们可以通过umount掉/data再mount测试

 

[root@desktop ~]# umount /data

[root@desktop ~]# mount /dev/mapper/mpatha /data

[root@desktop ~]# cd /data/

[root@desktop data]# ls

test test1

网卡全部断掉后，后面在操作没有被保存

 

十一、更改# vi /etc/iscsi/iscsid.conf配置文件的timeout参数，当一条链路故障后快速切换

 

# vi /etc/iscsi/iscsid.conf

node.session.timeo.replacement_timeout = 120

将默认的120秒，修改成5秒或者更少

node.session.timeo.replacement_timeout = 5

修改后重启iscsi 和iscsid 服务

[root@iscsi-client data]# systemctl restart iscsid

[root@iscsi-client data]# systemctl restart iscsi

 

参考文章 http://www.linuxboy.net/linuxjc/144375.html