CentOS 7 iSCSI 多路径（Multipath）配置
1 设置精简预配置LVM逻辑卷
1.1 使用pvcreate命令创建物理卷
假设检测到新磁盘为/dev/sdb
pvcreate /dev/sdb
1.2 使用vgcreate命令创建卷组
vgcreate volgrp /dev/sdb
1.3 从卷组创建精简池
精简池就像使用lvcreate命令创建的逻辑卷
句法 ：
＃lvcreate –L <池的大小> -T <卷组> / <池名称>
其中–L用于指定池的大小，–T用于指定精简池
lvcreate -L 100G -T volgrp/lvpool
验证精简池大小:
[root@target ~]# lvs /dev/volgrp/lvpool
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lvpool volgrp twi-aotz-- 100.00g 0.01 0.43
[root@target ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root centos -wi-ao---- 150.00g
swap centos -wi-ao---- <3.88g
lvpool volgrp twi-aotz-- 100.00g 0.01 0.43
node1 volgrp Vwi-aotz-- 4.00g lvpool 0.26
1.4 从精简池创建逻辑卷。
让我们创建两个逻辑卷，每个大小为4 GB。
句法 ：
＃lvcreate -V <逻辑卷大小> -T <卷组名> / <名称thinpool> -n <逻辑卷名>
lvcreate -V 4G -T volgrp/lvpool -n node1
1.5 格式化精简配置逻辑卷并挂载到/mnt，写入文件
mkfs.xfs /dev/volgrp/node1
mount /dev/volgrp/node1 /mnt
touch /mnt/wangjinxiong.txt
备注：lvm查看命令如下：
[root@target ~]# vgdisplay
--- Volume group ---
VG Name centos
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size <153.88 GiB
PE Size 4.00 MiB
Total PE 39393
Alloc PE / Size 39392 / <153.88 GiB
Free PE / Size 1 / 4.00 MiB
VG UUID 90zR5I-cTJK-6NsO-MgUr-o9Pe-4W4b-e50bCh
--- Volume group ---
VG Name volgrp
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 6
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size <500.00 GiB
PE Size 4.00 MiB
Total PE 127999
Alloc PE / Size 25650 / <100.20 GiB
Free PE / Size 102349 / 399.80 GiB
VG UUID MTrVFc-Ax3a-EXym-vViS-mqa5-RPfU-LzcHYf
[root@target ~]# lvdisplay
--- Logical volume ---
LV Path /dev/centos/root
LV Name root
VG Name centos
LV UUID wdZfj5-UKzO-uiq7-pjsr-OmT8-1hkL-Nb1WHl
LV Write Access read/write
LV Creation host, time localhost, 2020-01-31 18:29:18 +0800
LV Status available
# open 1
LV Size 150.00 GiB
Current LE 38400
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:0
--- Logical volume ---
LV Path /dev/centos/swap
LV Name swap
VG Name centos
LV UUID 7swxQq-7Dsk-jWzI-uEcR-sI2x-hucx-OHe73v
LV Write Access read/write
LV Creation host, time localhost, 2020-01-31 18:29:19 +0800
LV Status available
# open 2
LV Size <3.88 GiB
Current LE 992
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:1
--- Logical volume ---
LV Name lvpool
VG Name volgrp
LV UUID VyTn0Q-SFMY-TExw-zjD9-aDZ9-x9Cg-HQyVhP
LV Write Access read/write
LV Creation host, time target, 2020-09-11 09:50:17 +0800
LV Pool metadata lvpool_tmeta
LV Pool data lvpool_tdata
LV Status available
# open 2
LV Size 100.00 GiB
Allocated pool data 0.01%
Allocated metadata 0.43%
Current LE 25600
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:4
--- Logical volume ---
LV Path /dev/volgrp/node1
LV Name node1
VG Name volgrp
LV UUID Yf9tsK-urFw-kbrP-kA0y-dWCA-qZ4e-fRMkhU
LV Write Access read/write
LV Creation host, time target, 2020-09-11 09:57:47 +0800
LV Pool name lvpool
LV Status available
# open 1
LV Size 4.00 GiB
Mapped size 0.26%
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 8192
Block device 253:6
参考文档：https://blog.csdn.net/allway2/article/details/103113057
2 target服务配置
2.1 安装前准备
关闭selinux
setenforce 0
sed -i '/^SELINUX=.*/ s//SELINUX=disabled/' /etc/selinux/config
关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
安装epel扩展包
yum install -y epel-release
2.2 target设置
安装 scsi-target-utils
yum --enablerepo=epel -y install iscsi-initiator-utils.x86_64
配置
vi /etc/tgt/targets.conf
<target iqn.1994-05.com.redhat:c1ec1f96a0b7>
# 把/dev/sdb作为块设备共享出去
backing-store /dev/volgrp/node1
# 可选，iSCSI Initiator限制
initiator-address 10.5.29.55
# 可选，认证配置，username和password配置成你自己的信息
# incominguser username password
</target>
iqn.1994-05.com.redhat:c1ec1f96a0b7为客户端的iqn号，initiator-address为客户端地址，以下是客户端查看iqn号：
# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:c1ec1f96a0b7
# ip add | grep ens
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
inet 10.5.29.55/28 brd 10.5.29.63 scope global noprefixroute ens3
开启服务
systemctl enable tgtd.service
systemctl start tgtd.service
查看服务
tgtadm --mode target --op show
Target 1: iqn.1994-05.com.redhat:c1ec1f96a0b7
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 1
Initiator: iqn.1994-05.com.redhat:c1ec1f96a0b7 alias: docker1
Connection: 0
IP Address: 10.5.29.55
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB, Block size: 1
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: null
Backing store path: None
Backing store flags:
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 4295 MB, Block size: 512
Online: Yes
Removable media: No
Prevent removal: No
Readonly: No
SWP: No
Thin-provisioning: No
Backing store type: rdwr
Backing store path: /dev/volgrp/node1
Backing store flags:
Account information:
ACL information:
10.5.29.55
2.3 Initiator配置（服务器及客户端都安装）
安装
yum -y install iscsi-initiator-utils
查看客户端iqn号，iqn号可以自定义，也可以采用默认。
cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:c1ec1f96a0b7
如果有账号及密码，需要添加以下：（非必须）
# vi /etc/iSCSI/iSCSId.conf
# 以下配置，如果你target没有配置ACL限制，可以直接跳过，使用默认配置即可
# 57行:取消注释
node.session.auth.authmethod = CHAP
# 61,62行:取消注释，用户名和密码设置成前面在target里面设置好的
node.session.auth.username = username
node.session.auth.password = password
扫描设备并展示设备
iscsiadm -m discovery -t sendtargets -p 10.5.30.2
10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7
iscsiadm -m node -o show
# BEGIN RECORD 6.2.0.874-17
node.name = iqn.1994-05.com.redhat:c1ec1f96a0b7
node.tpgt = 1
node.startup = automatic
node.leading_login = No
iface.hwaddress = <empty>
iface.ipaddress = <empty>
iface.iscsi_ifacename = default
iface.net_ifacename = <empty>
iface.gateway = <empty>
iface.subnet_mask = <empty>
iface.transport_name = tcp
iface.initiatorname = <empty>
iface.state = <empty>
iface.vlan_id = 0
iface.vlan_priority = 0
iface.vlan_state = <empty>
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.bootproto = <empty>
iface.dhcp_alt_client_id_state = <empty>
iface.dhcp_alt_client_id = <empty>
iface.dhcp_dns = <empty>
iface.dhcp_learn_iqn = <empty>
iface.dhcp_req_vendor_id_state = <empty>
iface.dhcp_vendor_id_state = <empty>
iface.dhcp_vendor_id = <empty>
iface.dhcp_slp_da = <empty>
iface.fragmentation = <empty>
iface.gratuitous_arp = <empty>
iface.incoming_forwarding = <empty>
iface.tos_state = <empty>
iface.tos = 0
iface.ttl = 0
iface.delayed_ack = <empty>
iface.tcp_nagle = <empty>
iface.tcp_wsf_state = <empty>
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.tcp_timestamp = <empty>
iface.redirect = <empty>
iface.def_task_mgmt_timeout = 0
iface.header_digest = <empty>
iface.data_digest = <empty>
iface.immediate_data = <empty>
iface.initial_r2t = <empty>
iface.data_seq_inorder = <empty>
iface.data_pdu_inorder = <empty>
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
iface.chap_auth = <empty>
iface.bidi_chap = <empty>
iface.strict_login_compliance = <empty>
iface.discovery_auth = <empty>
iface.discovery_logout = <empty>
node.discovery_address = 10.5.30.2
node.discovery_port = 3260
node.discovery_type = send_targets
node.session.initial_cmdsn = 0
node.session.initial_login_retry_max = 8
node.session.xmit_thread_priority = -20
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.nr_sessions = 1
node.session.auth.authmethod = None
node.session.auth.username = <empty>
node.session.auth.password = <empty>
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.timeo.replacement_timeout = 120
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.err_timeo.host_reset_timeout = 60
node.session.iscsi.FastAbort = Yes
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.session.iscsi.DefaultTime2Retain = 0
node.session.iscsi.DefaultTime2Wait = 2
node.session.iscsi.MaxConnections = 1
node.session.iscsi.MaxOutstandingR2T = 1
node.session.iscsi.ERL = 0
node.session.scan = auto
node.conn[0].address = 10.5.30.2
node.conn[0].port = 3260
node.conn[0].startup = manual
node.conn[0].tcp.window_size = 524288
node.conn[0].tcp.type_of_service = 0
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.auth_timeout = 45
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.HeaderDigest = None
node.conn[0].iscsi.IFMarker = No
node.conn[0].iscsi.OFMarker = No
# END RECORD
登陆
iscsiadm -m node --login
确认信息：
iscsiadm -m session -o show
tcp: [1] 10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7 (non-flash)
确认分区信息:
cat /proc/partitions
major minor #blocks name
8 0 209715200 sda
8 1 1048576 sda1
8 2 207494144 sda2
253 0 199229440 dm-0
253 1 8257536 dm-1
8 16 4194304 sdb <--------挂载的iscsi盘
可以看到，sdd和sde就是通过iSCSI挂载得到的磁盘，此时通过fdisk -l命令，你可以发现这两块磁盘和node01的/dev/sdb完全一样，为了正常使用这块磁盘，我们还需要配置磁盘多路径。
参考文档：https://blog.csdn.net/weixin_34082854/article/details/89369861
3 配置多路径软件(Multipath)
什么是多路径？
普通的电脑主机都是一个硬盘挂接到一个总线上，这里是一对一的关系。而到了有光纤组成的SAN环境，或者由iSCSI组成的IPSAN环境，由于主机和存储通过了光纤交换机或者多块网卡及IP来连接，这样的话，就构成了多对多的关系。也就是说，主机到存储可以有多条路径可以选择。主机到存储之间的IO由多条路径可以选择。每个主机到所对应的存储可以经过几条不同的路径，如果是同时使用的话，I/O流量如何分配？其中一条路径坏掉了，如何处理？还有在操作系统的角度来看，每条路径，操作系统会认为是一个实际存在的物理盘，但实际上只是通向同一个物理盘的不同路径而已，这样是在使用的时候，就给用户带来了困惑。多路径软件就是为了解决上面的问题应运而生的。
多路径的主要功能就是和存储设备一起配合实现如下功能：
- 故障的切换和恢复
- IO流量的负载均衡
- 磁盘的虚拟化
为了主机可以使用 iSCSI 多路径访问存储设备，我们需要在主机上安装多路径设备映射器 (DM-Multipath)。多路径设备映射器可以实现主机节点和后端存储之间的多个 I/O 路径配置为一个单一逻辑设备，从而可以提供链路冗余和提高性能。主机通过访问包含多个 I/O 路径的逻辑设备，就可以有效提高后端存储系统的可靠性。
3.1 Multipath安装
服务器端配置双地址，其他在客户端配置，包含mutipath服务。
# ip add
.......
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:95:34:bd brd ff:ff:ff:ff:ff:ff
inet 10.5.30.2/28 brd 10.5.30.15 scope global noprefixroute ens3
valid_lft forever preferred_lft forever
inet 10.5.30.3/28 brd 10.5.30.15 scope global secondary noprefixroute ens3
valid_lft forever preferred_lft forever
inet6 fe80::4dad:d1ff:bac4:fc10/64 scope link noprefixroute
valid_lft forever preferred_lft forever
以下在客户端配置
yum install device-mapper-multipath -y
设置开机启动
systemctl enable multipathd.service
添加配置文件
需要multipath正常工作只需要如下配置即可，如果想要了解详细的配置，请参考 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-single/dm_multipath/index
vi /etc/multipath.conf
blacklist {
devnode "^sda"
}
defaults {
user_friendly_names yes
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
启动服务
systemctl start multipathd.service
3.2 Multipath挂载
卸载之前挂载的lun及iscsi链接
umount /mnt
iscsiadm -m node -T iqn.1994-05.com.redhat:c1ec1f96a0b7 -p 10.5.30.2:3260 --logout
Logging out of session [sid: 1, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260]
Logout of [sid: 1, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260] successful.
查看服务
multipath -ll
mpatha (360000000000000000e00000000010001) dm-2 IET ,VIRTUAL-DISK
size=4.0G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 4:0:0:1 sdb 8:16 active ready running
`- 5:0:0:1 sdc 8:32 active ready running
重新建立2个iscsi链接
[root@docker1 /]# iscsiadm -m discovery -t st -p 10.5.30.2
10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7
[root@docker1 /]# iscsiadm -m discovery -t st -p 10.5.30.3
10.5.30.3:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7
[root@docker1 /]# iscsiadm -m node --login
Logging in to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260] (multiple)
Logging in to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.3,3260] (multiple)
Login to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.2,3260] successful.
Login to [iface: default, target: iqn.1994-05.com.redhat:c1ec1f96a0b7, portal: 10.5.30.3,3260] successful.
查看iscsi连接
iscsiadm -m session
tcp: [2] 10.5.30.3:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7 (non-flash)
tcp: [3] 10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7 (non-flash)
查看硬盘情况
[root@docker1 /]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 197.9G 0 part
├─centos-root 253:0 0 190G 0 lvm /
└─centos-swap 253:1 0 7.9G 0 lvm [SWAP]
sdb 8:16 0 4G 0 disk
└─mpatha 253:2 0 4G 0 mpath
sdc 8:32 0 4G 0 disk
└─mpatha 253:2 0 4G 0 mpath
挂载硬盘
mount /dev/mapper/mpatha /mnt
备注命令：
发现target,询问target名称bai
[root@RedHat5 ~]# iscsiadm -m discovery -t sendtargets -p targetIP地址：端口号du
该命令将报告出target名称zhi
建立连接
得到了target名称后，dao使用下面命令建立连接
[root@RedHat5 ~]# iscsiadm -m node -T target名称 -p targetIP地址：端口号 --login
这时应该看到映射到的逻辑磁盘名号，使用fdisk -l 可以看到有关逻辑磁盘的详细内容，可以象使用本地硬盘一样对其进行分区、创建文件系统、挂载、卸载等操作。
断开连接
如果逻辑磁盘已经挂载，断开连接前应先卸载，然后使用下面命令
[root@RedHat5 ~]# iscsiadm -m node -T target名称 -p targetIP地址：端口号 --logout
关机之前应该正确的卸载和断开逻辑磁盘，否则会在target服务器上留下僵死的进程，造成服务器资源的浪费。
3.3 Multipath盘开机自动挂载
修改/etc/fstab，加入以下内容:
/dev/mapper/mpatha /mnt xfs defaults,_netdev 0 0
修改/etc/rc.d/rc.local，加入以下内容：
rm -rf /var/lib/iscsi/nodes/*
iscsiadm -m discovery -t st -p 10.5.30.2
iscsiadm -m discovery -t st -p 10.5.30.3
iscsiadm -m node --login
修改rc.local权限
chmod +x /etc/rc.d/rc.local
3.4 iscsi建立故障
重新启动服务器，发生以下故障：
iscsiadm -m discovery -t st -p 10.5.30.2
iscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.5.30.2,3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7]
10.5.30.2:3260,1 iqn.1994-05.com.redhat:c1ec1f96a0b7
将/var/lib/iscsi/nodes/所有文件删除
rm -rf /var/lib/iscsi/nodes/*
备注：查看已经建立iscsi
iscsiadm -m node
卸载iscsi盘
iscsiadm -m node --logoutall=all
iscsiadm常用命令
增加iscsi存储
(1)发现iscsi存储:iscsiadm
-m discovery -t st -p ISCSI_IP
(2)查看iscsi发现记录:iscsiadm
-m node
(3)登录iscsi存储:iscsiadm
-m node -T LUN_NAME -p ISCSI_IP -l
(4)开机自动:
iscsiadm -m node –T LUN_NAME -p ISCSI_IP -- op
update -n node.startup - v
automatic
删除iscsi存储
(1)登出iscsi存储
iscsiadm -m node -T LUN_NAME -p ISCSI_IP -u
(2)对出iscsi所有登录
iscsiadm -m node --logoutall=all
(3)删除iscsi发现记录:iscsiadm
-m node -o delete -T LUN_NAME -p ISCSI_IP
登入需验证码的节点
(1)开启认证
iscsiadm
-m node -T LUN_NAME -o update --name node.session.auth.authmethod --value=CHAP
*.使用-o同-- op
(2)添加用户
iscsiadm
-m node -T LUN_NAME -- op
update --name node.session.auth.username --value=[用户名]
(3)添加密码
iscsiadm
–m node –T LUN_NAME – op
update –name node.session.auth.password –value=[密码]
建立链接例子1：
iscsiadm -m discovery -t st -p 10.186.9.233
iscsiadm -m discovery -t st -p 10.186.9.236
iscsiadm -m node --login
建立链接例子2：
iscsiadm -m discovery -t st -p 192.168.96.104
iscsiadm -m node -T iqn.2016-02.com.smartx:system:iscsi1 -p 192.168.96.104 -l
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
