一、ISCSI服务简介
当我们的系统需要大量的磁盘容量,但是身边却没有足够的存储设备,此时,我们可以使用通过网络的scsi磁盘,即Internet scsi(iscsi)。iscsi主要是通过TCP/IP的技术,将存储设备端通过iscsi target功能,做成可以提供磁盘的服务器端,再通过iscsi initiator(iscsi初始化用户)功能,做成能够挂载使用iscsi target的客户端,这样就能够通过iscsi协议来进行磁盘的应用了。
iscsi服务端需要使用的软件为targetcli
iscsi客户端需要的软件为iscsi-initiator-utils
二、实验
(1)准备要共享的设备或文件
iSCSI常用的共享有块设备(磁盘,分区,逻辑卷等)和大文件
这里演示常用的四种分区
,磁盘
,逻辑卷
,文件
。
分区
使用sdb的sdb1
[root@server /]# lsblk | grep sdb1
├─sdb1 8:17 0 1G 0 part
磁盘
使用sdc
[root@server ~]# lsblk | grep sdc
sdc 8:32 0 5G 0 disk
逻辑卷
使用sdb2作为pv,创建lv1
[root@server /]# pvcreate /dev/sdb2
Physical volume "/dev/sdb2" successfully created
[root@server /]# vgcreate vg0 /dev/sdb2
Volume group "vg0" successfully created
[root@server /]# lvcreate -n lv1 -L 700M vg0
Logical volume "lv1" created.
文件
使用dd命令创建一个500M大小的文件iscsi
[root@server /]# dd if=/dev/zero of=/iscsi bs=500M count=1
1+0 records in
1+0 records out
524288000 bytes (524 MB) copied, 7.97686 s, 65.7 MB/s
[root@server /]# ll -h /| grep iscsi
-rw-r--r--. 1 root root 500M Dec 6 16:54 iscsi
(2)服务端配置
1、装包
[root@server /]# yum install targetcli -y
targetcli是用于管理iscsi服务端存储资源的专用配置命令,它能够提供类似于fdisk命令的交互式配置功能,将iscsi共享资源的配置内容抽象成“目录”的形式,我们只需要将各类配置信息填入到相应的“目录”中即可。
2、进入交互式界面
[root@server /]# targetcli
[root@server /]# targetcli
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
/>
名词 | 解释 |
---|---|
backstores | 后备存储 |
block | 块设备磁盘驱动器,磁盘分区,逻辑卷,以及服务器上定义的任何b类型的设备文件 |
fileio | 大文件 |
pscsi | 物理scsi,通常不用 |
ramdisk | 内存盘,其中存储的数据在服务器重启后将全部丢失 |
3、创建共享的设备或文件
分区,磁盘,逻辑卷都属于块设备,进入block底下创建
/> cd /backstores/block
/backstores/block> create dev=/dev/sdb1 name=lun0
/backstores/block> create dev=/dev/sdc name=lun1
/backstores/block> create dev=/dev/vg0/lv1 name=lun2
文件不属于块设备,进入fileio底下创建
/backstores/block> cd /backstores/fileio
/backstores/fileio> create file_or_dev=/iscsi name=lun3
查看一下,结构非常清晰
/backstores/fileio> cd /
/> ls
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 3]
| | o- lun0 ...................... [/dev/sdb1 (1.0GiB) write-thru deactivated]
| | o- lun1 ....................... [/dev/sdc (5.0GiB) write-thru deactivated]
| | o- lun2 ................. [/dev/vg0/lv1 (700.0MiB) write-thru deactivated]
| o- fileio ............................................. [Storage Objects: 1]
| | o- lun3 ....................... [/iscsi (500.0MiB) write-back deactivated]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
4、设置共享的iscsi target名称及配置共享资源
说明:
1.使用lun0
和lun1
作为iqn.2019-12.com.redhat.linux.1:server
的共享,并且设置自动生成acl节点,即客户端不需要设置iscsi客户端名称
2.使用lun2
和lun3
作为iqn.2019-12.com.redhat.linux.2:server
的共享,并且手动设置acl节点,即客户端的iscsi客户端名称需要和服务端设置的acl一致
3.位于生产环境中的服务器可能有多块网卡,那么就可以在这儿指定由哪个网卡或IP地址对外提供共享存储资源,关闭自动创建默认portal
/> set group=global auto_add_default_portal=false
注:若未关闭则更改IP地址时需要把所有target名称底下的0.0.0.0:3260删除才能更改
设置iqn.2019-12.com.redhat.linux.1:server
1.进入iscsi目录创建target名称
/> cd /iscsi
/iscsi> create iqn.2019-12.com.redhat.linux.1:server
2.此时的目录结构
/iscsi> ls
o- iscsi .......................................................... [Targets: 1]
o- iqn.2019-12.com.redhat.linux.1:server ........................... [TPGs: 1]
o- tpg1 ................................................ [gen-acls, no-auth]
o- acls ........................................................ [ACLs: 0]
o- luns ........................................................ [LUNs: 2]
o- portals .................................................. [Portals: 0]
3.设置自动生成acl节点
/iscsi> cd iqn.2019-12.com.redhat.linux.1:server/tpg1///进入tpg1目录下
/iscsi/iqn.20...1:server/tpg1> set attribute authentication=0 //关闭账号密码访问,默认关闭
/iscsi/iqn.20...1:server/tpg1> set attribute generate_node_acls=1//自动生成acl节点
4.设置共享的lun0和lun1
/iscsi/iqn.20...1:server/tpg1> cd luns
/iscsi/iqn.20...ver/tpg1/luns> create /backstores/block/lun0
/iscsi/iqn.20...ver/tpg1/luns> create /backstores/block/lun1
5.设置监听的网卡IP地址和端口号
/iscsi/iqn.20...ver/tpg1/luns> cd ..
/iscsi/iqn.20...1.server/tpg1> cd portals
/iscsi/iqn.20.../tpg1/portals> create 192.168.19.101 3260
6.此时的目录结构
/iscsi/iqn.20.../tpg1/portals> ls /
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 3]
| | o- lun0 ........................ [/dev/sdb1 (1.0GiB) write-thru activated]
| | o- lun1 ......................... [/dev/sdc (5.0GiB) write-thru activated]
| | o- lun2 ................. [/dev/vg0/lv1 (700.0MiB) write-thru deactivated]
| o- fileio ............................................. [Storage Objects: 1]
| | o- lun3 ....................... [/iscsi (500.0MiB) write-back deactivated]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 1]
| o- iqn.2019-12.com.redhat.linux.1:server ......................... [TPGs: 1]
| o- tpg1 .............................................. [gen-acls, no-auth]
| o- acls ...................................................... [ACLs: 0]
| o- luns ...................................................... [LUNs: 2]
| | o- lun0 ..................................... [block/lun0 (/dev/sdb1)]
| | o- lun1 ...................................... [block/lun1 (/dev/sdc)]
| o- portals ................................................ [Portals: 1]
| o- 192.168.19.101:3260 .......................................... [OK]
o- loopback ..................................................... [Targets: 0]
设置iqn.2019-12.com.redhat.linux.2:server
1.进入iscsi目录创建target名称
/> cd /iscsi
/iscsi> create iqn.2019-12.com.redhat.linux.2:server
2.此时的目录结构
/iscsi> ls
o- iscsi .......................................................... [Targets: 2]
o- iqn.2019-12.com.redhat.linux.1:server ........................... [TPGs: 1]
| o- tpg1 ................................................ [gen-acls, no-auth]
| o- acls ........................................................ [ACLs: 0]
| o- luns ........................................................ [LUNs: 2]
| | o- lun0 ....................................... [block/lun0 (/dev/sdb1)]
| | o- lun1 ........................................ [block/lun1 (/dev/sdc)]
| o- portals .................................................. [Portals: 1]
| o- 192.168.19.201:3260 ............................................ [OK]
o- iqn.2019-12.com.redhat.linux.2:server ........................... [TPGs: 1]
o- tpg1 ............................................. [no-gen-acls, no-auth]
o- acls ........................................................ [ACLs: 0]
o- luns ........................................................ [LUNs: 0]
o- portals .................................................. [Portals: 0]
3.手动设置acl节点
/iscsi> cd iqn.2019-12.com.redhat.linux.2:server/tpg1///进入tpg1目录下
/iscsi/iqn.20...2:server/tpg1> get attribute authentication//查看是否关闭账号密码,默认关闭
authentication=0
/iscsi/iqn.20...2:server/tpg1> get attribute authentication//默认需要手动设置acl节点
authentication=0
/iscsi/iqn.20...2:server/tpg1> cd acls//进入此目录设置acl节点
/iscsi/iqn.20...ver/tpg1/acls> create iqn.2019-12.com.redhat.linux:client//创建客户端名称
4.设置共享的lun2和lun3
/iscsi/iqn.20...ver/tpg1/acls> cd ..
/iscsi/iqn.20...2:server/tpg1> cd luns
/iscsi/iqn.20...ver/tpg1/luns> create /backstores/block/lun2
/iscsi/iqn.20...ver/tpg1/luns> create /backstores/fileio/lun3
5.设置监听的网卡IP地址和端口号
/iscsi/iqn.20...ver/tpg1/luns> cd ..
/iscsi/iqn.20...2:server/tpg1> cd portals
/iscsi/iqn.20.../tpg1/portals> create 192.168.19.101 3260
6.此时的目录结构
/iscsi/iqn.20...ver/tpg1/luns> ls /
o- / ...................................................................... [...]
o- backstores ........................................................... [...]
| o- block ............................................... [Storage Objects: 3]
| | o- lun0 ......................... [/dev/sdb1 (1.0GiB) write-thru activated]
| | o- lun1 .......................... [/dev/sdc (5.0GiB) write-thru activated]
| | o- lun2 .................... [/dev/vg0/lv1 (700.0MiB) write-thru activated]
| o- fileio .............................................. [Storage Objects: 1]
| | o- lun3 .......................... [/iscsi (500.0MiB) write-back activated]
| o- pscsi ............................................... [Storage Objects: 0]
| o- ramdisk ............................................. [Storage Objects: 0]
o- iscsi ......................................................... [Targets: 2]
| o- iqn.2019-12.com.redhat.linux.1:server .......................... [TPGs: 1]
| | o- tpg1 ............................................... [gen-acls, no-auth]
| | o- acls ....................................................... [ACLs: 0]
| | o- luns ....................................................... [LUNs: 2]
| | | o- lun0 ...................................... [block/lun0 (/dev/sdb1)]
| | | o- lun1 ....................................... [block/lun1 (/dev/sdc)]
| | o- portals ................................................. [Portals: 1]
| | o- 192.168.19.101:3260 ........................................... [OK]
| o- iqn.2019-12.com.redhat.linux.2:server .......................... [TPGs: 1]
| o- tpg1 ............................................ [no-gen-acls, no-auth]
| o- acls ....................................................... [ACLs: 1]
| | o- iqn.2019-12.com.redhat.linux:client ............... [Mapped LUNs: 2]
| | o- mapped_lun0 ............................... [lun0 block/lun2 (rw)]
| | o- mapped_lun1 .............................. [lun1 fileio/lun3 (rw)]
| o- luns ....................................................... [LUNs: 2]
| | o- lun0 ................................... [block/lun2 (/dev/vg0/lv1)]
| | o- lun1 ........................................ [fileio/lun3 (/iscsi)]
| o- portals ................................................. [Portals: 1]
| o- 192.168.19.101:3260 ........................................... [OK]
o- loopback ...................................................... [Targets: 0]
退出保存
/iscsi/iqn.20...2.server/tpg1> exit
5.关闭selinux,防火墙放行端口号
[root@server /]# setenforce 0
[root@server /]# firewall-cmd --add-port=3260/tcp --per
[root@server /]# firewall-cmd --reload
6.重启服务
[root@server /]# systemctl restart target
总结步骤:
1.将要共享的设备或文件添加进后备存储
2.创建服务器target名称
3.将后备存储添加到某一个target名称底下
4.设置acls(步骤3或4可颠倒)
5.设置监听的网卡IP和端口号
6.退出保存
ls /
就可以很清楚的看到目录结构,然后进行配置
(3)客户端配置
1.装包
[root@client ~]# yum install iscsi-initiator-utils -y
2.重启服务
[root@client ~]# systemctl restart iscsi iscsid
3.发现iSCSI服务器
[root@client ~]# iscsiadm -m discovery -t st -p 192.168.19.101
192.168.19.101:3260,1 iqn.2012-12.com.redhat.linux.2:server
192.168.19.101:3260,1 iqn.2012-12.com.redhat.linux.1:server
4.登陆iSCSI服务器
[root@client ~]# iscsiadm -m node -T iqn.2012-12.com.redhat.linux.1:server -l
Logging in to [iface: default, target: iqn.2012-12.com.redhat.linux.1:server, portal: 192.168.19.101,3260] (multiple)
Login to [iface: default, target: iqn.2012-12.com.redhat.linux.1:server, portal: 192.168.19.101,3260] successful.
[root@client ~]# iscsiadm -m node -T iqn.2012-12.com.redhat.linux.2:server -l
Logging in to [iface: default, target: iqn.2012-12.com.redhat.linux.2:server, portal: 192.168.19.101,3260] (multiple)
iscsiadm: Could not login to [iface: default, target: iqn.2012-12.com.redhat.linux.2:server, portal: 192.168.19.101,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
这里登陆.2服务器时出错,是因为服务器做了acl,因此客户端的iSCSI名称必须和服务端acl设置的一致
5.更改iSCSI客户端名称
[root@client ~]# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2019-12.com.redhat.linux:client
重启服务
[root@client ~]# systemctl restart iscsi iscsid
重新登陆
[root@client ~]# iscsiadm -m node -T iqn.2012-12.com.redhat.linux.2:server -l
Logging in to [iface: default, target: iqn.2012-12.com.redhat.linux.2:server, portal: 192.168.19.101,3260] (multiple)
Login to [iface: default, target: iqn.2012-12.com.redhat.linux.2:server, portal: 192.168.19.101,3260] successful.
此时客户端看起来会多了几个硬盘(sdb,sdc,sdd,sde)
[root@client ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 18G 0 part
├─rhel-root 253:0 0 17G 0 lvm /
└─rhel-swap 253:1 0 1G 0 lvm [SWAP]
sdb 8:16 0 700M 0 disk
sdc 8:32 0 500M 0 disk
sdd 8:48 0 1G 1 disk
sde 8:64 0 5G 1 disk
注意:没有通过acl挂载服务端的硬盘无法分区或格式化
尝试格式化上述.1服务端的sdb1(1G),客户端挂载到了sdd(1G)
[root@client ~]# mkfs.ext4 /dev/sdd
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdd is entire device, not just one partition!
Proceed anyway? (y,n) y
/dev/sdd: Read-only file system while setting up superblock
尝试格式化上述.1服务端的sdc(5G),客户端挂载到了sde(5G)
[root@client ~]# fdisk /dev/sde
最终会提示
fdisk: cannot write disk label: Bad file descriptor
6.格式化后挂载到本地文件夹,这里以sdb为例
[root@client ~]# mkfs.ext4 /dev/sdb
[root@client ~]# mkdir /iscsi
[root@client ~]# blkid | grep sdb
/dev/sdb: UUID="a1d7c240-fbe4-40ae-b7e1-684b9e92d849" TYPE="ext4"
[root@client ~]# vim /etc/fstab
UUID=a1d7c240-fbe4-40ae-b7e1-684b9e92d849 /iscsi ext4 defaults,_netdev 0 0
[root@client ~]# mount -a
[root@client ~]# df -h | grep sdb
/dev/sdb 673M 1.4M 623M 1% /iscsi
注:
1.编辑/etc/fstab文件时,注意使用UUID(因为磁盘的文件名不一定是固定的)。
2.由于共享过来的是网络存储设备,而ISCSI协议是基于TCP/IP网络传输数据的,因此必须在/etc/fstab配置文件中添加上_netdev参数,表示当系统联网后再进行挂载操作,以免系统开机时间过长或开机失败。
3.若想要开机自动启动并挂载,则服务端需要设置target服务开机自动启动,客户端需要设置iscli和iscsid服务自动启动,并编辑fatab文件自动挂载。
5.实测,必须使用-T指定要登陆的节点进行登陆,重启后才可自动挂载服务端硬盘
查看连接会话
[root@client ~]# iscsiadm -m session
tcp: [5] 192.168.19.101:3260,1 iqn.2012-12.com.redhat.linux.2:server (non-flash)
tcp: [6] 192.168.19.101:3260,1 iqn.2012-12.com.redhat.linux.1:server (non-flash)
登出
-T指定节点,不指定则登出所有节点
[root@client ~]# iscsiadm -m node -u
Logging out of session [sid: 5, target: iqn.2012-12.com.redhat.linux.2:server, portal: 192.168.19.101,3260]
Logging out of session [sid: 6, target: iqn.2012-12.com.redhat.linux.1:server, portal: 192.168.19.101,3260]
Logout of [sid: 5, target: iqn.2012-12.com.redhat.linux.2:server, portal: 192.168.19.101,3260] successful.
Logout of [sid: 6, target: iqn.2012-12.com.redhat.linux.1:server, portal: 192.168.19.101,3260] successful.