按照鸟哥的文档,在两台机器上玩一下iscsi+tgt。
Role | Hostname | IP | OS |
---|---|---|---|
Target | maqi-openstack | 10.133.6.83 | CentOS |
Initiator | maqi-kilo | 10.133.16.195 | Ubuntu |
命令总结
扫描某个portal上的所有target iscsiadm -m discovery -t sendtargets -p 10.133.6.83:3260
显示initiator上所有已知的target iscsiadm -m node
显示initiator上login的session iscsiadm -m session
在initiator上login某个target iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target --login
logout某个target iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target --logout
在initiator上delete target iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target -o delete
在initiator上删除所有target iscsiadm -m node -o delete
更新某个target iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target -p 10.133.6.83:3260 -o update -n node.startup -v automatic
显示某个target的细节 iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target -p 10.133.6.83:3260 -o show
安装软件
Target上安装scsi-target-utils
sudo yum install scsi-target-utils
他提供:
- /etc/tgt/targets.conf: 主要配置文件,设定要分享的磁盘格式与哪几颗;
- /usr/sbin/tgt-admin: 在线查询、删除 target 等功能的设定工具;
- /usr/sbin/tgt-setup-lun:建立 target 以及设定分享的磁盘与可使用的客户端等工具软件。
- /usr/sbin/tgtadm: 手动直接管理的管理员工具 (可使用配置文件取代);
- /usr/sbin/tgtd: 主要提供 iSCSI target 服务的主程序;
- /usr/sbin/tgtimg: 建置预计分享的映像文件装置的工具 (以映像文件仿真磁盘);
Initiator上安装iscsi-initiator-utils
sudo yum install iscsi-initiator-utils
他提供:
- /etc/iscsi/iscsid.conf:主要的配置文件,用来连结到 iSCSI target 的设定;
- /sbin/iscsid: 启动 iSCSI initiator 的主要服务程序;
- /sbin/iscsiadm: 用来管理 iSCSI initiator 的主要设定程序;
- /etc/init.d/iscsid: 让本机模拟成为 iSCSI initiater 的主要服务;
- /etc/init.d/iscsi: 在本机成为 iSCSI initiator 之后,启动此脚本,让我们可以登入 iSCSI target。所以 iscsid 先启动后,才能启动这个服务。为了防呆,所以 /etc/init.d/iscsi 已经写了一个启动指令, 启动 iscsi 前尚未启动 iscsid ,则会先呼叫 iscsid 才继续处理 iscsi 喔!
配置target
target主机上有100G的vdd,划分出两个2G点分区vdd1、vdd2
配置/etc/tgt/targets.conf:
[root@maqi-openstack ~]# cat /etc/tgt/targets.conf include /etc/tgt/stack.d/* <target iqn.2015-12.com.example:maqi-first-target> direct-store /dev/vdd1 # Becomes LUN 1 direct-store /dev/vdd2 # Becomes LUN 2 write-cache off vendor_id MyCompany Inc. incominguser myName myPassword </target>
可配参数很多。这里加上了login的用户名密码。
启动tgtd
[root@maqi-openstack ~]# systemctl start tgtd [root@maqi-openstack log]# systemctl status tgtd tgtd.service - tgtd iSCSI target daemon Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled) Active: active (running) since Thu 2015-12-24 08:11:18 UTC; 3min 1s ago Process: 17527 ExecStop=/usr/sbin/tgtadm --op delete --mode system (code=exited, status=0/SUCCESS) Process: 17511 ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null (code=exited, status=0/SUCCESS) Process: 17509 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Process: 17617 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS) Process: 17535 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS) Process: 17533 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Process: 17530 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS) Main PID: 17529 (tgtd) CGroup: /system.slice/tgtd.service └─17529 /usr/sbin/tgtd -f Dec 24 08:11:12 maqi-openstack.novalocal tgtd[17529]: tgtd: bs_init(390) use signalfd notification Dec 24 08:11:18 maqi-openstack.novalocal tgtd[17529]: tgtd: device_mgmt(246) sz:72 params:path=/dev/cinder-new-volume/vo...38344 Dec 24 08:11:18 maqi-openstack.novalocal tgtd[17529]: tgtd: bs_thread_open(412) 16 Dec 24 08:11:18 maqi-openstack.novalocal tgt-admin[17535]: Both SCSI INQUIRY and fetching ATA information failed on /dev/vdd2 Dec 24 08:11:18 maqi-openstack.novalocal tgt-admin[17535]: inquiry: scsi status: Unknown status Dec 24 08:11:18 maqi-openstack.novalocal tgt-admin[17535]: inquiry: failed, res=-1 Dec 24 08:11:18 maqi-openstack.novalocal tgt-admin[17535]: read capacity (16): scsi status: Unknown status Dec 24 08:11:18 maqi-openstack.novalocal tgt-admin[17535]: READ CAPACITY (16) failed [res=-1], try with '-v' Dec 24 08:11:18 maqi-openstack.novalocal tgtd[17529]: tgtd: device_mgmt(246) sz:15 params:path=/dev/vdd2 Dec 24 08:11:18 maqi-openstack.novalocal tgtd[17529]: tgtd: bs_thread_open(412) 16 Dec 24 08:11:18 maqi-openstack.novalocal systemd[1]: Started tgtd iSCSI target daemon. Hint: Some lines were ellipsized, use -l to show in full.
看到:
Dec 24 08:11:18 maqi-openstack.novalocal tgt-admin[17535]: Both SCSI INQUIRY and fetching ATA information failed on /dev/vdd2 [root@maqi-openstack log]# sg_inq /dev/vdd1 Both SCSI INQUIRY and fetching ATA information failed on /dev/vdd1 [root@maqi-openstack log]# sg_inq /dev/vdd2 Both SCSI INQUIRY and fetching ATA information failed on /dev/vdd2
只看到一个LUN,而且用到了vdd2:
[root@maqi-openstack log]# tgt-admin --show ...... Target 8: iqn.2015-12.com.example:maqi-first-target <=== 我们的target System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller <=== LUN 0 是controller,不是真正可用的LUN SCSI ID: IET 00080000 SCSI SN: beaf80 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk <=== LUN1是第一个可用的LUN,type=disk SCSI ID: IET 00080001 SCSI SN: beaf81 Size: 2147 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: rdwr Backing store path: /dev/vdd2 Backing store flags: Account information: ACL information: ALL [root@maqi-openstack log]# tgt-admin --show | grep vdd Backing store path: /dev/vdd2 [root@maqi-openstack log]# netstat -tlunp | grep tgt tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 17529/tgtd tcp6 0 0 :::3260 :::* LISTEN 17529/tgtd
这个问题先不管,以后再说。
配置initiator
扫描target
先扫描一下target主机上有多少target(默认扫描3260端口):
root@maqi-kilo:/etc/iscsi# iscsiadm -m discovery -t sendtargets -p 10.133.6.83
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-04dfd6b7-eeb3-40b2-9284-228582838344
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-47964a24-a8e1-4497-a1dc-a77c709c6a67
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-61732a4f-cf62-4410-afc0-dec66bded985
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-bf7f32d0-6163-4d66-b712-ecf8ebc2ef63
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-f702b34b-af5d-47c2-b2d0-eefc7e3b50a5
10.133.6.83:3260,1 iqn.2015-12.com.example:maqi-first-target
选项与参数:
- -m discovery :使用侦测的方式进行 iscsiadmin 指令功能;
- -t sendtargets :透过 iscsi 的协议,侦测后面的设备所拥有的 target 数据
- -p IP:port :就是那部 iscsi 设备的 IP 与端口,不写端口预设是 3260 啰!
在Ubuntu下,这些信息会放在/etc/iscsi/nodes/目录:
root@maqi-kilo:/etc/iscsi# tree /etc/iscsi/ -L 2
/etc/iscsi/
├── initiatorname.iscsi
├── iscsid.conf
├── nodes
│ ├── iqn.2010-10.org.openstack:volume-04dfd6b7-eeb3-40b2-9284-228582838344
│ ├── iqn.2010-10.org.openstack:volume-47964a24-a8e1-4497-a1dc-a77c709c6a67
│ ├── iqn.2010-10.org.openstack:volume-61732a4f-cf62-4410-afc0-dec66bded985
│ ├── iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
│ ├── iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
│ ├── iqn.2010-10.org.openstack:volume-bf7f32d0-6163-4d66-b712-ecf8ebc2ef63
│ ├── iqn.2010-10.org.openstack:volume-f702b34b-af5d-47c2-b2d0-eefc7e3b50a5
│ └── iqn.2015-12.com.example:maqi-first-target
└── send_targets
└── 10.133.6.83,3260
11 directories, 2 files
在CentOS上,在/var/lib/iscsi/nodes/目录下:
[root@maqi-openstack devstack]# tree /etc/iscsi/
/etc/iscsi/
├── initiatorname.iscsi
└── iscsid.conf
0 directories, 2 files
[root@maqi-openstack devstack]# tree /var/lib/iscsi/ -L 2
/var/lib/iscsi/
├── ifaces
├── isns
├── nodes
│ ├── iqn.2010-10.org.openstack:volume-32bf53fd-9d53-465e-b853-698cfe5843cc
│ ├── iqn.2010-10.org.openstack:volume-3b26209f-8e31-4f0a-b5d6-cf063de16625
│ ├── iqn.2010-10.org.openstack:volume-56c4f328-8e77-4ac7-aa85-6332605eea51
│ ├── iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
│ ├── iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
│ ├── iqn.2010-10.org.openstack:volume-94b69d69-229d-45e3-9acc-2dca423f958c
│ ├── iqn.2010-10.org.openstack:volume-bb154e5e-83d9-45c8-b27f-7981310da35a
│ ├── iqn.2010-10.org.openstack:volume-bdd9920b-6601-4a35-b39a-5261590b2899
│ ├── iqn.2010-10.org.openstack:volume-ede0b75e-75c8-4c56-a372-052ebcb5e9db
│ └── iqn.2010-10.org.openstack:volume-ffe78d8a-d817-417f-9c9a-0c922354c62b
├── send_targets
├── slp
└── static
16 directories, 0 files
这些target下的default子目录包含了target的详细信息:
root@maqi-kilo:/etc/iscsi/nodes/iqn.2015-12.com.example:maqi-first-target/10.133.6.83,3260,1# more /etc/iscsi/nodes/iqn.2015-12.com.example\:maqi-first-target/10.133.6.83\,3260\,1/default
# BEGIN RECORD 2.0-873
node.name = iqn.2015-12.com.example:maqi-first-target
node.tpgt = 1
node.startup = manual
node.leading_login = No
iface.iscsi_ifacename = default
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
node.discovery_address = 10.133.6.83
node.discovery_port = 3260
node.discovery_type = send_targets
在EBS做的multipath环境下:
[root@compute-3 iqn.2010-10.org.openstack:volume-cc6b3cf2-246f-4d7a-8b57-92b6677952dc]# ll
total 0
drw-------. 2 root root 28 Dec 30 04:37 10.254.4.22,3260,1
drw-------. 2 root root 28 Dec 22 18:51 10.254.4.25,3260,1
所以,扫描完成之后,在initiator的文件系统中能看到对应的target。
用iscsiadm -m node
命令同样可以看到:
root@maqi-kilo:/etc/iscsi/nodes# iscsiadm -m node
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-61732a4f-cf62-4410-afc0-dec66bded985
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-f702b34b-af5d-47c2-b2d0-eefc7e3b50a5
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-04dfd6b7-eeb3-40b2-9284-228582838344
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-47964a24-a8e1-4497-a1dc-a77c709c6a67
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-bf7f32d0-6163-4d66-b712-ecf8ebc2ef63
10.133.6.83:3260,1 iqn.2015-12.com.example:maqi-first-target
选项与参数:
- -m node:找出目前本机上面所有侦测到的 target 信息,可能并未登入哦
和nodes目录下完全一致:
root@maqi-kilo:/etc/iscsi/nodes# ls /etc/iscsi/nodes
iqn.2010-10.org.openstack:volume-04dfd6b7-eeb3-40b2-9284-228582838344
iqn.2010-10.org.openstack:volume-47964a24-a8e1-4497-a1dc-a77c709c6a67
iqn.2010-10.org.openstack:volume-61732a4f-cf62-4410-afc0-dec66bded985
iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
iqn.2010-10.org.openstack:volume-bf7f32d0-6163-4d66-b712-ecf8ebc2ef63
iqn.2010-10.org.openstack:volume-f702b34b-af5d-47c2-b2d0-eefc7e3b50a5
iqn.2015-12.com.example:maqi-first-target
和target端也一致:
[root@maqi-openstack admin]# tgt-admin --show | grep Target
Target 1: iqn.2010-10.org.openstack:volume-04dfd6b7-eeb3-40b2-9284-228582838344
Target 2: iqn.2010-10.org.openstack:volume-47964a24-a8e1-4497-a1dc-a77c709c6a67
Target 3: iqn.2010-10.org.openstack:volume-61732a4f-cf62-4410-afc0-dec66bded985
Target 4: iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
Target 5: iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
Target 6: iqn.2010-10.org.openstack:volume-bf7f32d0-6163-4d66-b712-ecf8ebc2ef63
Target 7: iqn.2010-10.org.openstack:volume-f702b34b-af5d-47c2-b2d0-eefc7e3b50a5
Target 8: iqn.2015-12.com.example:maqi-first-target
login target
扫描到target之后,下一步就是使用这些target了,这个动作称为“login”。
在login之前,先看看有没有iscsi session:
root@maqi-kilo:~# iscsiadm -m session
iscsiadm: No active sessions.
没有,很好。开始login:
root@maqi-kilo:~# iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target --login
Logging in to [iface: default, target: iqn.2015-12.com.example:maqi-first-target, portal: 10.133.6.83,3260] (multiple)
iscsiadm: Could not login to [iface: default, target: iqn.2015-12.com.example:maqi-first-target, portal: 10.133.6.83,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals
关于认证失败,请参考http://blog.csdn.net/sinchb/article/details/8433994
貌似配置复杂,所以先把targets.conf中的用户名密码去掉,重启tgtd。
再次登录之后就能看到一个session了:
root@maqi-kilo:~# iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target --login
root@maqi-kilo:~# iscsiadm -m session
tcp: [3] 10.133.6.83:3260,1 iqn.2015-12.com.example:maqi-first-target
Note:加-p参数可以指定portal,如-p 10.133.6.83:3260
此时fdisk -l可以看到多了一个/dev/sda,大小为2G:
root@maqi-kilo:~# fdisk -l
......
Disk /dev/sda: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sda doesn't contain a valid partition table
这个disk就像普通磁盘设备一样可以直接使用。
login之后,需要在target端看到这个LUN多了一些initiator的信息:
Target 8: iqn.2015-12.com.example:maqi-first-target
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 3
Initiator: iqn.1993-08.org.debian:01:708b992430d alias: maqi-kilo
Connection: 0
IP Address: 10.133.16.195
logout target
用完之后,需要logout这个设备:
root@maqi-kilo:~# iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target --logout
Logging out of session [sid: 3, target: iqn.2015-12.com.example:maqi-first-target, portal: 10.133.6.83,3260]
Logout of [sid: 3, target: iqn.2015-12.com.example:maqi-first-target, portal: 10.133.6.83,3260] successful.
root@maqi-kilo:~# iscsiadm -m session
iscsiadm: No active sessions.
在initiator上delete target
root@maqi-kilo:~# iscsiadm -m node -T iqn.2015-12.com.example:maqi-first-target -o delete
root@maqi-kilo:~# iscsiadm -m node
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-6adf7e3b-1903-4ee5-9b31-2af066570e7b
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-61732a4f-cf62-4410-afc0-dec66bded985
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-f702b34b-af5d-47c2-b2d0-eefc7e3b50a5
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-8f7754d9-a407-4798-be01-26668defec7e
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-04dfd6b7-eeb3-40b2-9284-228582838344
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-47964a24-a8e1-4497-a1dc-a77c709c6a67
10.133.6.83:3260,1 iqn.2010-10.org.openstack:volume-bf7f32d0-6163-4d66-b712-ecf8ebc2ef63
选项与参数:
- –logout :就是注销 target,但是并没有删除 /var/lib/iscsi/nodes/ 内的数据
- -o delete:删除后面接的那部 target 链接信息 (/var/lib/iscsi/nodes/*)
- -o update:更新相关的信息
- -o new :增加一个新的 target 信息。
文档
其他问题
discovery
和discoverydb
有何区别?
https://groups.google.com/forum/#!msg/open-iscsi/WPiI46L3xCo/pn3RHroDUNUJThe difference is what you quoted from the man page where discovery mode uses the iscsid.conf settings and discoverydb uses the discovery record settings if it exists. The latter is useful when you have multiple
targets that you need to do discovery to, but something like the CHAP
settings are different for each one.The discovery settings you are most interested in are the ones you see
in the README/iscsid.conf discovery.*:是不是这个意思:
discovery
只会用iscsid.conf中的配置来操作target;而discoverydb
可以用动态的根据发现的target的属性来操作target?
root@maqi-kilo:/etc/iscsi/nodes# iscsiadm -m discoverydb -t sendtargets -p 10.133.6.83
# BEGIN RECORD 2.0-873
discovery.startup = manual
discovery.type = sendtargets
discovery.sendtargets.address = 10.133.6.83
discovery.sendtargets.port = 3260
discovery.sendtargets.auth.authmethod = None
discovery.sendtargets.auth.username = <empty>
discovery.sendtargets.auth.password = <empty>
discovery.sendtargets.auth.username_in = <empty>
discovery.sendtargets.auth.password_in = <empty>
discovery.sendtargets.timeo.login_timeout = 15
discovery.sendtargets.use_discoveryd = No
discovery.sendtargets.discoveryd_poll_inval = 30
discovery.sendtargets.reopen_max = 5
discovery.sendtargets.timeo.auth_timeout = 45
discovery.sendtargets.timeo.active_timeout = 30
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
# END RECORD
什么时候rescan?
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/scanningnewdevs-iscsi.html
With some device models (e.g. from EMC and Netapp), however, a single target may have multiple logical units and/or portals. In this case, issue a sendtargets command to the host first to find new portals on the target. Then, rescan the existing sessions using:iscsiadm -m session --rescan
You can also rescan a specific session by specifying the session’s SID value, as in:iscsiadm -m session -r SID --rescan
[4]
即:一个target有多个LUN and/or portals,先用sendtargets发现新的portals,再对existing sessions执行rescanIf your device supports multiple targets, you will need to issue a sendtargets command to the hosts to find new portals for each target. Then, rescan existing sessions to discover new logical units on existing sessions (i.e. using the –rescan option).
即:支持多个targets的话,先对每个target发送sendtargets来发现新的portals,再对existing sessions执行rescan来发现新的LUN
https://groups.google.com/forum/#!topic/open-iscsi/Us6Pne5fVUs
Q: I have a question, when you add a new lun to iscsi target, if client
iSCSI initiator is already logged in to target, do i have to restart
the client iSCSI initiator for it to see the new lun? how can i make
client iSCSI initiator see new lun with out restarting iSCSI initiator
and losing connection to current Target and Lun’s?
A:
If you use a target that sends the iscsi async pdu with
REPORT_LUNS_DATA_CHANGED then the new luns should be added automatically
for you (deleted luns are not removed though and need to be manually
deleted).
If it is not getting automatically added then you need to run either
//rescan all sessions for new luns
iscsiadm -m session –rescan
// rescan specific session
iscsiadm -m session -R $SID –rescan
(see the README for other variants)
This commands only add new luns. If you want something that adds and
deletes luns then use the rescan-bus.sh script in the sg utils.