-Rajeev Karamchedu
Introduction
How does it work?
Implementation Environment
Target Setup on the Filer
Initiator Setup
Automounting iSCSI
How Fast Is It ?
Implementing iSCSI Security
iSNS, Internet Storage Name Service
References
Introduction
iSCSI (Internet Small Computer System Interface), an IP-based storage networking standard for connecting data storage entities, developed by the Internet Engineering Task Force (IETF). Essentially a protocol carrying SCSI commands over IP networks, iSCSI can be a cheap and effective alternative for storage administrators who are otherwise struggling to strike a balance between the high-cost and complex implementations of FC-SAN (Fibre Channel Storage Area Networks) and the performance limitations and overhead of NAS (Network Attached Storage).
How does it work?
iSCSI is a Client-Server based protocol, except that the client process requesting data is called an “initiator” and the server process serving the data is called a “target”. In an iSCSI implementation, the storage that is offered by the target appears as local disk to the initiator. The client can make block-based operations on that storage. Since this involves formatting the disk, partitioning, creating a file system on the disk, no more than one initiator can have read/write access to the specific iSCSI device at a time. However, it is possible that you can mount a iSCSI device as read-only on multiple initiators.
This document reflects the following environment-specific implementation details and command sets. Consult the Resources section for links to other implementations.
Initiator: Intel P4 Desktop running Suse 9.2 (2.6.8-24.14-smp)
Fibre Channel implementation uses WWPN (World Wide Port Names) and WWNN (World Wide Node Names) to identify devices. iSCSI uses iSCSI addresses. Once all the targets and initiators are assigned and configured with iSCSI addresses, these “nodes” need to know about each other, akin to the DNS resolution. Once they are aware of each other, they can communicate and the initiator can access the storage from the target. Details of how this is accomplished is discussed later in this document.
- iqn Format: iqn.yyyy-mm.backward_naming_authority:unique_device_name
For e.g. on a linux box, the iSCSI initiator address may look like his:
iqn.1987-05.com.cisco:01.12a14c2dcab9 - eui Format: eui.nnnnnnnnnnnnnnnn
eui Format is used when a manufacturer is already registered with the IEEE Registration Authority and uses EUI-64 formatted worldwide unique names for its products
Target Setup on the Filer
The iSCSI protocol on the filer is implemented via a software driver, much like the Linux software driver. Data ONTAP includes a virual adapter for iSCSI communications ( iswt). This adapter provides two logical iSCSI adapaters: iswta and iswtb for active and failover purposes.
-
Make sure you have a valid iSCSI license. Depending upon your filer support status, make and model, you can get a iSCSI license for free via the Netapp iSCSI enabling program. Once the license is installed, you can stop and start the iscsi service on the filer’s console using these commands:
filer:> iscsi {start|stop}
- The filer uses gives the iscsi interface a default target address. It uses the filer serial number to generate the nodename. You can choose to keep it or change it following the naming schemes.
filer:> iscsi nodename
iSCSI target nodename: iqn.1992-08.com.netapp:sn.11111111 - Set the filer iSCSI security to NONE. We will discuss later how to add security.
filer:> iscsi security default -s none
default sec is none - Setup LUNs on the Target (Filer):
Using Data ONTAP, disk drives are logically carved into Volumes. Volumes can hold files, LUNs and qtree. A qtree is a sub-directory of the root of a volume. Qtrees can hold LUNs, files and other sub-directories. It is also possible to create two qtrees, one containing LUNs for iSCSI purposes and another containing files/directories for NFS purposes. For the purposes of testing, we chose to have two qtrees in a volume, one for iSCSI LUNs and one for NFS sharing. Performance data is collected individually on these areas for comparision.
When setting up volumes on the filer for iSCSI purposes, note the following NetApp recommendations:
- Do not create volumes on the filer’s root volume (/vol/vol0)
- Use qtrees to separate iSCSI LUNs and non-iSCSI files/directories
- Use qtrees to hold all LUNs for a particular host
- Set snapshot reserve percentage to 0% (or disable snapshots)
- Turn off automatic snapshot scheduling
- Ensure that the volume option create_ucode is turned ON.
We used a spare volume called “scratch” that we already had on the filer. We created a qtree called “iscsi”and in there we created a LUN called iscsi0.
filer:> qtree create /vol/scratch/iscsi
We also disabled snapshots on this volume completely.
filer:> vol options scratch
nosnap=on, nosnapdir=on, minra=off, no_atime_update=off,
raidtype=raid4, raidsize=8, nvfail=off, snapmirrored=off,
resyncsnaptime=60, create_ucode=on, convert_ucode=off,
maxdirsize=10240, fs_size_fixed=off, create_reserved=off,
fractional_reserve=100
filer>
Use the Data ONTAP “lun setup” commands to create and setup LUNS for iSCSI. lun setup is a command line wizard that will walk you through the process of creating LUNS and associated igroups and the mapping process. In Data ONTAP, LUNS and igroups are created. Think if an igroup and a unix group. It contains a list of all initiators as members. When a LUN is mapped to an igroup, all the initiators listed in the igroup have access to that LUN provided the underlying iSCSI authentication is successful (more on that later).filer:> lun setup
This setup will take you through the steps needed to create LUNs
and to make them accessible by initiators. You can type ^C (Control-C)
at any time to abort the setup and no unconfirmed changes will be made
to the system.Do you want to create a LUN? [y]: y
OS type of LUN (image/solaris/windows/hpux/aix/linux) [image]:A LUN path must be absolute. A LUN can only reside in a volume
or qtree root. For example, to create a LUN with the name “lun0″
in the qtree root /vol/vol/q0, specify the path as “/vol/vol1/q0/lun0″.Enter LUN path: /vol/scratch/iscsi/iscsi0
A LUN can be created with or without space reservations being enabled.
Space reservation guarantees that data writes to that LUN will never fail.Do you want the LUN to be space reserved? [y]: y
Size for a LUN is specified in bytes. You can use single-character
multiplier suffixes: b(sectors), k(KB), m(MB), g(GB) or t(TB).Enter LUN size: 4g
You can add a comment string to describe the contents of the LUN.
Please type a string (without quotes), or hit ENTER if you don’t
want to supply a comment.Enter comment string: iSCSI Testing
The LUN will be accessible to an initiator group. You can use an
existing group name, or supply a new name to create a new initiator
group. Enter `?’ to see existing initiator group names.Name of initiator group[]: iSCSI_Test
Type of initiator group iSCSI_TEst (FCP/iSCSI)[FCP]: iscsi
An iSCSI initiator group is a collection of initiator node names. Each
node name can begin with either `eui.’ or `iqn.’ and should be in the
following formats: eui.{EUI-64 address} or iqn.yyyy-mm.{reserved domain
name}:{any string}.
Eg. iqn.2001-04.com.acme:storage.tape.sys1.xyz or eui.02004567A25678DYou can separate node names by commas. Enter `?’ to display a list of
connected initiators. Hit ENTER when you are done adding port names to this
group.[You can obtain the initiator node name from /etc/initiatorname.iscsi on the initiator system.]
Enter comma separated nodenames: iqn.1987-05.com.cisco:01.12a14c2dcab9
The initiator group has an associated OS type. The following are
currently supported: solaris, windows, hpux, aix, linux, or default
OS type of initiator group “iSCSI_Test” [windows]: defaultThe LUN will be accessible to all the initiators in the
initiator group. Enter `?’ to display LUNs already in use
by one or more initiators in group “iSCSI_Test“.LUN ID at which initiator group “iSCSI_Test” sees “/vol/vol1/lun0″ [0]:
[ If you press Enter to accept the default, Data ONTAP issues the lowest
valid unallocated LUN ID to map it to the initiator, starting with
zero. Alternatively, you can enter any valid number. For information
about valid LUN IDs for your host initiator, see the documentation
provided with your iSCSI host Initiator Support Kit or with your SAN
Host Attach Kit for iSCSI Protocol on your host. ]After pressing “Enter” here, the program displays the summary of the selections for confirmation.
LUN Path : /vol/vol1/q0/lun0
OS Type : windows
Size : 4g
Comment : iSCSI Test LUN
Initiator Group : iSCSI_Test
Initiator Group Type : iscsi
Initiator Group Members : iqn.1987-05.com.cisco:01.12a14c2dcab9
Mapped to LUN-ID : 0Do you want to accept this configuration? [y]
Do you want to create another LUN? [n]
At this point, the target is pretty much setup and is ready to receive iSCSI requests.
- In SUSE 9.2, the open source version of the Cisco iSCSI driver comes pre-installed. The module has to be loaded using the command “modprobe iscsi”. Ensure that it is loaded
suse92:# lsmod | grep iscsi
iscsi 208428 2
scsi_mod 121412 6 iscsi,sg,st,sr_mod,libata,sd_mod
suse92: # - The configuration file for iscsi is /etc/iscsi.conf. Make sure the file has the following permissions. Atleast one vendor has shipped this file in the past with world-readable permissions. This file is used to store the authentication information (to be dealt later), so ensure that this file is secure
suse92:# ls -l /etc/iscsi.conf
-rw——- 1 root root 17264 Jun 11 16:36 /etc/iscsi.conf
suse92:# cp /etc/iscsi.conf /etc/iscsi.conf.orig - Edit the file /etc/iscsi.conf and set the following values
DiscoveryAddress=[Enter the Netapp Filer IP Address]
Continuous=no
HeaderDigest=never
DataDigest=never
ImmediateData=yes - Start the iscsi service
suse92: # /etc/init.d/iscsi start
- Verify that you were able to see that iSCSI devices from the target using the iscsi-ls command
suse92: # /sbin/iscsi-ls
*******************************************************************************
Cisco iSCSI Driver Version … 4.0.197 ( 21-May-2004 )
*******************************************************************************
TARGET NAME : iqn.1992-08.com.netapp:sn.11111111
TARGET ALIAS :
HOST NO : 2
BUS NO : 0
TARGET ID : 0
TARGET ADDRESS : 172.xx.xx.xx:3260
SESSION STATUS : ESTABLISHED AT Sat Jun 11 16:36:35 2005
NO. OF PORTALS : 1
PORTAL ADDRESS 1 : 172.xx.xx.xx:3260,2
SESSION ID : ISID 00023d000001 TSID 585
******************************************************************************* - You can also check out the file /var/lib/iscsi/bindings file to find out which iSCSI targets have been discovered by the iSCSI daemon and the LUN IDs. This file is where the Linux iSCSI driver stores and gets its persistent binding information from.
suse92:/var/lib/iscsi # cat /var/lib/iscsi/bindings
# iSCSI bindings, file format version 1.0.
# NOTE: this file is automatically maintained by the iSCSI daemon.
# You should not need to edit this file under most circumstances.
# If iSCSI targets in this file have been permanently deleted, you
# may wish to delete the bindings for the deleted targets.
#
# Format:
# bus target iSCSI
# id id TargetName
#
0 0 iqn.1992-08.com.netapp:sn.11111111 - Look in the /var/log/messages to see the activity
Jun 11 16:18:24 suse92 kernel: iSCSI: bus 0 target 0 = iqn.1992-08.com.netapp:sn.11111111
Jun 11 16:18:24 suse92 kernel: iSCSI: bus 0 target 0 portal 0 = address 172.xx.xx.xx port 3260 group 2
Jun 11 16:18:24 suse92 kernel: iSCSI: starting timer thread at 418674703
Jun 11 16:18:24 suse92 kernel: iSCSI: bus 0 target 0 trying to establish session to portal 0, address 172.xx.xx.xx port 3260 group 292 kernel: iSCSI: bus 0 target 0 established session #1, portal 0, address 172.xx.xx.xx port 3260 group 2
Jun 11 16:18:24 suse92 kernel: Vendor: NETAPP Model: LUN Rev: 0.2
Jun 11 16:18:24 suse92 kernel: Type: Direct-Access ANSI SCSI revision: 04
Jun 11 16:18:24 suse92 kernel: iSCSI: session iscsi_bus 0 target id 0 recv_cmd cdb 0×0, status 0×2, response 0×0, senselen 18, key 06, ASC/ASCQ 29/00, itt 4 to (4 0 0 0), iqn.1992-08.com.netapp:sn.11111111
Jun 11 16:18:24 suse92 kernel: iSCSI: Sense f0000600 0000000a 00000000 29000000 0000
Jun 11 16:18:24 suse92 kernel: SCSI device sda: 8388608 512-byte hdwr sectors (4295 MB)
Jun 11 16:18:24 suse92 kernel: SCSI device sda: drive cache: write through
Jun 11 16:18:24 suse92 kernel: SCSI device sda: 8388608 512-byte hdwr sectors (4295 MB)
Jun 11 16:18:24 suse92 kernel: SCSI device sda: drive cache: write through
Jun 11 16:18:24 suse92 kernel: sda: sda1
Jun 11 16:18:24 suse92 kernel: Attached scsi disk sda at scsi4, channel 0, id 0, lun 0
Jun 11 16:18:24 suse92 kernel: Attached scsi generic sg0 at scsi4, channel 0, id 0, lun 0, type 0Jun 11 16:18:24 suse
- We see that the 4GB LUN we have created on the target is available to the initiator at the scsi device address sda. Due to the fact that the SCSI devices nodes are mapped to the LUNS dynamically upon detection, there is no guarantee that the device node mappings will be consistent across reboots. In order to provide a consistent device mapping, the linux iscsi driver provides a new device tree under /dev/iscsi that provide device paths to the iSCSI LUNs. For e.g., the device path /dev/iscsi/bus0/target0/lun0/disk refers to the entire LUN with ID 0, on target 0, bus 0. The last part “disk” refers to the entire disk (in Solaris terms, it would be slice 2, s2)
In practice however, on the SUSE Linux system, we noticed that that is not the case. Using udev, SUSE prefers to create the persistent device names in /dev/disk. In our setup, after loading the module, we found the dev entries created under /dev/disk/by-id/iqn.1992-08.com.netapp:sn.11111111. Actually, they are just symlinks to /dev/sdX devices.
For the client system (initiator), it is just another SCSI disk. You can now use the OS specific disk formatting tools such as fdisk and mkfs to format and partition the disk and install the filesystem of our choice. As we create partitions in that disk, you will notice that the new device paths will become available in /dev/disk/by-id directory for mounting. In our setup, we chose to create 1 primary partition with id 1 and give it all the space. We also a ext3 on it. After doing that, these are the entries you see in /dev/disk/by-id directory as a result:
suse92:/dev/disk/by-id # ls -l
lrwxrwxrwx 1 root root 9 Jun 12 18:46 iscsi-iqn.1992-08.com.netapp:sn.11111111-0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 12 18:46 iscsi-iqn.1992-08.com.netapp:sn.11111111-0p1 -> ../../sda1To mount the filesystem, we should use the appropriate device name.suse92: mount /dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:sn.11111111-0p1 /mnt
suse92: cd /mnt
suse92: df -h .Filesystem Size Used Avail Use% Mounted on
/dev/sda1 4.0G 33M 3.8G 1% /mntYou will also notice that the usable space is not equal to the 4 GB. That is because we have build a file system on it and that will take up some space.
At this point, we have a working iSCSI implementation. The following sections describe some of the important aspects of the implementation.Automounting iSCSI
Note that iSCSI uses IP Protocol for searching and accessing LUNS. During the normal system boot up, however, when the system is ready to mount its devices, the IP protocol is not yet configured and hence it cannot see the LUNs. In order to mount the iSCSI LUNs at boot time, the Linux driver uses the file /etc/fstab.iscsi. The format is identical to the OS /etc/fstab. After the system boots up, the iSCSI driver issues the command iscsi-mountall which consults the file /etc/fstab.iscsi and mounts the file systems provided within.
suse92: cat /etc/fstab.iscsi
# /etc/fstab.iscsi file for filesystems built on iscsi devices.
#
# A typical entry here would look like:
# /dev/disk/by-id/iscsi-iqn.2001-04.example.com:storage:disk2.sys1 /mnt ext2 defaults 0 0
#
/dev/disk/by-id/iscsi-iqn.1992-08.com.netapp:sn.11111111-0p1 /mnt ext3 defaults 0 0
How Fast Is It?
In order to see how fast iSCSI really is in comparision with NFS, we used Bonnie, a tool used to benchmark filesystem operations. As discussed before, we have one volume on the Netapp Filer called scratch that has both iSCSI LUNs and NFS exports. We mounted the NFS share from /vol/scratch/nfs at /local/scratch_nfs and we know that we have iSCSI LUN mounted at /mnt. The Dell Desktop we were using has about 1GB of Memory, so we asked bonnie to create two volumes and specified 1GB of size each to take care of any cacheing issues (flags to the command are: -s 1024 -v 2).
| Sequential Output (nosync) | Sequential Input | Random Seek | |||||
|---|---|---|---|---|---|---|---|
| Protocol | File Size | per char KB/s | per block KB/sec | rewrite KB/s | per char KB/s | per block KB/s | Random Seeks per sec |
| iSCSI | 2048 | 13408 | 13896 | 5180 | 7875 | 9766 | 277.2 |
| NFS | 2048 | 10968 | 11134 | 4474 | 11260 | 11181 | 338.6 |
Implementing iSCSI Security
In this section, we will outline how iSCSI security is implemented and managed, specific to the environment described above. In an iSCSI implementation, two complementary security mechanisms are employed.
- In-Band Authentication: Authentication carried out between targets and initiators, using CHAP
- Packet Protection: Achieved by implementing IPSec on the packets
On the Netapp Target Filer, the iscsi service can be setup to perform authentication. When the filer recieves a login request from an initiator to begin an iSCSI sessions, the filer can be configured to:
- perform (inbound and/or outbound) CHAP based authentication
- Deny service (service is denied in this case if the initiators are not in the list.)
- None - Filer is not setup to do authentication and all iSCSI sessions are allowed by default.
To bolster our current iSCSI setup, we will add inbound CHAP authentication on the filer as the default method. We will specify a inbound username and password on the Filer and activate a default CHAP authentication method. We will then use the same username and password and configure the initiator to send to the target at connection time. By doing so, only those initiators that are provided with the Target authentication information will be able to access the LUNs.
- On the filer, generate a random CHAP password
filer:> iscsi security generate
Generated Random Secret: 0xbb86df2afa74063955e6e95607031116 - Define a CHAP username and password as a default authentication method
filer:> iscsi security default -s CHAP -n iSCSI_ntap -p 0xbb86df2afa74063955e6e95607031116 - Verify the setup on the target
filer:> iscsi security show
default sec is CHAP Inbound password: **** Inbound username: iSCSI_ntap Outbound password: **** Outbound username: - Login to the initiator system and specify the authentication credentials
suse92:# vi /etc/iscsi.conf
Below the line “DiscoveryAddress”, add these two lines and make sure there is a tab before the variable. The reason for tabs is that you can specify multiple usernames and passwords for different targets and the file is parsed using indents by the iscsi driver.
DiscoveryAddress=[IP Address/HostName of the Target]
[INSERT TAB]OutgoingUsername=iSCSI_ntap
[INSERT TAB]OutgoingPassword=0xbb86df2afa74063955e6e95607031116
Note that on the target, the username and password are specified as “incoming username” and “incoming passwords” and on the initiator, they are configured as “OutgoingUsername” and “OutgoingPassword”. If bi-directional authentication is desired, then a different username and password combination can be specified on the target as “Outgoing Username” and “Outgoing Password” and on the initiator as “Incoming Username” and “IncomingPassword”. - Ensure the secure permissions on the /etc/iscsi.conf file and reload the iscsi driver.
suse92:/tmp # ls -l /etc/iscsi.conf
-rw——- 1 root root 17264 Jun 11 16:36 /etc/iscsi.conf
suse92:/tmp # /etc/init.d/iscsi reload
Reload service iSCSI done
suse92:/tmp # tail /var/log/messagesJun 11 16:40:40 suse92 iscsid[5739]: authenticated by target
iSNS, Internet Storage Name Service
In the above sections, we were working with iSCSI target and initiator addresses. We edited the /etc/iscsi.conf file on the initiator and specified the Target’s address. This is strikingly similar to /etc/hosts setup that we use for DNS resolution. For hostname resolution, we use DNS protocol to specify multiple repositories that will automate and manage hostname resolution information. A similar protocol, dubbed iSNS, is being considered by IETF to facilitate automated discovery, management and configuration of FC and iSCSI devices on the TCP/IP network. More information about iSNS can be found in the Resources Section
- iSCSI Protocol http://www.ietf.org/rfc/rfc3720.txt
- Linux-iSCSI Project http://linux-iscsi.sourceforge.net
- Cuddletech iSCSI “Manual” http://www.cuddletech.com/articles/iscsi/index.html
- Bonnie http://www.textuality.com/bonnie/
- Network Appliance http://www.netapp.com
- Cisco iSCSI Drivers http://www.cisco.com/pcgi-bin/tablebuild.pl/sn5420-scsi (Customer Login Required)
- iSNS Overview http://www.san-ip.com/iSCSI/techoverviewart.htm
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
