CentOS7下编译安装SPDK iSCSI Target

操作环境

Centos 7

操作步骤

1.下载SPDK

#git clone https://github.com/spdk/spdk.git

2.安装依赖包

# dnf install -y gcc gcc-c++ CUnit-devel libaio-devel openssl-devel

3.下载安装DPDK

#cd spdk
#wget http://fast.dpdk.org/rel/dpdk-17.02.tar.xz
#tar xf dpdk-17.02.tar.xz
#cd dpdk-17.02 && make install T=x86_64-native-linuxapp-gcc DESTDIR=.
#make DPDK_DIR=./dpdk-17.02/x86_64-native-linuxapp-gcc
#sudo scripts/setup.sh

4.在安装完成后，我们可以在spdk/app/iscsi_tgt/目录下面找到iscsi_tgt可执行文件

5. 编辑iscsi tgt配置文件，可以根据spdk/etc/spdk/iscsi.conf.in文件，进行修改

# iSCSI target configuration file
#
# Please write all parameters using ASCII.
# The parameter must be quoted if it includes whitespace.
#
# Configuration syntax:
# Leading whitespace is ignored.
# Lines starting with '#' are comments.
# Lines ending with '\' are concatenated with the next line.
# Bracketed ([]) names define sections

[Global]
  # Shared Memory Group ID. SPDK applications with the same ID will share memory.
  # Default: <the process PID>
  #SharedMemoryID 0

  # Users can restrict work items to only run on certain cores by
  #  specifying a ReactorMask.  Default is to allow work items to run
  #  on core 0.
  #ReactorMask 0xFFFF

  # Tracepoint group mask for spdk trace buffers
  # Default: 0x0 (all tracepoint groups disabled)
  # Set to 0xFFFFFFFFFFFFFFFF to enable all tracepoint groups.
  #TpointGroupMask 0x0

  # syslog facility
  LogFacility "local7"

[iSCSI]
  # node name (not include optional part)
  # Users can optionally change this to fit their environment.
  NodeBase "iqn.2016-06.io.spdk"

#  AuthFile /usr/local/etc/spdk/auth.conf  //注释掉

  MinConnectionsPerCore 4
  # Power saving related variable, this parameter defines how long an iSCSI
  # connection must be idle before moving it to a state where it will consume
  # less power. This variable is defined in terms of microseconds. We set default
  # value as 5ms.
  MinConnectionIdleInterval 5000

  # Socket I/O timeout sec. (0 is infinite)
  Timeout 30

  # authentication information for discovery session
  DiscoveryAuthMethod Auto

  #MaxSessions 128
  #MaxConnectionsPerSession 2

  # iSCSI initial parameters negotiate with initiators
  # NOTE: incorrect values might crash
  DefaultTime2Wait 2
  DefaultTime2Retain 60

  ImmediateData Yes
  ErrorRecoveryLevel 0

[Rpc]
  # Defines whether to enable configuration via RPC.
  # Default is disabled.  Note that the RPC interface is not
  # authenticated, so users should be careful about enabling
  # RPC in non-trusted environments.
  Enable No
  # Listen address for the RPC service.
  # May be an IP address or an absolute path to a Unix socket.
  Listen 127.0.0.1

# Users must change the PortalGroup section(s) to match the IP addresses
#  for their environment.
# PortalGroup sections define which TCP ports the iSCSI server will use
#  to listen for incoming connections.  These are also used to determine
#  which targets are accessible over each portal group.
[PortalGroup1]
  Portal DA1 10.10.200.61:3260   //设置为iscsi target server ip

# Users must change the InitiatorGroup section(s) to match the IP
#  addresses and initiator configuration in their environment.
# Netmask can be used to specify a single IP address or a range of IP addresses
#  Netmask 192.168.1.20   <== single IP address
#  Netmask 192.168.1.0/24 <== IP range 192.168.1.*
[InitiatorGroup1]
  InitiatorName ALL
  Netmask 10.10.200.0/24         //设置为iscsi initiator 的ip范围

# NVMe configuration options
#[Nvme]
# NVMe Device Whitelist
# Users may specify which NVMe devices to claim by their PCI
# domain, bus, device, and function. The format is dddd:bb:dd.f, which is
# the same format displayed by lspci or in /sys/bus/pci/devices. The second
# argument is a "name" for the device that can be anything. The name
# is referenced later in the Subsystem section.
#
# Alternatively, the user can specify ClaimAllDevices. All
# NVMe devices will be claimed and named Nvme0, Nvme1, etc.
#  BDF 0000:00:00.0 Nvme0
#  BDF 0000:01:00.0 Nvme1

  # The number of attempts per I/O when an I/O fails. Do not include
  # this key to get the default behavior.
#  NvmeRetryCount 4
  # The maximum number of NVMe controllers to claim. Do not include this key to
  # claim all of them.
#  NumControllers 2
  # Registers the application to receive timeout callback and to reset the controller.
  ResetControllerOnTimeout Yes
  # Timeout value.
#  NvmeTimeoutValue 30
  # Set how often the admin queue is polled for asynchronous events.
  # Units in microseconds.
#  AdminPollRate 100000

# Users may change this section to create a different number or size of
#  malloc LUNs.
# If the system has hardware DMA engine, it will use an IOAT
# (i.e. Crystal Beach DMA) channel to do the copy instead of memcpy.
# Of course, users can disable offload even it is available.
#[Malloc]
  # Number of Malloc targets
#  NumberOfLuns 3
  # Malloc targets are 128M
#  LunSizeInMB 128
  # Block size. Default is 512 bytes.
#  BlockSize 4096

# Users may not want to use offload even it is available.
# Users may use the whitelist to initialize specified devices, IDS
#  uses BUS:DEVICE.FUNCTION to identify each Ioat channel.
[Ioat]
  Disable Yes
  Whitelist 00:04.0
  Whitelist 00:04.1

# Users must change this section to match the /dev/sdX devices to be
#  exported as iSCSI LUNs. The devices are accessed using Linux AIO.
[AIO]
  AIO /dev/sdb           //设置硬盘盘符
#  AIO /dev/sdc

# The Split virtual block device slices block devices into multiple smaller bdevs.
#[Split]
  # Syntax:
  #   Split <bdev> <count> [<size_in_megabytes>]

  # Split Malloc1 into two equally-sized portions, Malloc1p0 and Malloc1p1
#  Split Malloc1 2

  # Split Malloc2 into eight 1-megabyte portions, Malloc2p0 ... Malloc2p7,
  # leaving the rest of the device inaccessible
#  Split Malloc2 8 1

# Users should change the TargetNode section(s) below to match the
#  desired iSCSI target node configuration.
# TargetName, Mapping, LUN0 are minimum required
[TargetNode1]
  TargetName disk1
  TargetAlias "Data Disk1"
  Mapping PortalGroup1 InitiatorGroup1
#  AuthMethod Auto
#  AuthGroup AuthGroup1         //注释掉
  # Enable header and data digest
  # UseDigest Header Data
  UseDigest Auto
  # Use the first malloc target
#  LUN0 Malloc0
  # Using the first AIO target
  LUN0 AIO0           //设置LUN映射
  # Using the second storage target
#  LUN2 AIO1
  # Using the third storage target
#  LUN3 AIO2
  QueueDepth 128

#[TargetNode2]
#  TargetName disk2
#  TargetAlias "Data Disk2"
#  Mapping PortalGroup1 InitiatorGroup1
#  AuthMethod Auto
#  AuthGroup AuthGroup1
#  UseDigest Auto
#  LUN0 Nvme0
#  QueueDepth 32

6.启动iscsi tgt

[root@localhost iscsi_tgt]# ./iscsi_tgt 
Starting Intel(R) DPDK initialization ... 
[ DPDK EAL parameters: iscsi -c 1 --file-prefix=spdk2697 --base-virtaddr=0x1000000000 --proc-type=auto ]
EAL: Detected 8 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Occupied cpu core mask is 0x1
Occupied cpu socket mask is 0x1
device 0 HDD UNIT
device 0: LUN0 AIO0
Total cores available: 1
Using net framework default
Reactor started on core 0 on socket 0

7.从iscsi initiator端连接tgt

[root@lnmp-190 nodes]# iscsiadm -m discovery -t sendtargets -p 10.10.200.61
10.10.200.61:3260,1 iqn.2016-06.io.spdk:disk1
[root@lnmp-190 nodes]# iscsiadm -m node -T iqn.2016-06.io.spdk:disk1 -l
Logging in to [iface: default, target: iqn.2016-06.io.spdk:disk1, portal: 10.10.200.61,3261] (multiple)
Login to [iface: default, target: iqn.2016-06.io.spdk:disk1, portal: 10.10.200.61,3261] successful.
[root@lnmp-190 nodes]# fdisk -l

Disk /dev/sda: 53.7 GB, 53687091200 bytes
64 heads, 32 sectors/track, 51200 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d3782

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           2         501      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2             502       51200    51915776   8e  Linux LVM
Partition 2 does not end on cylinder boundary.

Disk /dev/mapper/vg_lnmp190-lv_root: 48.9 GB, 48930750464 bytes
255 heads, 63 sectors/track, 5948 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_lnmp190-lv_swap: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdd: 4783.5 GB, 4783519825920 bytes
255 heads, 63 sectors/track, 581563 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 1048576 bytes
Disk identifier: 0x00000000

查看iscsi 磁盘，以上信息中的sdd为iscsi硬盘

[root@lnmp-190 nodes]# iscsiadm -m session -P 3 | grep "Attached scsi disk" | awk '{print $4}'
sdd