redhat7使用iscsi提供仲裁服务

1、仲裁服务器[服务端]安装scsi-target-utils
perl-Config-General-2.61-1.el7.noarch.rpm
scsi-target-utils-1.0.73-1.x86_64.rpm

2、修改targets.conf 文件[该文件安装完就有了]

vi /etc/tgt/targets.conf（末尾注添加，需要修改lv名称[/dev/vg00/lvocr]，修改客户端[innitiator]的ip，其他可以不用改）

<target iqn.b0422.target:iscsid>
        backing-store /dev/vg00/lvocr
        initiator-address 192.168.13.28
        initiator-address 192.168.13.29
        vendor_id eCloud
        product_id eCloudTech
        scsi_id 36000000x0000001
        scsi_sn b0422
</target>

3、重启tgtd服务
service tgtd restart

4、查看磁盘发送情况
[root@B09-03 tgt]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.b0903.target:iscsid
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: 36000000x0000001
            SCSI SN: b0903
            Size: 733909 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/cciss/c0d1
            Backing store flags: 
    Account information:
    ACL information:
        192.168.0.191
        
        
        
5、initiator端磁盘接收（客户端发现磁盘）
#发现iqn
[root@cdm ~]# iscsiadm -m discovery -t sendtargets -p 192.168.13.45
192.168.0.192:3260,1 iqn.b0903.target:iscsid

6、在客户端接收磁盘(ip是服务端的ip[仲裁机器的ip]，iqn就是上面命令输出的iqn)
[root@cdm ~]# iscsiadm -m node -T iqn.lszyy.target:iscsid -p 192.168.9.40:3260 --login
Logging in to [iface: default, target: iqn.b0903.target:iscsid, portal: 192.168.0.192,3260] (multiple)
Login to [iface: default, target: iqn.b0903.target:iscsid, portal: 192.168.0.192,3260] successful.

7、指定仲裁盘到本地裸设备
7.1、查找刚过来的中裁盘
lsblk
sdr                      65:16   0   1.8T  0 disk  
└─bbyx02_data_d05       253:16   0   1.8T  0 mpath 
sds                      65:32   0   1.8T  0 disk  
└─bbyx02_data_d06       253:17   0   1.8T  0 mpath 
sdt                      65:48   0   1.8T  0 disk  
└─bbyx02_data_d06       253:17   0   1.8T  0 mpath 
sdu                      65:64   0   1.8T  0 disk  
└─bbyx02_data_d07       253:18   0   1.8T  0 mpath 
sdv                      65:80   0   1.8T  0 disk  
└─bbyx02_data_d07       253:18   0   1.8T  0 mpath 
sdw                      65:96   0   1.8T  0 disk  
└─bbyx02_data_d08       253:19   0   1.8T  0 mpath 
sdx                      65:112  0   1.8T  0 disk  
└─bbyx02_data_d08       253:19   0   1.8T  0 mpath 
sdy                      65:128  0   1.8T  0 disk  
└─bbyx02_data_d09       253:10   0   1.8T  0 mpath 
sdz                      65:144  0   1.8T  0 disk  
└─bbyx02_data_d09       253:10   0   1.8T  0 mpath 
sdaa                     65:160  0     1T  0 disk  
└─bbyx02_arch_d01       253:11   0     1T  0 mpath 
sdab                     65:176  0     1T  0 disk  
└─bbyx02_arch_d01       253:11   0     1T  0 mpath 
sdac                     65:192  0    40G  0 disk  
loop0                     7:0    0   3.8G  0 loop  /mnt
或者
fdisk -l
或者
[root@cdm ~]# lsscsi 
[0:0:0:0]    storage HP       P410i            3.52  -        
[0:1:0:0]    disk    HP       LOGICAL VOLUME   3.52  /dev/sda 
[1:0:0:0]    cd/dvd  TEAC     DV-28S-W         C.2D  /dev/sr0 
[3:0:0:0]    storage IET      Controller       0001  -        
[3:0:0:1]    disk    eCloud   eCloudTech       0001  /dev/sdb 
可以看到sdac是刚过来的40g的仲裁盘
ll /dev/disk/by-id/
7.2、找到by-id
lrwxrwxrwx 1 root root 10 Aug  3 16:50 scsi-360000000000000003600000000000001 -> ../../sdac
可以通过by-id找到对应的sdac的盘，然后使用by-id去raw到本地裸设备
7.3、通过by-id映射为本地裸设备
raw /dev/raw/raw2001 /dev/sdv
raw /dev/raw/raw3001 /dev/disk/by-id/scsi-360000000000000003600000000000001

8、dd测试磁盘可用性
[root@cdm ~]# dd if=/dev/zero of=/dev/sdb bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0261143 s, 19.6 MB/s
[root@cdm ~]# dd if=/dev/zero of=/dev/sdc bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0276131 s, 18.5 MB/s
[root@cdm ~]# dd if=/dev/zero of=/dev/sdd bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.026127 s, 19.6 MB/s
[root@cdm ~]# dd if=/dev/zero of=/dev/sde bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0270495 s, 18.9 MB/s
[root@cdm ~]# dd if=/dev/zero of=/dev/sdf bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0268998 s, 19.0 MB/s
[root@cdm ~]# dd if=/dev/zero of=/dev/sdg bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0256557 s, 20.0 MB/s
[root@cdm ~]# dd if=/dev/zero of=/dev/sdh bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0264621 s, 19.3 MB/s

9、fio磁盘性能测试
分别测试磁盘性能在
4k,8k,1M大小下的磁盘性能
9.1、安装
fio-fio-3.0.tar.gz
9.2、创建fio配置文件
[root@bbyxy02 core]# cat fio
[global]
bs=1024k
iodepth=32
direct=1
ioengine=libaio
group_reporting
time_based
runtime=300
numjobs=50

[job1]
rw=read
filename=/dev/mapper/bbyx01_sys_d01
[root@bbyxy02 core]# 
bs取值有4k,8k,1024K
rw取值有read,write,randread,randwrite

9.3、执行测试[前面的fio是命令，后面的是文件名fio]
[root@bbyxy02 core]# cat fio
[global]
bs=1024k
iodepth=32
direct=1
ioengine=libaio
group_reporting
time_based
runtime=300
numjobs=50

[job1]
rw=read
filename=/dev/mapper/bbyx01_sys_d01
[root@bbyxy02 core]# fio fio
job1: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.0
Starting 50 processes
Jobs: 50 (f=50): [R(50)][100.0%][r=1047MiB/s,w=0KiB/s][r=1047,w=0 IOPS][eta 00m:00s]
job1: (groupid=0, jobs=50): err= 0: pid=154183: Tue Aug  3 17:29:56 2021
   read: IOPS=1052, BW=1053MiB/s (1104MB/s)(309GiB/300089msec)
    slat (usec): min=34, max=1416.6k, avg=47478.13, stdev=143692.58
    clat (msec): min=27, max=2407, avg=1468.68, stdev=154.26
     lat (msec): min=75, max=2886, avg=1516.15, stdev=195.51
    clat percentiles (msec):
     |  1.00th=[  961],  5.00th=[ 1418], 10.00th=[ 1435], 20.00th=[ 1435],
     | 30.00th=[ 1452], 40.00th=[ 1452], 50.00th=[ 1452], 60.00th=[ 1452],
     | 70.00th=[ 1469], 80.00th=[ 1469], 90.00th=[ 1485], 95.00th=[ 1921],
     | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1972], 99.95th=[ 1989],
     | 99.99th=[ 2005]
   bw (  KiB/s): min= 2048, max=45146, per=2.00%, avg=21560.68, stdev=4219.83, samples=29868
   iops        : min=    2, max=   44, avg=21.04, stdev= 4.12, samples=29868
  lat (msec)   : 50=0.01%, 100=0.01%, 250=0.05%, 500=0.10%, 750=0.07%
  lat (msec)   : 1000=2.30%, 2000=97.45%, >=2000=0.01%
  cpu          : usr=0.00%, sys=0.12%, ctx=49145, majf=0, minf=411297
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.3%, 32=99.5%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwt: total=315956,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1053MiB/s (1104MB/s), 1053MiB/s-1053MiB/s (1104MB/s-1104MB/s), io=309GiB (331GB), run=300089-300089msec

10、发送文件的时候
[root@bbyxy01 ~]# cat /etc/ecsend.conf
# This is the eccli send configuration file.
# It is used to build the send envionment conveniently.
#
# Note: Use "/dev/disk/by-path/..." as a path, please!
#

# send gids
[send]
fe80:0000:0000:0000:506b:4b03:007d:81d2
fe80:0000:0000:0000:506b:4b03:007d:81a2

# recvive gids
[recv]
fe80:0000:0000:0000:506b:4b03:007d:81b2
fe80:0000:0000:0000:506b:4b03:007d:8182

# ocr device paths
[ocr]
/dev/mapper/vg00-bbyx01_ocr_d02

# sys device paths
[sys]
/dev/disk/by-id/scsi-3600605b00ed90ae0288865090d993b23

# arch device paths
[arch]
/dev/disk/by-id/scsi-3600605b00ed90ae0288864f70c7e748e-part1

# data device paths
[data]
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part1
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part2
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part3
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part4
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part5
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part6
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part7
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part8
/dev/disk/by-id/scsi-3600605b00ed90ae0288962cc05b48581-part9

该配置文件/etc/ecsend.conf只会最开始用一次，后面就不会在用了
后面用的配置文件是/etc/scst.conf

[root@bbyxy02 core]# cat /etc/scst.conf
# Automatically generated by SCST Configurator v3.3.0-pre1.

HANDLER vdisk_blockio {
    DEVICE bbyx02_arch_d01 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff024acc71d4f256b17-part1
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_arch_d01
        write_through 1
    }

    DEVICE bbyx02_data_d01 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part1
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d01
        write_through 1
    }

    DEVICE bbyx02_data_d02 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part2
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d02
        write_through 1
    }

    DEVICE bbyx02_data_d03 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part3
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d03
        write_through 1
    }

    DEVICE bbyx02_data_d04 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part4
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d04
        write_through 1
    }

    DEVICE bbyx02_data_d05 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part5
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d05
        write_through 1
    }

    DEVICE bbyx02_data_d06 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part6
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d06
        write_through 1
    }

    DEVICE bbyx02_data_d07 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part7
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d07
        write_through 1
    }

    DEVICE bbyx02_data_d08 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part8
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d08
        write_through 1
    }

    DEVICE bbyx02_data_d09 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff02889610805ddea82-part9
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_data_d09
        write_through 1
    }

    DEVICE bbyx02_ocr_d01 {
        filename /dev/mapper/vg00-bbyx02_ocr_d02
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_ocr_d01
        write_through 1
    }

    DEVICE bbyx02_sys_d01 {
        filename /dev/disk/by-id/scsi-3600605b00ed90ff028885d7b0c1ddeb3
        rotational 0
        t10_vend_id eCloud
        usn bbyx02_sys_d01
        write_through 1
    }
}

TARGET_DRIVER copy_manager {
    TARGET copy_manager_tgt {
        LUN 0 bbyx02_ocr_d01
        LUN 1 bbyx02_sys_d01
        LUN 2 bbyx02_data_d01
        LUN 3 bbyx02_data_d02
        LUN 4 bbyx02_data_d03
        LUN 5 bbyx02_data_d04
        LUN 6 bbyx02_data_d05
        LUN 7 bbyx02_data_d06
        LUN 8 bbyx02_data_d07
        LUN 9 bbyx02_data_d08
        LUN 10 bbyx02_data_d09
        LUN 11 bbyx02_arch_d01
    }
}

TARGET_DRIVER ib_srpt {
    TARGET ac1f:6b7e:2dae:0000:0000:0000:0000:0000 {
        comp_v_mask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000ff,ffffffff
        enabled 0
    }

    TARGET ac1f:6b7e:2daf:0000:0000:0000:0000:0000 {
        comp_v_mask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000ff,ffffffff
        enabled 0
    }

    TARGET fe80:0000:0000:0000:506b:4b03:007d:8181 {
        comp_v_mask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,3fffffff,ffffffff
        enabled 0
    }

    TARGET fe80:0000:0000:0000:506b:4b03:007d:8182 {
        comp_v_mask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,3fffffff,ffffffff
        enabled 1
        rel_tgt_id 2

        GROUP bbyx02_group {
            LUN 0 bbyx02_ocr_d01
            LUN 1 bbyx02_sys_d01
            LUN 2 bbyx02_data_d01
            LUN 3 bbyx02_data_d02
            LUN 4 bbyx02_data_d03
            LUN 5 bbyx02_data_d04
            LUN 6 bbyx02_data_d05
            LUN 7 bbyx02_data_d06
            LUN 8 bbyx02_data_d07
            LUN 9 bbyx02_data_d08
            LUN 10 bbyx02_data_d09
            LUN 11 bbyx02_arch_d01

            INITIATOR fe80:0000:0000:0000:506b:4b03:007d:81a2

            INITIATOR fe80:0000:0000:0000:506b:4b03:007d:81d2
        }
    }

    TARGET fe80:0000:0000:0000:506b:4b03:007d:81b1 {
        comp_v_mask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,3fffffff,ffffffff
        enabled 0
    }

    TARGET fe80:0000:0000:0000:506b:4b03:007d:81b2 {
        comp_v_mask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,3fffffff,ffffffff
        enabled 1
        rel_tgt_id 1

        GROUP bbyx02_group {
            LUN 0 bbyx02_ocr_d01
            LUN 1 bbyx02_sys_d01
            LUN 2 bbyx02_data_d01
            LUN 3 bbyx02_data_d02
            LUN 4 bbyx02_data_d03
            LUN 5 bbyx02_data_d04
            LUN 6 bbyx02_data_d05
            LUN 7 bbyx02_data_d06
            LUN 8 bbyx02_data_d07
            LUN 9 bbyx02_data_d08
            LUN 10 bbyx02_data_d09
            LUN 11 bbyx02_arch_d01

            INITIATOR fe80:0000:0000:0000:506b:4b03:007d:81a2

            INITIATOR fe80:0000:0000:0000:506b:4b03:007d:81d2
        }
    }
}

[root@bbyxy02 core]# 

11、/etc/rc.local配置文件