SCSI PRs命令研究总结3 - Linux中的SCSI相关实现

最新推荐文章于 2023-02-11 16:37:26 发布

weixin_33901843

最新推荐文章于 2023-02-11 16:37:26 发布

阅读量1.3k

点赞数

文章标签：操作系统运维 python

原文链接：https://my.oschina.net/LastRitter/blog/1541364

版权

2019独角兽企业重金招聘Python工程师标准>>>

Linux下的SCSI驱动的实现，驱动与用户层交互接口规范的制定，以及用户态的配置管理工具主要由“The Linux SCSI Generic (sg) Driver”项目负责。

The Linux sg driver is a upper level SCSI subsystem device driver that is used primarily to handle devices _not_ covered by the 
other upper level drivers: sd (disks), st (tapes) and sr (CDROMs and DVDs). The sg driver is used for enclosure management, cd 
writers, applications that read cd audio digitally and scanners. Sg can also be used for less usual tasks performed on disks, tapes
and cdroms. Sg is a character device driver which, in some contexts, gives it advantages over block device drivers such as sd and
sr. The interface of sg is at the level of SCSI command requests and their associated responses.

The term SCSI has several meaning depending on the context. This leads to confusion. One practical way of defining it today is 
everything that the T10 INCITS committee controls, see www.t10.org . Probably the most succinct overview is this standards 
architecture page . For practical purposes a "SCSI device" in Linux is any device that uses the Linux SCSI subsystem and this often 
includes SATA disks.

From about Linux kernel 2.6.24, there is an alternate SCSI pass-through driver called "bsg" (block SCSI generic driver). The bsg 
driver has device names of the form /dev/bsg/0:1:2:3 and supports the SG_IO ioctl with the sg version 3 interface. The bsg driver 
also supports the sg version 4 interface which at this time the sg driver does not. Amongst other improvements the sg version 4 
interface supports SCSI bidirectional commands. All recent "sg" user space packages (i.e. sg3_utils, sdparm, ddpt and smp_utils) 
work equally well on both sg and bsg device names.

（来源：http://sg.danny.cz/sg/index.html。）

Linux SCSI设备的管理接口

用户层与驱动层通过ioctl的SG_IO命令发送SCSI命令和返回SCSI命令执行结果，从Linux 2.6内核开始，所有的块设备均支持ioctl的SG_IO命令。

The SG_IO ioctl permits user applications to send SCSI commands to a device. In the linux 2.4 series this ioctl was only available 
via the SCSI generic (sg) driver. In the linux 2.6 series the SG_IO ioctl is additionally available for block devices and SCSI tape
(st) devices.  So there are multiple implementations of this ioctl within the kernel with slightly different characteristics and 
describing these is the purpose of this document.

（来源：http://sg.danny.cz/sg/sg_io.html。）

SG_IO命令的数据结构

typedef struct sg_iovec { /* parallels "struct iovec" in readv() system call */ 
    void * iov_base;            /* start address */ 
    size_t iov_len;             /* length in bytes */ 
} sg_iovec_t; /* the scatter-gather list is an array of objects of this type */

typedef struct sg_io_hdr 
{ 
    int interface_id;           /* [i] 'S' for SCSI generic (required) */ 
    int dxfer_direction;        /* [i] data transfer direction  */ 
    unsigned char cmd_len;      /* [i] SCSI command length ( <= 16 bytes) */ 
    unsigned char mx_sb_len;    /* [i] max length to write to sbp */ 
    unsigned short iovec_count; /* [i] 0 implies no scatter gather */ 
    unsigned int dxfer_len;     /* [i] byte count of data transfer */ 
    void * dxferp;              /* [i] [*io] points to data transfer memory or 
                                             scatter gather list */ 
    unsigned char * cmdp;       /* [i] [*i] points to SCSI command to perform */ 
    unsigned char * sbp;        /* [i] [*o] points to sense_buffer memory */ 
    unsigned int timeout;       /* [i] MAX_UINT->no timeout (unit: millisec) */ 
    unsigned int flags;         /* [i] 0 -> default, see SG_FLAG... */ 
    int pack_id;                /* [i->o] unused internally (normally) */ 
    void * usr_ptr;             /* [i->o] unused internally */ 
    unsigned char status;       /* [o] scsi status */ 
    unsigned char masked_status;/* [o] shifted, masked scsi status */ 
    unsigned char msg_status;   /* [o] messaging level data (optional) */ 
    unsigned char sb_len_wr;    /* [o] byte count actually written to sbp */ 
    unsigned short host_status; /* [o] errors from host adapter */ 
    unsigned short driver_status;/* [o] errors from software driver */ 
    int resid;                  /* [o] dxfer_len - actual_transferred */ 
    unsigned int duration;      /* [o] time taken (unit: millisec) */ 
    unsigned int info;          /* [o] auxiliary information */ 
} sg_io_hdr_t;  /* around 64 bytes long (on i386) */

/* Use negative values to flag difference from original sg_header structure */ 
#define SG_DXFER_NONE -1        /* e.g. a SCSI Test Unit Ready command */ 
#define SG_DXFER_TO_DEV -2      /* e.g. a SCSI WRITE command */ 
#define SG_DXFER_FROM_DEV -3    /* e.g. a SCSI READ command */ 
#define SG_DXFER_TO_FROM_DEV -4 /* treated like SG_DXFER_FROM_DEV with the 
                                   additional property than during indirect 
                                   IO user buffer is copied into the kernel 
                                   buffers before the transfer */ 
#define SG_DXFER_UNKNOWN -5     /* Unknown data direction */

/* following flag values can be "or"-ed together */ 
#define SG_FLAG_DIRECT_IO 1     /* default is indirect IO */ 
#define SG_FLAG_LUN_INHIBIT 2   /* default is to put device's lun into */ 
                                /* the 2nd byte of SCSI command */ 
#define SG_FLAG_MMAP_IO 4       /* selects memory mapped IO. Introduced in 
                                   version 3.1.22 . May not be present in 
                                   GNU library headders for some time */ 
#define SG_FLAG_NO_DXFER 0x10000 /* no transfer of kernel buffers to/from */ 
                                /* user space (debug indirect IO) */ 
 

/* following 'info' values are "or"-ed together */ 
#define SG_INFO_OK_MASK 0x1 
#define SG_INFO_OK 0x0          /* no sense, host nor driver "noise" */ 
#define SG_INFO_CHECK 0x1       /* something abnormal happened */

#define SG_INFO_DIRECT_IO_MASK 0x6 
#define SG_INFO_INDIRECT_IO 0x0 /* data xfer via kernel buffers (or no xfer) */ 
#define SG_INFO_DIRECT_IO 0x2 
#define SG_INFO_MIXED_IO 0x4    /* part direct, part indirect IO */

（来源：http://sg.danny.cz/sg/s_packet.html。）

使用“Direct IO”的方式执行SCSI命令

读写的数据保存在用户分配的buffer中，需要一次拷贝：

/* 0 -> successful, SG_LIB_SYNTAX_ERROR -> unable to build cdb,
   SG_LIB_CAT_UNIT_ATTENTION -> try again,
   SG_LIB_CAT_MEDIUM_HARD_WITH_INFO -> 'io_addrp' written to,
   SG_LIB_CAT_MEDIUM_HARD -> no info field,
   SG_LIB_CAT_NOT_READY, SG_LIB_CAT_ABORTED_COMMAND,
   -2 -> ENOMEM
   -1 other errors */
static int
sg_read_low(int sg_fd, unsigned char * buff, int blocks, int64_t from_block,
            int bs, const struct flags_t * ifp, int * diop,
            uint64_t * io_addrp)
{
    unsigned char rdCmd[MAX_SCSI_CDBSZ];
    unsigned char senseBuff[SENSE_BUFF_LEN];
    const unsigned char * sbp;
    struct sg_io_hdr io_hdr;
    int res, k, info_valid, slen;

    if (sg_build_scsi_cdb(rdCmd, ifp->cdbsz, blocks, from_block, 0,
                          ifp->fua, ifp->dpo)) {
        fprintf(stderr, ME "bad rd cdb build, from_block=%" PRId64
                ", blocks=%d\n", from_block, blocks);
        return SG_LIB_SYNTAX_ERROR;
    }

    memset(&io_hdr, 0, sizeof(struct sg_io_hdr));
    io_hdr.interface_id = 'S';
    io_hdr.cmd_len = ifp->cdbsz;
    io_hdr.cmdp = rdCmd;
    io_hdr.dxfer_direction = SG_DXFER_FROM_DEV;
    io_hdr.dxfer_len = bs * blocks;
    io_hdr.dxferp = buff;
    io_hdr.mx_sb_len = SENSE_BUFF_LEN;
    io_hdr.sbp = senseBuff;
    io_hdr.timeout = DEF_TIMEOUT;
    io_hdr.pack_id = (int)from_block;
    if (diop && *diop)
        io_hdr.flags |= SG_FLAG_DIRECT_IO;

    if (verbose > 2) {
        fprintf(stderr, "    read cdb: ");
        for (k = 0; k < ifp->cdbsz; ++k)
            fprintf(stderr, "%02x ", rdCmd[k]);
        fprintf(stderr, "\n");
    }
    while (((res = ioctl(sg_fd, SG_IO, &io_hdr)) < 0) && (EINTR == errno))
        ;
    if (res < 0) {
        if (ENOMEM == errno)
            return -2;
        perror("reading (SG_IO) on sg device, error");
        return -1;
    }
    if (verbose > 2)
        fprintf(stderr, "      duration=%u ms\n", io_hdr.duration);
    res = sg_err_category3(&io_hdr);
    sbp = io_hdr.sbp;
    slen = io_hdr.sb_len_wr;
    switch (res) {
    case SG_LIB_CAT_CLEAN:
        break;
    case SG_LIB_CAT_RECOVERED:
        ++recovered_errs;
        info_valid = sg_get_sense_info_fld(sbp, slen, io_addrp);
        if (info_valid) {
            fprintf(stderr, "    lba of last recovered error in this "
                    "READ=0x%" PRIx64 "\n", *io_addrp);
            if (verbose > 1)
                sg_chk_n_print3("reading", &io_hdr, 1);
        } else {
            fprintf(stderr, "Recovered error: [no info] reading from "
                    "block=0x%" PRIx64 ", num=%d\n", from_block, blocks);
            sg_chk_n_print3("reading", &io_hdr, verbose > 1);
        }
        break;
    case SG_LIB_CAT_ABORTED_COMMAND:
    case SG_LIB_CAT_UNIT_ATTENTION:
        sg_chk_n_print3("reading", &io_hdr, verbose > 1);
        return res;
    case SG_LIB_CAT_MEDIUM_HARD:
        if (verbose > 1)
            sg_chk_n_print3("reading", &io_hdr, verbose > 1);
        ++unrecovered_errs;
        info_valid = sg_get_sense_info_fld(sbp, slen, io_addrp);
        /* MMC devices don't necessarily set VALID bit */
        if ((info_valid) || ((5 == ifp->pdt) && (*io_addrp > 0)))
            return SG_LIB_CAT_MEDIUM_HARD_WITH_INFO;
        else {
            fprintf(stderr, "Medium, hardware or blank check error but "
                    "no lba of failure in sense\n");
            return res;
        }
        break;
    case SG_LIB_CAT_NOT_READY:
        ++unrecovered_errs;
        if (verbose > 0)
            sg_chk_n_print3("reading", &io_hdr, verbose > 1);
        return res;
    case SG_LIB_CAT_ILLEGAL_REQ:
        if (5 == ifp->pdt) {    /* MMC READs can go down this path */
            struct sg_scsi_sense_hdr ssh;
            int ili;

            if (verbose > 1)
                sg_chk_n_print3("reading", &io_hdr, verbose > 1);
            if (sg_scsi_normalize_sense(sbp, slen, &ssh) &&
                (0x64 == ssh.asc) && (0x0 == ssh.ascq)) {
                if (sg_get_sense_filemark_eom_ili(sbp, slen, NULL, NULL,
                                                  &ili) && ili) {
                    info_valid = sg_get_sense_info_fld(sbp, slen, io_addrp);
                    if (*io_addrp > 0) {
                        ++unrecovered_errs;
                        return SG_LIB_CAT_MEDIUM_HARD_WITH_INFO;
                    } else
                        fprintf(stderr, "MMC READ gave 'illegal mode for "
                                "this track' and ILI but no LBA of failure\n");
                }
                ++unrecovered_errs;
                return SG_LIB_CAT_MEDIUM_HARD;
            }
        }
        /* drop through */
    default:
        ++unrecovered_errs;
        if (verbose > 0)
            sg_chk_n_print3("reading", &io_hdr, verbose > 1);
        return res;
    }
    if (diop && *diop &&
        ((io_hdr.info & SG_INFO_DIRECT_IO_MASK) != SG_INFO_DIRECT_IO))
        *diop = 0;      /* flag that dio not done (completely) */
    sum_of_resids += io_hdr.resid;
    return 0;
}

（来源：src/sg_dd.c）

缓冲应该按照页面对齐：

    psz = getpagesize(); 
    if (NULL == (alloc_bp = malloc(sz + psz)))  /* 'sz' bytes required */ 
        exit(1);  /* out of memory */ 
    buffp = (unsigned char *) 
            (((unsigned long)alloc_bp + psz - 1) & (~(psz - 1)));

（来源：http://sg.danny.cz/sg/s_packet.html）

使用“MMap”的方式执行SCSI命令

读写的数据保存在驱动分配的一块内存中，不需要额外的拷贝数据：

/* 0 -> successful, SG_LIB_CAT_UNIT_ATTENTION, SG_LIB_SYNTAX_ERROR,
 * SG_LIB_CAT_NOT_READY, SG_LIB_CAT_MEDIUM_HARD, SG_LIB_CAT_ILLEGAL_REQ,
 * SG_LIB_CAT_ABORTED_COMMAND, -2 -> recoverable (ENOMEM),
 * -1 -> unrecoverable error */
static int
sg_read(int sg_fd, unsigned char * buff, int blocks, int64_t from_block,
        int bs, int cdbsz, int fua, int dpo, int do_mmap)
{
    unsigned char rdCmd[MAX_SCSI_CDBSZ];
    unsigned char senseBuff[SENSE_BUFF_LEN];
    struct sg_io_hdr io_hdr;
    int k, res;

    if (sg_build_scsi_cdb(rdCmd, cdbsz, blocks, from_block, 0, fua, dpo)) {
        fprintf(stderr, ME "bad rd cdb build, from_block=%" PRId64
                ", blocks=%d\n", from_block, blocks);
        return SG_LIB_SYNTAX_ERROR;
    }
    memset(&io_hdr, 0, sizeof(struct sg_io_hdr));
    io_hdr.interface_id = 'S';
    io_hdr.cmd_len = cdbsz;
    io_hdr.cmdp = rdCmd;
    io_hdr.dxfer_direction = SG_DXFER_FROM_DEV;
    io_hdr.dxfer_len = bs * blocks;
    if (! do_mmap)
        io_hdr.dxferp = buff;
    io_hdr.mx_sb_len = SENSE_BUFF_LEN;
    io_hdr.sbp = senseBuff;
    io_hdr.timeout = DEF_TIMEOUT;
    io_hdr.pack_id = (int)from_block;
    if (do_mmap)
        io_hdr.flags |= SG_FLAG_MMAP_IO;
    if (verbose > 2) {
        fprintf(stderr, "    read cdb: ");
        for (k = 0; k < cdbsz; ++k)
            fprintf(stderr, "%02x ", rdCmd[k]);
        fprintf(stderr, "\n");
    }

#if 1
    while (((res = ioctl(sg_fd, SG_IO, &io_hdr)) < 0) && (EINTR == errno))
        sleep(1);
    if (res < 0) {
        perror(ME "SG_IO error (sg_read)");
        return -1;
    }
#else
    while (((res = write(sg_fd, &io_hdr, sizeof(io_hdr))) < 0) &&
           (EINTR == errno))
        ;
    if (res < 0) {
        if (ENOMEM == errno)
            return -2;
        perror("reading (wr) on sg device, error");
        return -1;
    }

    while (((res = read(sg_fd, &io_hdr, sizeof(io_hdr))) < 0) &&
           (EINTR == errno))
        ;
    if (res < 0) {
        perror("reading (rd) on sg device, error");
        return -1;
    }
#endif
    if (verbose > 2)
        fprintf(stderr, "      duration=%u ms\n", io_hdr.duration);
    res =  sg_err_category3(&io_hdr);
    switch (res) {
    case SG_LIB_CAT_CLEAN:
        break;
    case SG_LIB_CAT_RECOVERED:
        sg_chk_n_print3("Reading, continuing", &io_hdr, verbose > 1);
        break;
    case SG_LIB_CAT_NOT_READY:
    case SG_LIB_CAT_MEDIUM_HARD:
        return res;
    case SG_LIB_CAT_ABORTED_COMMAND:
    case SG_LIB_CAT_UNIT_ATTENTION:
    case SG_LIB_CAT_ILLEGAL_REQ:
    default:
        sg_chk_n_print3("reading", &io_hdr, verbose > 1);
        return res;
    }
    sum_of_resids += io_hdr.resid;
#ifdef SG_DEBUG
    fprintf(stderr, "duration=%u ms\n", io_hdr.duration);
#endif
    return 0;
}

...

int
main(int argc, char * argv[])
{
...
            wrkMmap = (unsigned char *)mmap(NULL, in_res_sz,
                                 PROT_READ | PROT_WRITE, MAP_SHARED, infd, 0);
            if (MAP_FAILED == wrkMmap) {
                snprintf(ebuff, EBUFF_SZ,
                         ME "error using mmap() on file: %s", inf);
                perror(ebuff);
                return SG_LIB_FILE_ERROR;
...
    if (wrkMmap) {
        wrkPos = wrkMmap;
#ifdef SG_WANT_SHARED_MMAP_IO
        if (! (mmap_shareable && out_flags.smmap &&  (FT_SG == out_type)))
            mmap_shareable = 0;
#endif
    } else {
...
        if (FT_SG == in_type) {
            ret = sg_read(infd, wrkPos, blocks, skip, blk_sz, scsi_cdbsz_in,
                          in_flags.fua, in_flags.dpo, 1);
            if ((SG_LIB_CAT_UNIT_ATTENTION == ret) ||
                (SG_LIB_CAT_ABORTED_COMMAND == ret)) {
                fprintf(stderr, "Unit attention or aborted command, "
                        "continuing (r)\n");
                ret = sg_read(infd, wrkPos, blocks, skip, blk_sz,
                              scsi_cdbsz_in, in_flags.fua, in_flags.dpo, 1);
            }
            if (0 != ret) {
                fprintf(stderr, "sg_read failed, skip=%" PRId64 "\n", skip);
                break;
            }
            else
                in_full += blocks;
...

（来源：src/sgm_dd.c）

可以使用SG_SET_RESERVED_SIZE ioctl命令来设置内核buffer的大小，这个buffer是一个fd对应一个。

SG工具包（sg3_utils）

概述

sg3_utils软件包支持50多种SCSI命令，支持所有SPC、SBC和MMC标准中的命令，基本格式为：

UTILITY_NAME [OPTIONS] DEVICE

其中的“DEVICE”可以是“/dev/sda”, “/dev/scd0”, “/dev/st0”或者/dev/hdd，在2.6.28之后的版本中还支持“/dev/bsg/3:0:0:0”这类设备名。

每个命令都有详细的man手册，此处不再详述。

一些简单的例子

(“/dev/sde”和“/dev/sdf”是连接至一个使用“Linux tgt”软件包搭建的“iSCSI Target”服务上的一个LUN的两条路径)

查询设备的基本信息：

$ sg_inq /dev/sde
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=1  Resp_data_format=2
  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  [BQue=0]
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  [SPI: Clocking=0x0  QAS=0  IUS=0]
    length=66 (0x42)   Peripheral device type: disk
 Vendor identification: IET     
 Product identification: VIRTUAL-DISK
 Product revision level: 0001
 Unit serial number:                               beaf21

可以看到该设备采用SPC-3标准，说明这个设备支持新的PRs命令，且废弃了旧的“RESERVE”和“RELEASE”命令。

查询设备的基本信息的同时，查看可解释的描述信息：

$ sg_inq -d /dev/sde
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x05  [SPC-3]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=1  Resp_data_format=2
  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  [BQue=0]
  EncServ=0  MultiP=0  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  [SPI: Clocking=0x0  QAS=0  IUS=0]
    length=66 (0x42)   Peripheral device type: disk
 Vendor identification: IET     
 Product identification: VIRTUAL-DISK
 Product revision level: 0001
 Unit serial number:                               beaf21

  Version descriptors:
    SBC-3 (no version claimed)
    iSCSI (no version claimed)
    SPC-3 (no version claimed)

查看设备的VPD（Vital Product Data，重要产品数据）数据：

$ sg_inq -e /dev/sde
VPD INQUIRY, page code=0x00:
   [PQual=0  Peripheral device type: disk]
   Supported VPD pages:
     0x0	Supported VPD pages
     0x80	Unit serial number
     0x83	Device identification
     0xb0	Block limits (sbc2)
     0xb2	Logical block provisioning (sbc3)

或者：

$ sg_vpd /dev/sde
Supported VPD pages VPD page:
  Supported VPD pages [sv]
  Unit serial number [sn]
  Device identification [di]
  Block limits (SBC) [bl]
  Logical block provisioning (SBC) [lbpv]

可以看到该设备含有5个参数页。

查看lbpv参数页：

$ sg_vpd --page=lbp /dev/sde
abbreviation doesn't match a VPD page
available VPD pages:
  ai         0x89      ATA information (SAT)
  aod        0x82      ASCII implemented operating definition (obsolete)
  adsn       0xb3      Automation device serial number (SSC)
  bl         0xb0      Block limits (SBC)
  bdc        0xb1      Block device characteristics (SBC)
  cfa        0x8c      CFA profile information
  dc         0x8b      Device constituents
  di         0x83      Device identification
  di_asis    0x83      Like 'di' but designators ordered as found
  di_lu      0x83      Device identification, lu only
  di_port    0x83      Device identification, target port only
  di_target  0x83      Device identification, target device only
  dtde       0xb4      Data transfer device element address (SSC)
  ei         0x86      Extended inquiry data
  iod        0x81      Implemented operating definition (obsolete)
  lbpv       0xb2      Logical block provisioning (SBC)
  mas        0xb1      Manufacturer assigned serial number (SSC)
  masa       0xb1      Manufacturer assigned serial number (ADC)
  mna        0x85      Management network addresses
  mpp        0x87      Mode page policy
  oi         0xb0      OSD information
  pc         0x8a      Power condition
  psm        0x8d      Power consumption
  pslu       0x90      Protocol-specific logical unit information
  pspo       0x91      Protocol-specific port information
  ref        0xb3      Referrals (SBC)
  sad        0xb0      Sequential access device capabilities (SSC)
  sii        0x84      Software interface identification
  sinq       -1        Standard inquiry response
  sn         0x80      Unit serial number
  sp         0x88      SCSI ports
  st         0xb1      Security token (OSD)
  sv         0x00      Supported VPD pages
  tas        0xb2      TapeAlert supported flags (SSC)
  tpc        0x8f      Third party copy

Vendor specific VPD pages:
  datc       0xc1,0      Date code (Seagate)
  devb       0xc3,0      Device behavior (Seagate)
  edid       0xc8,0      Extended device identification (RDAC)
  prm4       0xc3,1      Feature Parameters (RDAC)
  firm       0xc0,0      Firmware numbers (Seagate)
  fwr4       0xc1,1      Firmware version (RDAC)
  hp3par     0xc0,2      Volume information (HP/3PAR)
  hwr4       0xc0,3      Hardware version (RDAC)
  jump       0xc2,0      Jump setting (Seagate)
  rvsi       0xca,0      Replicated volume source identifier (RDAC)
  said       0xd0,0      Storage array world wide name (RDAC)
  subs       0xc4,0      Subsystem identifier (RDAC)
  swr4       0xc2,1      Software version (RDAC)
  upr        0xc0,1      Unit path report (EMC)
  vac1       0xc9,0      Volume access control (RDAC)

$ sg_vpd --page=lbpv /dev/sde
Logical block provisioning VPD page (SBC):
  Unmap command supported (LBPU): 0
  Write same (16) with unmap bit supported (LBWS): 0
  Write same (10) with unmap bit supported (LBWS10): 0
  Logical block provisioning read zeros (LBPRZ): 0
  Anchored LBAs supported (ANC_SUP): 0
  Threshold exponent: 0
  Descriptor present (DP): 0
  Provisioning type: 0

当输入错误的参数页名称时，命令会打印所有可以使用的参数页名称，当然，并不是所有设备都支持全部这些参数页。

查看当前系统上有哪些SCSI设备及其映射关系：

$ cat /proc/scsi/scsi
Attached devices:
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 01 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 02 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi2 Channel: 00 Id: 03 Lun: 00
  Vendor: VMware,  Model: VMware Virtual S Rev: 1.0 
  Type:   Direct-Access                    ANSI  SCSI revision: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: NECVMWar Model: VMware IDE CDR10 Rev: 1.00
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi8 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 01
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi8 Channel: 00 Id: 00 Lun: 01
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi6 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi5 Channel: 00 Id: 00 Lun: 01
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi6 Channel: 00 Id: 00 Lun: 01
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi9 Channel: 00 Id: 00 Lun: 00
  Vendor: IET      Model: Controller       Rev: 0001
  Type:   RAID                             ANSI  SCSI revision: 05
Host: scsi9 Channel: 00 Id: 00 Lun: 01
  Vendor: IET      Model: VIRTUAL-DISK     Rev: 0001
  Type:   Direct-Access                    ANSI  SCSI revision: 05

或者：

$ sg_scan
/dev/sg0: scsi2 channel=0 id=0 lun=0
/dev/sg1: scsi2 channel=0 id=1 lun=0
/dev/sg2: scsi2 channel=0 id=2 lun=0
/dev/sg3: scsi2 channel=0 id=3 lun=0
/dev/sg4: scsi1 channel=0 id=0 lun=0 [em]
/dev/sg5: scsi3 channel=0 id=0 lun=0
/dev/sg6: scsi3 channel=0 id=0 lun=1
/dev/sg7: scsi4 channel=0 id=0 lun=0
/dev/sg8: scsi8 channel=0 id=0 lun=0
/dev/sg9: scsi4 channel=0 id=0 lun=1
/dev/sg10: scsi9 channel=0 id=0 lun=0
/dev/sg11: scsi8 channel=0 id=0 lun=1
/dev/sg12: scsi5 channel=0 id=0 lun=0
/dev/sg13: scsi9 channel=0 id=0 lun=1
/dev/sg14: scsi6 channel=0 id=0 lun=0
/dev/sg15: scsi5 channel=0 id=0 lun=1
/dev/sg16: scsi6 channel=0 id=0 lun=1

或者：

$ sg_scan -i
/dev/sg0: scsi2 channel=0 id=0 lun=0
    VMware,   VMware Virtual S  1.0  [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg1: scsi2 channel=0 id=1 lun=0
    VMware,   VMware Virtual S  1.0  [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg2: scsi2 channel=0 id=2 lun=0
    VMware,   VMware Virtual S  1.0  [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg3: scsi2 channel=0 id=3 lun=0
    VMware,   VMware Virtual S  1.0  [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg4: scsi1 channel=0 id=0 lun=0 [em]
    NECVMWar  VMware IDE CDR10  1.00 [rmb=1 cmdq=0 pqual=0 pdev=0x5] 
/dev/sg5: scsi3 channel=0 id=0 lun=0
    IET       Controller  0001 [rmb=0 cmdq=1 pqual=0 pdev=0xc] 
/dev/sg6: scsi3 channel=0 id=0 lun=1
    IET       VIRTUAL-DISK  0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg7: scsi4 channel=0 id=0 lun=0
    IET       Controller  0001 [rmb=0 cmdq=1 pqual=0 pdev=0xc] 
/dev/sg8: scsi8 channel=0 id=0 lun=0
    IET       Controller  0001 [rmb=0 cmdq=1 pqual=0 pdev=0xc] 
/dev/sg9: scsi4 channel=0 id=0 lun=1
    IET       VIRTUAL-DISK  0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg10: scsi9 channel=0 id=0 lun=0
    IET       Controller  0001 [rmb=0 cmdq=1 pqual=0 pdev=0xc] 
/dev/sg11: scsi8 channel=0 id=0 lun=1
    IET       VIRTUAL-DISK  0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg12: scsi5 channel=0 id=0 lun=0
    IET       Controller  0001 [rmb=0 cmdq=1 pqual=0 pdev=0xc] 
/dev/sg13: scsi9 channel=0 id=0 lun=1
    IET       VIRTUAL-DISK  0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg14: scsi6 channel=0 id=0 lun=0
    IET       Controller  0001 [rmb=0 cmdq=1 pqual=0 pdev=0xc] 
/dev/sg15: scsi5 channel=0 id=0 lun=1
    IET       VIRTUAL-DISK  0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0] 
/dev/sg16: scsi6 channel=0 id=0 lun=1
    IET       VIRTUAL-DISK  0001 [rmb=0 cmdq=1 pqual=0 pdev=0x0]

或者：

$ sg_map
/dev/sg0  /dev/sda
/dev/sg1  /dev/sdb
/dev/sg2  /dev/sdc
/dev/sg3  /dev/sdd
/dev/sg4  /dev/sr0
/dev/sg5
/dev/sg6  /dev/sde
/dev/sg7
/dev/sg8
/dev/sg9  /dev/sdf
/dev/sg10
/dev/sg11  /dev/sdg
/dev/sg12
/dev/sg13  /dev/sdh
/dev/sg14
/dev/sg15  /dev/sdi
/dev/sg16  /dev/sdj

查看sg_persist命令帮助：

$ sg_persist -h
Usage: sg_persist [OPTIONS] [DEVICE]
  where OPTIONS include:
    --alloc-length=LEN|-l LEN    allocation length hex value (used with
                                 PR In only) (default: 8192 (2000 in hex))
    --clear|-C                 PR Out: Clear
    --device=DEVICE|-d DEVICE    query or change DEVICE
    --help|-h                  output this usage message
    --hex|-H                   output response in hex
    --in|-i                    request PR In command (default)
    --no-inquiry|-n            skip INQUIRY (default: do INQUIRY)
    --out|-o                   request PR Out command
    --param-alltgpt|-Y         PR Out parameter 'ALL_TG_PT'
    --param-aptpl|-Z           PR Out parameter 'APTPL'
    --param-rk=RK|-K RK        PR Out parameter reservation key
                               (RK is in hex)
    --param-sark=SARK|-S SARK    PR Out parameter service action
                               reservation key (SARK is in hex)
    --preempt|-P               PR Out: Preempt
    --preempt-abort|-A         PR Out: Preempt and Abort
    --prout-type=TYPE|-T TYPE    PR Out command type
    --read-full-status|-s      PR In: Read Full Status
    --read-keys|-k             PR In: Read Keys
    --read-reservation|-r      PR In: Read Reservation
    --read-status|-s           PR In: Read Full Status
    --register|-G              PR Out: Register
    --register-ignore|-I       PR Out: Register and Ignore
    --register-move|-M         PR Out: Register and Move
    --relative-target-port=RTPI|-Q RTPI    relative target port identifier
                               for '--register-move'
    --release|-L               PR Out: Release
    --report-capabilities|-c   PR In: Report Capabilities
    --reserve|-R               PR Out: Reserve
    --transport-id=TIDS|-X TIDS    one or more TransportIDs can
                               be given in several forms
    --unreg|-U                 optional with PR Out Register and Move
    --verbose|-v               output additional debug information
    --version|-V               output version string
    -?                         output this usage message

Performs a SCSI PERSISTENT RESERVE (IN or OUT) command

查询块设备的功能特性：

$ sg_persist -c /dev/sde -vvv
open /dev/sde with flags=0x800
    inquiry cdb: 12 00 00 00 24 00 
      duration=1 ms
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
open /dev/sde with flags=0x802
    Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00 
      duration=0 ms
    persistent reservation in: pass-through requested 8192 bytes but got 8 bytes
    persistent reserve in: response
        00 08 00 80 ea 01 00 00
Report capabilities response:
  Compatible Reservation Handling(CRH): 0
  Specify Initiator Ports Capable(SIP_C): 0
  All Target Ports Capable(ATP_C): 0
  Persist Through Power Loss Capable(PTPL_C): 0
  Type Mask Valid(TMV): 1
  Allow Commands: 0
  Persist Through Power Loss Active(PTPL_A): 0
    Support indicated in Type mask:
      Write Exclusive, all registrants: 1
      Exclusive Access, registrants only: 1
      Write Exclusive, registrants only: 1
      Exclusive Access: 1
      Write Exclusive: 1
      Exclusive Access, all registrants: 1

如果不支持PRs功能，则会直接返回错误。同时这里面的参数描述显示可能不全，可根据“persistent reserve in: response”下面的数据参照PRs命令规范进行对照查看。

查看设备上有哪些“Registered Reservation Key”：

$ sg_persist -k /dev/sde -vvv
open /dev/sde with flags=0x800
    inquiry cdb: 12 00 00 00 24 00 
      duration=0 ms
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
open /dev/sde with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00 
      duration=1 ms
    persistent reservation in: pass-through requested 8192 bytes but got 16 bytes
    persistent reserve in: response
        00 00 00 07 00 00 00 08  00 00 00 00 00 00 0a bc
  PR generation=0x7, 1 registered reservation key follows:
    0xabc

查看设备上的“Persistent Reservation”状态：

$ sg_persist -r /dev/sde -vvv
open /dev/sde with flags=0x800
    inquiry cdb: 12 00 00 00 24 00 
      duration=0 ms
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
open /dev/sde with flags=0x802
    Persistent Reservation In cmd: 5e 01 00 00 00 00 00 20 00 00 
      duration=0 ms
    persistent reservation in: pass-through requested 8192 bytes but got 8 bytes
    persistent reserve in: response
        00 00 00 07 00 00 00 00
  PR generation=0x7, there is NO reservation held

在节点1上的两条链路注册“abc”：

$ sg_persist --out --register --param-sark=abc /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist --out --register --param-sark=abc /dev/sdf
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist /dev/sde
>> No service action given; assume Persistent Reserve In command
>> with Read Keys service action
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x16, 2 registered reservation keys follow:
    0xabc
    0xabc

在节点2上的两条链路注册“123”：

$ sg_persist --out --register --param-sark=123 /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist --out --register --param-sark=123 /dev/sdf
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist /dev/sde
>> No service action given; assume Persistent Reserve In command
>> with Read Keys service action
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x18, 4 registered reservation keys follow:
    0xabc
    0xabc
    0x123
    0x123

获取“Exclusive Access”类型的“Persistent Reservation”，

在节点1上使“abc”获取“Exclusive Access”类型的“Persistent Reservation”，并进行读取：

$ sg_persist --out --reserve --prout-type=3 --param-rk=abc /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist -r /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x18, Reservation follows:
    Key=0xabc
    scope: LU_SCOPE,  type: Exclusive Access

$ sg_persist -r /dev/sdf
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x18, Reservation follows:
    Key=0xabc
    scope: LU_SCOPE,  type: Exclusive Access

$ dd if=/dev/sde of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000620821 秒，825 kB/秒

$ dd if=/dev/sdf of=/dev/null bs=512 count=1 iflag=direct
dd: 读取"/dev/sdf" 时出错: 输入/输出错误
记录了0+0 的读入
记录了0+0 的写出
0字节(0 B)已复制，0.000495205 秒，0.0 kB/秒

在节点2上进行读取操作：

$ dd if=/dev/sde of=/dev/null bs=512 count=1 iflag=direct
dd: 读取"/dev/sde" 时出错: 输入/输出错误
记录了0+0 的读入
记录了0+0 的写出
0字节(0 B)已复制，0.000766011 秒，0.0 kB/秒

$ dd if=/dev/sdf of=/dev/null bs=512 count=1 iflag=direct
dd: 读取"/dev/sdf" 时出错: 输入/输出错误
记录了0+0 的读入
记录了0+0 的写出
0字节(0 B)已复制，0.000623766 秒，0.0 kB/秒

释放“Persistent Reservation”，并在节点1上进行读取：

$ sg_persist --out --release --prout-type=3 --param-rk=abc /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist -k /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x18, 4 registered reservation keys follow:
    0xabc
    0xabc
    0x123
    0x123

$ sg_persist -r /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x18, there is NO reservation held

$ dd if=/dev/sde of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000321204 秒，1.6 MB/秒

$ dd if=/dev/sdf of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000342411 秒，1.5 MB/秒

在节点2上进行读取操作：

$ dd if=/dev/sde of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.00289339 秒，177 kB/秒

$ dd if=/dev/sdf of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000511372 秒，1.0 MB/秒

获取“Exclusive Access − Registrants Only”类型的“Persistent Reservation”，

在节点1上使“abc”获取“Exclusive Access − Registrants Only”类型的“Persistent Reservation”，并进行读取：

$ sg_persist --out --reserve --prout-type=6 --param-rk=abc /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
$ sg_persist -r /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x18, Reservation follows:
    Key=0xabc
    scope: LU_SCOPE,  type: Exclusive Access, registrants only

$ dd if=/dev/sde of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000351323 秒，1.5 MB/秒
$ dd if=/dev/sdf of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000422081 秒，1.2 MB/秒

在节点2上进行读取操作：

$ dd if=/dev/sde of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.00289339 秒，177 kB/秒

$ dd if=/dev/sdf of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000511372 秒，1.0 MB/秒

在同一个LUN上的多个LV上进行测试：

$ sg_persist -k /dev/storage/dmd-test01 
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x0, there are NO registered reservation keys

$ sg_persist --out --register --param-sark=abc /dev/storage/dmd-test01 
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist -k /dev/storage/dmd-test01
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x1, 1 registered reservation key follows:
    0xabc

$ sg_persist -k /dev/storage/dmd-test02
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x1, 1 registered reservation key follows:
    0xabc

$ dd if=/dev/storage/dmd-test02 of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.00127375 秒，402 kB/秒

$ sg_persist --out --reserve --prout-type=3 --param-rk=abc /dev/storage/dmd-test01
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist -r /dev/storage/dmd-test01
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0xb, Reservation follows:
    Key=0xabc
    scope: LU_SCOPE,  type: Exclusive Access

$ sg_persist -r /dev/storage/dmd-test02
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0xb, Reservation follows:
    Key=0xabc
    scope: LU_SCOPE,  type: Exclusive Access

$ dd if=/dev/storage/dmd-test02 of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000892315 秒，574 kB/秒

$ dd if=/dev/storage/dmd-test01 of=/dev/null bs=512 count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
512字节(512 B)已复制，0.000615345 秒，832 kB/秒

反注册“Reservation Key”：

$ sg_persist --out --register --param-rk=abc /dev/sde
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk

$ sg_persist /dev/sde
>> No service action given; assume Persistent Reserve In command
>> with Read Keys service action
  IET       VIRTUAL-DISK  0001
  Peripheral device type: disk
  PR generation=0x10, there are NO registered reservation keys

转载于:https://my.oschina.net/LastRitter/blog/1541364