Windows NVMe 驱动开发支持P2P

作者

QQ群：852283276
微信：arm80x86
微信公众号：青儿创客基地
B站：主页 https://space.bilibili.com/208826118

参考

简单修改使WDK 22H2（VS2022）兼容旧版WDK10
PCIe Peer-to-Peer (P2P)¶
NVIDIA GPUDirect
关于NVIDIA GPUDirect和P2P各种混淆之处

方法一：开发自定义NVMe驱动

只需要支持3条命令，Identify，Read，Write即可，配合自定义的IOCTL命令，实现P2P功能，简单高效。

方法二：SPDK WPDK

SPDK on Windows (Experimental)

使用开源代码，减少开发工作量，兼容Linux代码，缺点是代码庞大，效率低，

测试

perf支持本地的NVMe设备，同时也支持远端的NVMeoF的设备，对于同时测试多块盘，只需要添加-r并指定设备地址即可，例如一个core测试三块盘：

PS C:\dog\program\spdk\build\examples> .\perf.exe -h
C:\dog\program\spdk\build\examples\perf.exe options
        [-b, --allowed-pci-addr <addr> allowed local PCIe device address]
         Example: -b 0000:d8:00.0 -b 0000:d9:00.0
        [-q, --io-depth <val> io depth]
        [-o, --io-size <val> io size in bytes]
        [-O, --io-unit-size io unit size in bytes (4-byte aligned) for SPDK driver. default: same as io size]
        [-P, --num-qpairs <val> number of io queues per namespace. default: 1]
        [-U, --num-unused-qpairs <val> number of unused io queues per controller. default: 0]
        [-w, --io-pattern <pattern> io pattern type, must be one of
                (read, write, randread, randwrite, rw, randrw)]
        [-M, --rwmixread <0-100> rwmixread (100 for reads, 0 for writes)]
        [-F, --zipf <theta> use zipf distribution for random I/O
        [-L, --enable-sw-latency-tracking enable latency tracking via sw, default: disabled]
                -L for latency summary, -LL for detailed histogram
        [-l, --enable-ssd-latency-tracking enable latency tracking via ssd (if supported), default: disabled]
        [-t, --time <sec> time in seconds]
        [-a, --warmup-time <sec> warmup time in seconds]
        [-c, --core-mask <mask> core mask for I/O submission/completion.]
                (default: 1)
        [-D, --disable-sq-cmb disable submission queue in controller memory buffer, default: enabled]
        [-H, --enable-tcp-hdgst enable header digest for TCP transport, default: disabled]
        [-I, --enable-tcp-ddgst enable data digest for TCP transport, default: disabled]
        [-N, --no-shst-notification no shutdown notification process for controllers, default: disabled]
        [-r, --transport <fmt> Transport ID for local PCIe NVMe or NVMeoF]
         Format: 'key:value [key:value] ...'
         Keys:
          trtype      Transport type (e.g. PCIe, RDMA)
          adrfam      Address family (e.g. IPv4, IPv6)
          traddr      Transport address (e.g. 0000:04:00.0 for PCIe or 192.168.100.8 for RDMA)
          trsvcid     Transport service identifier (e.g. 4420)
          subnqn      Subsystem NQN (default: nqn.2014-08.org.nvmexpress.discovery)
          ns          NVMe namespace ID (all active namespaces are used by default)
          hostnqn     Host NQN
         Example: -r 'trtype:PCIe traddr:0000:04:00.0' for PCIe or
                  -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.100.8 trsvcid:4420' for NVMeoF
         Note: can be specified multiple times to test multiple disks/targets.
        [-e, --metadata <fmt> metadata configuration]
         Keys:
          PRACT      Protection Information Action bit (PRACT=1 or PRACT=0)
          PRCHK      Control of Protection Information Checking (PRCHK=GUARD|REFTAG|APPTAG)
         Example: -e 'PRACT=0,PRCHK=GUARD|REFTAG|APPTAG'
                  -e 'PRACT=1,PRCHK=GUARD'
        [-k, --keepalive <ms> keep alive timeout period in millisecond]
        [-s, --hugemem-size <MB> DPDK huge memory size in MB.]
        [-g, --mem-single-seg use single file descriptor for DPDK memory segments]
        [-C, --max-completion-per-poll <val> max completions per poll]
                (default: 0 - unlimited)
        [-i, --shmem-grp-id <id> shared memory group ID]
        [-Q, --skip-errors log I/O errors every N times (default: 1)
         -T, --logflag <flag>    enable log flag (all, json_util, log, nvme, opal, rpc, rpc_client, sock, trace, vmd)
        [-V, --enable-vmd enable VMD enumeration]
        [-z, --disable-zcopy <impl> disable zero copy send for the given sock implementation. Default for posix impl]
        [-Z, --enable-zcopy <impl> enable zero copy send for the given sock implementation]
        [-A, --buffer-alignment IO buffer alignment. Must be power of 2 and not less than cache line (64)]
        [-S, --default-sock-impl <impl> set the default sock impl, e.g. "posix"]
        [-m, --cpu-usage display real-time overall cpu usage on used cores]
        [-G, --enable-debug enable debug logging]
        [--transport-stats dump transport statistics]
        [--iova-mode <mode> specify DPDK IOVA mode: va|pa]
        [--io-queue-size <val> size of NVMe IO queue. Default: maximum allowed by controller]

顺序写，稳态速度估计也就1.3GB/s，队列深度作用不大，core mask对性能影响不大，

PS C:\dog\program\spdk\build\examples> .\perf.exe -q 32 -s 1024 -w write -t 20 -c 0x1 -o 1048576 -r 'trtype:PCIe traddr:0000:01:00.0'
EAL: Multi-process support is requested, but not available.
Initializing NVMe Controllers
[2023-04-13 16:22:17.847447] pci_event.c: 256:spdk_pci_event_listen: *ERROR*: Non-Linux does not support this operation
Attached to NVMe Controller at 0000:01:00.0 [0000:a808]
Associating PCIE (0000:01:00.0) NSID 1 with lcore 0
========================================================
                                                                           Latency(us)
Device Information                     :       IOPS      MiB/s    Average        min        max
PCIE (0000:01:00.0) NSID 1 from core  0