通过spdk 脚本iostat.py 观测bdev 的IO统计

fio_plugin 使用

1. 编译fio

git clone https://github.com/axboe/fio
cd fio && git checkout fio-3.25
make
make install

2. 下载编译SPDK

2.1 下载spdk

git clone https://github.com/spdk/spdk
cd spdk &&  git checkout v21.07
git submodule update --init

2.2 安装依赖

./scripts/pkgdep.sh
./scripts/pkgdep/debian.sh

2.3 编译

./configure --with-fio=/home/kuhan/fio
make
make install 

PS:
由于fio_plugin要依赖fio中提供的一些依赖包，因此在运行configure时，必须指定fio目录，否则默认fio_plugin不会编译
虚拟机

3. 将设备绑定到用户态

cd <spdk_dir>/scripts
HUGEMEM=4096  ./setup.sh
或者 HUGEMEM=4096 PCI_ALLOWED="0000:00:0a.0" ./setup.sh

4. 对NVMe 下发fio

4.1 配置spdk_nvme_test.fio 文件

[global]
ioengine=spdk
thread=1
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
norandommap=1
runtime=60
iodepth=32
rw=randwrite
bs=4k

[test]
numjobs=1
filename=trtype=PCIe traddr=00:0a.0 ns=1

4.2 直接对NVME下发fio

LD_PRELOAD=/home/kuhan/spdk/build/fio/spdk_nvme fio /home/kuhan/spdk/examples/nvme/fio_plugin/spdk_nvme_test.fio 

5.对NVMe bdev下发fio

5.1 配置yhc_NVMe_bdev.json文件

{
  "subsystems": [
    {
      "subsystem": "bdev",
      "config": [
        {
          "method": "bdev_nvme_attach_controller",
          "params": {
            "trtype": "PCIe",
            "name": "NVMe0",
            "traddr":"0000:00:0a.0"
          }
        }
      ]
    }
  ]
}

5.2 配置yhc_NVMe_bdev.fio文件

[global]
ioengine=spdk_bdev
spdk_json_conf=/home/kuhan/spdk/examples/bdev/fio_plugin/yhc_NVMe_bdev.json
thread=1
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=180
iodepth=128
rw=randrw
bs=4k

[test]
filename=NVMe0n1
numjobs=1

5.3 对NVME bdev下发fio

LD_PRELOAD=/home/kuhan/spdk/build/fio/spdk_bdev fio /home/kuhan/spdk/examples/bdev/fio_plugin/yhc_NVMe_bdev.fio 

6.通过iostat.py 观测bdev

6.1 启动nvmf_target

cd {spdkpath}/build/bin/
./nvmf_tgt &

6.2 配置nvmf 设备

6.2.1 准备工作，把nvme从内核解绑

HUGEMEM=4096 PCI_ALLOWED="0000:00:08.0" ./scripts/setup.sh

6.2.2 创建相应的transport

创建相应的transport，可以使TCP或者RDMA,这里使用TCP

# -t TRTYPE, --trtype TRTYPE
#                        Transport type (ex. RDMA)
# -u IO_UNIT_SIZE, --io-unit-size IO_UNIT_SIZE
#                        I/O unit size (bytes)
# -p MAX_QPAIRS_PER_CTRLR, --max-qpairs-per-ctrlr MAX_QPAIRS_PER_CTRLR
#                       Max number of SQ and CQ per controller. Deprecated, use max-io-qpairs-per-ctrlr
# -c IN_CAPSULE_DATA_SIZE, --in-capsule-data-size IN_CAPSULE_DATA_SIZE
#                        Max number of in-capsule data size
./scripts/rpc.py nvmf_create_transport -t tcp -u 8192 -p 4 -c 0
./scripts/rpc.py nvmf_get_transports

6.2.3 映射 bdev

把nvme 盘映射为一个 bdev

./scripts/rpc.py bdev_nvme_attach_controller -b Nvme0 -t PCIe -a 0000:00:08.0
./scripts/rpc.py bdev_nvme_get_controllers

6.2.4创建一个NVM subsystem

 ./scripts/rpc.py nvmf_create_subsystem nqn.2022-03.io.spdk:cnode1 -a -s SPDK00000000000002 -d SPDK_Controller1

6.2.5用nvme bdev给NVMf subsystem 增加一个namespace,即把nvme bdev和NVMf subsystem 关联

./scripts/rpc.py nvmf_subsystem_add_ns nqn.2022-03.io.spdk:cnode1 Nvme0n1

6.2.6监听nvmf subsystem 端口

让nvmf subsystem监听对应的端口，至此一块nvmf 盘已经建立成功，可以成功的被远端host discover到。

./scripts/rpc.py nvmf_subsystem_add_listener nqn.2022-03.io.spdk:cnode1 -t tcp -a 172.20.18.147 -s 4420

6.3 perf 下发io

./build/examples/perf -i 0 -q 128 -o 4096 -w rw -M 50 -t 60 -r 'trtype:TCP adrfam:IPv4 traddr:172.20.18.147 trsvcid:4420'

6.4 iostat.py 观察

./scripts/iostat.py -m -i 1 -t 180