在虚拟机上部署SPDK NVME测试案例

最新推荐文章于 2023-11-10 16:59:16 发布

饿狗007

最新推荐文章于 2023-11-10 16:59:16 发布

阅读量1.4k

点赞数

分类专栏： RocksDB SPDK Nvme 文章标签：数据库

本文链接：https://blog.csdn.net/qq_41108558/article/details/120024134

版权

SPDK 同时被 3 个专栏收录

9 篇文章 7 订阅

订阅专栏

RocksDB

3 篇文章 0 订阅

订阅专栏

Nvme

2 篇文章 1 订阅

订阅专栏

准备spdk环境

详见：浅谈SPDK（二.1）虚拟机搭建SPDK环境–简话版

build目录

我们编译spdk后

./configure
make

，会在build文件下生成如下文件

vagrant@vagrant:~/spdk/build$ ll
total 28
drwxrwxr-x  7 vagrant vagrant 4096 Aug 31 09:28 ./
drwxrwxr-x 28 vagrant vagrant 4096 Aug 31 07:57 ../
drwxr-xr-x  2 root    root    4096 Aug 31 09:29 bin/ 
drwxr-xr-x  2 root    root    4096 Aug 31 09:29 examples/ #存放案例，有helloword、perf、identify等等可执行文件
drwxr-xr-x  2 root    root    4096 Aug 31 09:28 fio/ #含有spdk_bdev和spdk_nvme
drwxr-xr-x  3 root    root    4096 Aug 31 09:28 include/
drwxrwxr-x  3 vagrant vagrant 4096 Aug 31 09:29 lib/

SPDK提供两种形态的fio_plugin：

基于裸盘NVMe的fio_plugin(spdk_nvme)，其特点为I/O通过SPDK用户态驱动直接访问裸盘，常用于评估SPDK用户态驱动在裸盘上的性能。
基于bdev的fio_plugin(spdk_bdev)，其特点为I/O测试基于SPDK块设备bdev之上，所有I/O经由块设备层bdev，再传送至裸盘设备。常用于评估SPDK块设备bdev的性能。

在examples目录下运行NVMe的案例

1、在examples目录下运行NVMe的hello_world测试案例来确保环境已经能够正常使用虚拟的nvme设备

vagrant@vagrant:~/spdk/build$ sudo ./examples/hello_world
[2021-08-31 10:42:31.257911] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:42:31.257986] [ DPDK EAL parameters: [2021-08-31 10:42:31.257999] hello_world [2021-08-31 10:42:31.258005] -c 0x1 [2021-08-31 10:42:31.258012] --log-level=lib.eal:6 [2021-08-31 10:42:31.258016] --log-level=lib.cryptodev:5 [2021-08-31 10:42:31.258023] --log-level=user1:6 [2021-08-31 10:42:31.258026] --iova-mode=pa [2021-08-31 10:42:31.258033] --base-virtaddr=0x200000000000 [2021-08-31 10:42:31.258045] --match-allocations [2021-08-31 10:42:31.258051] --file-prefix=spdk0 [2021-08-31 10:42:31.258055] --proc-type=auto [2021-08-31 10:42:31.258059] ]
EAL: No legacy callbacks, legacy socket not created
Initializing NVMe Controllers
Attaching to 0000:00:0e.0
Attached to 0000:00:0e.0
Using controller ORCL-VBOX-NVME-VER12 (VB1234-56789        ) with 1 namespaces.
  Namespace ID: 1 size: 1GB
Initialization complete.
[2021-08-31 10:42:31.370662] nvme_pcie.c: 462:nvme_pcie_ctrlr_map_io_cmb: *ERROR*: CMB is already in use for submission queues.
INFO: using host memory buffer for IO
Hello world!

2、在examples目录下运行NVMe的perf测试案例测试各种IO的性能

vagrant@vagrant:~/spdk/build$ sudo /home/vagrant/spdk/build/examples/perf -q 128 -o 4096 -w randread -r 'trtype:PCIe traddr:00:0e.0' -t 10
[2021-08-31 10:43:06.591688] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:43:06.591760] [ DPDK EAL parameters: [2021-08-31 10:43:06.591769] perf [2021-08-31 10:43:06.591775] --no-shconf [2021-08-31 10:43:06.591782] -c 0x1 [2021-08-31 10:43:06.591786] --log-level=lib.eal:6 [2021-08-31 10:43:06.591792] --log-level=lib.cryptodev:5 [2021-08-31 10:43:06.591800] --log-level=user1:6 [2021-08-31 10:43:06.591804] --iova-mode=pa [2021-08-31 10:43:06.591808] --base-virtaddr=0x200000000000 [2021-08-31 10:43:06.591812] --match-allocations [2021-08-31 10:43:06.591815] --file-prefix=spdk_pid29908 [2021-08-31 10:43:06.591819] ]
EAL: No legacy callbacks, legacy socket not created
Initializing NVMe Controllers
Attached to NVMe Controller at 0000:00:0e.0 [80ee:4e56]
Associating PCIE (0000:00:0e.0) NSID 1 with lcore 0
Initialization complete. Launching workers.
========================================================
                                                                           Latency(us)
Device Information                     :       IOPS      MiB/s    Average        min        max
PCIE (0000:00:0e.0) NSID 1 from core  0:   79225.90     309.48    1615.63     182.54   25222.40
========================================================
Total                                  :   79225.90     309.48    1615.63     182.54   25222.40

3、在examples目录下运行NVMe的nvme_manage测试案例，这里nvme_manage可用来查看controller的信息，创建namespace以及更新firmware等，具体可以通过选择相应数字选项来操作

vagrant@vagrant:~/spdk/build$ sudo /home/vagrant/spdk/build/examples/nvme_manage 
[2021-08-31 10:44:04.254280] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:44:04.254364] [ DPDK EAL parameters: [2021-08-31 10:44:04.254370] nvme_manage [2021-08-31 10:44:04.254381] --no-shconf [2021-08-31 10:44:04.254387] -c 0x1 [2021-08-31 10:44:04.254391] --log-level=lib.eal:6 [2021-08-31 10:44:04.254395] --log-level=lib.cryptodev:5 [2021-08-31 10:44:04.254399] --log-level=user1:6 [2021-08-31 10:44:04.254403] --iova-mode=pa [2021-08-31 10:44:04.254413] --base-virtaddr=0x200000000000 [2021-08-31 10:44:04.254420] --match-allocations [2021-08-31 10:44:04.254426] --file-prefix=spdk_pid29914 [2021-08-31 10:44:04.254429] ]
EAL: No legacy callbacks, legacy socket not created
NVMe Management Options
	[1: list controllers]
	[2: create namespace]
	[3: delete namespace]
	[4: attach namespace to controller]
	[5: detach namespace from controller]
	[6: format namespace or controller]
	[7: firmware update]
	[8: opal]
	[9: quit]

5、在examples目录下运行NVMe的identify测试案例，用于查看SSD的vendorID，firmware以及型号等信息

vagrant@vagrant:~/spdk/build$ sudo /home/vagrant/spdk/build/examples/identify
[2021-08-31 10:45:29.491650] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:45:29.491732] [ DPDK EAL parameters: [2021-08-31 10:45:29.491743] identify [2021-08-31 10:45:29.491748] --no-shconf [2021-08-31 10:45:29.491751] -c 0x1 [2021-08-31 10:45:29.491755] -n 1 [2021-08-31 10:45:29.491758] -m 0 [2021-08-31 10:45:29.491762] --log-level=lib.eal:6 [2021-08-31 10:45:29.491766] --log-level=lib.cryptodev:5 [2021-08-31 10:45:29.491769] --log-level=user1:6 [2021-08-31 10:45:29.491773] --iova-mode=pa [2021-08-31 10:45:29.491777] --base-virtaddr=0x200000000000 [2021-08-31 10:45:29.491781] --match-allocations [2021-08-31 10:45:29.491786] --file-prefix=spdk_pid29921 [2021-08-31 10:45:29.491789] ]
EAL: No legacy callbacks, legacy socket not created
=====================================================
NVMe Controller at 0000:00:0e.0 [80ee:4e56]
=====================================================
Controller Capabilities/Features
================================
Vendor ID:                             80ee
Subsystem Vendor ID:                   80ee
Serial Number:                         VB1234-56789
Model Number:                          ORCL-VBOX-NVME-VER12
Firmware Version:                      1.0
Recommended Arb Burst:                 0
IEEE OUI Identifier:                   00 00 00
Multi-path I/O
  May have multiple subsystem ports:   No
  May be connected to multiple hosts:  No
  Associated with SR-IOV VF:           No
Max Data Transfer Size:                Unlimited
Max Number of Namespaces:              1
Error Recovery Timeout:                Unlimited
NVMe Specification Version (VS):       1.2
NVMe Specification Version (Identify): 1.2
Maximum Queue Entries:                 4096
Contiguous Queues Required:            Yes
Arbitration Mechanisms Supported
  Weighted Round Robin:                Not Supported
  Vendor Specific:                     Not Supported
Reset Timeout:                         5000 ms
Doorbell Stride:                       4 bytes
NVM Subsystem Reset:                   Not Supported
Command Sets Supported
  NVM Command Set:                     Supported
Boot Partition:                        Not Supported
Memory Page Size Minimum:              4096 bytes
Memory Page Size Maximum:              4096 bytes
Persistent Memory Region:              Not Supported
Optional Asynchronous Events Supported
  Namespace Attribute Notices:         Not Supported
  Firmware Activation Notices:         Not Supported
128-bit Host Identifier:               Not Supported

Controller Memory Buffer Support
================================
Supported:                             Yes
Total Size:                            5242880 bytes
Submission Queues in CMB:              Supported
Completion Queues in CMB:              Supported
Read data and metadata in CMB          Not Supported
Write data and metadata in CMB:        Not Supported

Persistent Memory Region Support
================================
Supported:                             No

Admin Command Set Attributes
============================
Security Send/Receive:                 Not Supported
Format NVM:                            Not Supported
Firmware Activate/Download:            Not Supported
Namespace Management:                  Not Supported
Device Self-Test:                      Not Supported
Directives:                            Not Supported
NVMe-MI:                               Not Supported
Virtualization Management:             Not Supported
Doorbell Buffer Config:                Not Supported
Abort Command Limit:                   5
Async Event Request Limit:             5
Number of Firmware Slots:              N/A
Firmware Slot 1 Read-Only:             N/A
Firmware Update Granularity:           No Information Provided
Per-Namespace SMART Log:               No
Asymmetric Namespace Access Log Page:  Not Supported
Command Effects Log Page:              Not Supported
Get Log Page Extended Data:            Not Supported
Telemetry Log Pages:                   Not Supported
Error Log Page Entries Supported:      1
Keep Alive:                            Not Supported

NVM Command Set Attributes
==========================
Submission Queue Entry Size
  Max:                       64
  Min:                       64
Completion Queue Entry Size
  Max:                       16
  Min:                       16
Number of Namespaces:        1
Compare Command:             Not Supported
Write Uncorrectable Command: Not Supported
Dataset Management Command:  Not Supported
Write Zeroes Command:        Not Supported
Set Features Save Field:     Not Supported
Reservations:                Not Supported
Timestamp:                   Not Supported
Volatile Write Cache:        Not Present
Atomic Write Unit (Normal):  1
Atomic Write Unit (PFail):   1
Atomic Compare & Write Unit: 1
Fused Compare & Write:       Not Supported
Scatter-Gather List
  SGL Command Set:           Not Supported
  SGL Keyed:                 Not Supported
  SGL Bit Bucket Descriptor: Not Supported
  SGL Metadata Pointer:      Not Supported
  Oversized SGL:             Not Supported
  SGL Metadata Address:      Not Supported
  SGL Offset:                Not Supported
  Transport SGL Data Block:  Not Supported
Replay Protected Memory Block:  Not Supported

Firmware Slot Information
=========================
Active slot:                 0


Error Log
=========

Arbitration
===========
Arbitration Burst:           1

Power Management
================
Number of Power States:      1
Current Power State:         Power State #0
Power State #0:  Max Power:   0.01 W
Non-Operational Permissive Mode: Not Supported

Health Information
==================
Critical Warnings:
  Available Spare Space:     OK
  Temperature:               OK
  Device Reliability:        OK
  Read Only:                 No
  Volatile Memory Backup:    OK
Current Temperature:         0 Kelvin (-273 Celsius)
Temperature Threshold:       0 Kelvin (-273 Celsius)
Available Spare:             0%
Available Spare Threshold:   0%
Life Percentage Used:        0%
Data Units Read:             0
Data Units Written:          0
Host Read Commands:          0
Host Write Commands:         0
Controller Busy Time:        0 minutes
Power Cycles:                0
Power On Hours:              0 hours
Unsafe Shutdowns:            0
Unrecoverable Media Errors:  0
Lifetime Error Log Entries:  0
Warning Temperature Time:    0 minutes
Critical Temperature Time:   0 minutes

Number of Queues
================
Number of I/O Submission Queues:      33
Number of I/O Completion Queues:      33

Active Namespaces
=================
Namespace ID:1
Command Set Identifier:                NVM (00h)
Deallocate:                            Not Supported
Deallocated/Unwritten Error:           Not Supported
Deallocated Read Value:                Unknown
Deallocate in Write Zeroes:            Not Supported
Deallocated Guard Field:               0xFFFF
Flush:                                 Not Supported
Reservation:                           Not Supported
Namespace Sharing Capabilities:        Private
Size (in LBAs):                        2097152 (1GiB)
Capacity (in LBAs):                    2097152 (1GiB)
Utilization (in LBAs):                 2097152 (1GiB)
NGUID:                                 1D3F8E91E0E61844949E6FC7FC126538
Thin Provisioning:                     Not Supported
Per-NS Atomic Units:                   No
NGUID/EUI64 Never Reused:              No
Number of LBA Formats:                 1
Current LBA Format:                    LBA Format #00
LBA Format #00: Data Size:   512  Metadata Size:     0

在test目录运行nvme测试案例

1、运行aer测试案例

vagrant@vagrant:~/spdk$ ./test/nvme/aer/aer -r 'trtype:PCIe traddr:00:0e.0'
[2021-08-31 10:47:59.658341] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:47:59.658461] [ DPDK EAL parameters: [2021-08-31 10:47:59.658469] aer [2021-08-31 10:47:59.658473] --no-shconf [2021-08-31 10:47:59.658486] -c 0x1 [2021-08-31 10:47:59.658495] --log-level=lib.eal:6 [2021-08-31 10:47:59.658500] --log-level=lib.cryptodev:5 [2021-08-31 10:47:59.658503] --log-level=user1:6 [2021-08-31 10:47:59.658510] --iova-mode=pa [2021-08-31 10:47:59.658514] --base-virtaddr=0x200000000000 [2021-08-31 10:47:59.658519] --match-allocations [2021-08-31 10:47:59.658523] --file-prefix=spdk_pid29926 [2021-08-31 10:47:59.658527] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2021-08-31 10:47:59.662344] init.c: 587:spdk_env_init: *ERROR*: Failed to initialize DPDK
Unable to initialize SPDK env
vagrant@vagrant:~/spdk$ sudo ./test/nvme/aer/aer -r 'trtype:PCIe traddr:00:0e.0'
[2021-08-31 10:48:04.770409] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:48:04.770507] [ DPDK EAL parameters: [2021-08-31 10:48:04.770549] aer [2021-08-31 10:48:04.770571] --no-shconf [2021-08-31 10:48:04.770598] -c 0x1 [2021-08-31 10:48:04.770622] --log-level=lib.eal:6 [2021-08-31 10:48:04.770637] --log-level=lib.cryptodev:5 [2021-08-31 10:48:04.770646] --log-level=user1:6 [2021-08-31 10:48:04.770656] --iova-mode=pa [2021-08-31 10:48:04.770665] --base-virtaddr=0x200000000000 [2021-08-31 10:48:04.770687] --match-allocations [2021-08-31 10:48:04.770695] --file-prefix=spdk_pid29929 [2021-08-31 10:48:04.770699] ]
EAL: No legacy callbacks, legacy socket not created
Asynchronous Event Request test
Attaching to 0000:00:0e.0
Attached to 0000:00:0e.0
Registering asynchronous event callbacks...
Cleaning up...

2、运行e2edp测试案例

vagrant@vagrant:~/spdk$ sudo ./test/nvme/e2edp/nvme_dp                          
[2021-08-31 10:49:23.562278] Starting SPDK v21.04.1-pre git sha1 7dc38f83f / DPDK 20.11.0 initialization...
[2021-08-31 10:49:23.562359] [ DPDK EAL parameters: [2021-08-31 10:49:23.562371] nvme_dp [2021-08-31 10:49:23.562376] -c 0x1 [2021-08-31 10:49:23.562379] --log-level=lib.eal:6 [2021-08-31 10:49:23.562383] --log-level=lib.cryptodev:5 [2021-08-31 10:49:23.562387] --log-level=user1:6 [2021-08-31 10:49:23.562391] --iova-mode=pa [2021-08-31 10:49:23.562395] --base-virtaddr=0x200000000000 [2021-08-31 10:49:23.562398] --match-allocations [2021-08-31 10:49:23.562402] --file-prefix=spdk0 [2021-08-31 10:49:23.562406] --proc-type=auto [2021-08-31 10:49:23.562410] ]
EAL: No legacy callbacks, legacy socket not created
NVMe Write/Read with End-to-End data protection test
Attaching to 0000:00:0e.0
Attached to 0000:00:0e.0
Cleaning up...

3、运行run_perf.sh命令

vagrant@vagrant:~/spdk$ sudo ./test/nvme/perf/run_perf.sh --numjobs=1 --iodepth=128  --rw=randread --fio-bin=/usr/src/fio/fio --ramp-time=10 --run-time=10

饿狗007

关注

0
点赞
踩
5

收藏

觉得还不错? 一键收藏
0
评论
在虚拟机上部署SPDK NVME测试案例

准备spdk环境详见：浅谈SPDK（二.1）虚拟机搭建SPDK环境–简话版build目录我们编译spdk后./configuremake，会在build文件下生成如下文件vagrant@vagrant:~/spdk/build$ lltotal 28drwxrwxr-x 7 vagrant vagrant 4096 Aug 31 09:28 ./drwxrwxr-x 28 vagrant vagrant 4096 Aug 31 07:57 ../drwxr-xr-x 2 root
复制链接

扫一扫