【PCIe测速】

dd

NVMe读

dd if=/dev/nvme0n1 of=/dev/null bs=1M count=100000 oflag=nocache

记录了100000+0 的读入
记录了100000+0 的写出
104857600000 bytes (105 GB, 98 GiB) copied, 75.9709 s, 1.4 GB/s

NVMe并发读

(dd if=/dev/nvme0n1 of=/dev/null bs=1M count=100000 oflag=nocache &);(dd if=/dev/nvme0n1 of=/dev/null bs=1M count=100000 oflag=nocache skip=100000 &)

记录了100000+0 的读入
记录了100000+0 的写出
104857600000 bytes (105 GB, 98 GiB) copied, 143.454 s, 731 MB/s
记录了100000+0 的读入
记录了100000+0 的写出
104857600000 bytes (105 GB, 98 GiB) copied, 144.082 s, 728 MB/

fio

NVMe顺序读

fio -filename=/dev/nvme0n1 -direct=1 -iodepth=4 -thread -rw=read -bs=1M -size=100G -numjobs=100 -name="nvme 1M read test" -group_reporting -runtime=100

nvme 1M read test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=4
...
fio-3.12
Starting 100 threads
Jobs: 100 (f=100): [R(100)][100.0%][r=575MiB/s][r=574 IOPS][eta 00m:00s] 
nvme 1M read test: (groupid=0, jobs=100): err= 0: pid=1089: Wed Apr 20 10:24:41 2022
  read: IOPS=755, BW=755MiB/s (792MB/s)(73.9GiB/100194msec)
    clat (msec): min=9, max=328, avg=132.27, stdev=66.50
     lat (msec): min=9, max=328, avg=132.27, stdev=66.50
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   57], 10.00th=[   64], 20.00th=[   77],
     | 30.00th=[  102], 40.00th=[  111], 50.00th=[  116], 60.00th=[  129],
     | 70.00th=[  136], 80.00th=[  153], 90.00th=[  271], 95.00th=[  288],
     | 99.00th=[  300], 99.50th=[  305], 99.90th=[  321], 99.95th=[  326],
     | 99.99th=[  330]
   bw (  KiB/s): min= 1980, max=24576, per=1.00%, avg=7737.93, stdev=3857.24, samples=19997
   iops        : min=    1, max=   24, avg= 7.50, stdev= 3.77, samples=19997
  lat (msec)   : 10=0.01%, 20=0.02%, 50=2.87%, 100=26.80%, 250=59.25%
  lat (msec)   : 500=11.06%
  cpu          : usr=0.00%, sys=0.02%, ctx=75896, majf=0, minf=6400
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=75673,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=755MiB/s (792MB/s), 755MiB/s-755MiB/s (792MB/s-792MB/s), io=73.9GiB (79.3GB), run=100194-100194msec

Disk stats (read/write):
  nvme0n1: ios=75634/66, merge=0/0, ticks=9997254/14425, in_queue=10013738, util=100.00%

NVMe随机读

fio -filename=/dev/nvme0n1 -direct=1 -iodepth=4 -thread -rw=randread -bs=1M -size=100G -numjobs=100 -name="nvme 1M randread test" -group_reporting -runtime=100

nvme 1M randread test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=4
...
fio-3.12
Starting 100 threads
Jobs: 100 (f=100): [r(100)][100.0%][r=1367MiB/s][r=1367 IOPS][eta 00m:00s]
nvme 1M randread test: (groupid=0, jobs=100): err= 0: pid=959: Wed Apr 20 10:22:05 2022
  read: IOPS=1370, BW=1370MiB/s (1437MB/s)(134GiB/100083msec)
    clat (usec): min=899, max=133746, avg=72939.62, stdev=26475.69
     lat (usec): min=904, max=133746, avg=72940.25, stdev=26475.69
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   45], 10.00th=[   45], 20.00th=[   45],
     | 30.00th=[   47], 40.00th=[   52], 50.00th=[   70], 60.00th=[   93],
     | 70.00th=[   96], 80.00th=[  100], 90.00th=[  104], 95.00th=[  112],
     | 99.00th=[  128], 99.50th=[  129], 99.90th=[  131], 99.95th=[  132],
     | 99.99th=[  133]
   bw (  KiB/s): min= 6131, max=26624, per=1.00%, avg=14028.66, stdev=5378.91, samples=19982
   iops        : min=    5, max=   26, avg=13.65, stdev= 5.26, samples=19982
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=38.47%
  lat (msec)   : 100=44.61%, 250=16.90%
  cpu          : usr=0.00%, sys=0.04%, ctx=137505, majf=0, minf=6400
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=137134,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=1370MiB/s (1437MB/s), 1370MiB/s-1370MiB/s (1437MB/s-1437MB/s), io=134GiB (144GB), run=100083-100083msec

Disk stats (read/write):
  nvme0n1: ios=137132/50, merge=0/0, ticks=9997138/3034, in_queue=9948996, util=99.57%