100% random / 100% read / 0% write / 4K
sudo fio -filename=/dev/nvme0n1p1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=4k -size=1000G -numjobs=56 -runtime=180 -group_reporting -name=rand_100read_4k
rand_100read_4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.1
Starting 56 threads
Jobs: 56 (f=56): [r(56)][100.0%][r=3061MiB/s,w=0KiB/s][r=784k,w=0 IOPS][eta 00m:00s]
rand_100read_4k: (groupid=0, jobs=56): err= 0: pid=15947: Tue Jan 21 21:22:11 2020
read: IOPS=783k, BW=3060MiB/s (3208MB/s)(538GiB/180001msec)
clat (usec): min=11, max=11073, avg=70.71, stdev=15.74
lat (usec): min=11, max=11078, avg=70.78, stdev=15.75
clat percentiles (usec):
| 1.00th=[ 43], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 60],
| 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 72],
| 70.00th=[ 76], 80.00th=[ 82], 90.00th=[ 91], 95.00th=[ 100],
| 99.00th=[ 122], 99.50th=[ 131], 99.90th=[ 155], 99.95th=[ 163],
| 99.99th=[ 188]
bw ( KiB/s): min=49018, max=58167, per=1.80%, avg=56422.98, stdev=684.67, samples=20144
iops : min=12254, max=14541, avg=14105.47, stdev=171.13, samples=20144
lat (usec) : 20=0.01%, 50=4.38%, 100=90.59%, 250=5.02%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%
cpu : usr=2.67%, sys=3.46%, ctx=140986678, majf=0, minf=56
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=140986096,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3060MiB/s (3208MB/s), 3060MiB/s-3060MiB/s (3208MB/s-3208MB/s), io=538GiB (577GB), run=180001-180001msec
Disk stats (read/write):
nvme0n1: ios=140821085/0, merge=0/0, ticks=9607838/0, in_queue=0, util=100.00%
------
100% random / 0% read / 100% write / 4K
sudo fio -filename=/dev/nvme0n1p1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=1000G -numjobs=56 -runtime=180 -group_reporting -name=rand_100write_4k
[sudo] password for ryu:
rand_100write_4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.1
Starting 56 threads
Jobs: 56 (f=56): [w(56)][100.0%][r=0KiB/s,w=1894MiB/s][r=0,w=485k IOPS][eta 00m:00s]
rand_100write_4k: (groupid=0, jobs=56): err= 0: pid=18134: Tue Jan 21 21:42:47 2020
write: IOPS=483k, BW=1888MiB/s (1979MB/s)(332GiB/180001msec)
clat (usec): min=10, max=9914, avg=115.30, stdev=37.22
lat (usec): min=10, max=9914, avg=115.36, stdev=37.22
clat percentiles (usec):
| 1.00th=[ 104], 5.00th=[ 106], 10.00th=[ 108], 20.00th=[ 110],
| 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 112], 60.00th=[ 113],
| 70.00th=[ 114], 80.00th=[ 115], 90.00th=[ 117], 95.00th=[ 119],
| 99.00th=[ 302], 99.50th=[ 453], 99.90th=[ 603], 99.95th=[ 635],
| 99.99th=[ 676]
bw ( KiB/s): min=28640, max=45448, per=1.80%, avg=34787.77, stdev=586.56, samples=20146
iops : min= 7160, max=11362, avg=8696.67, stdev=146.61, samples=20146
lat (usec) : 20=0.01%, 50=0.01%, 100=0.04%, 250=98.84%, 500=0.74%
lat (usec) : 750=0.37%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=1.04%, sys=2.34%, ctx=86984110, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,86982792,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1888MiB/s (1979MB/s), 1888MiB/s-1888MiB/s (1979MB/s-1979MB/s), io=332GiB (356GB), run=180001-180001msec
Disk stats (read/write):
nvme0n1: ios=55/86888639, merge=0/0, ticks=2/9814314, in_queue=32, util=100.00%
------
0% random / 100% read / 0% write / 4K
sudo fio -filename=/dev/nvme0n1p1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=1000G -numjobs=56 -runtime=180 -group_reporting -name=sqe_100read_4k
sqe_100read_4k: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.1
Starting 56 threads
Jobs: 56 (f=56): [R(56)][100.0%][r=3066MiB/s,w=0KiB/s][r=785k,w=0 IOPS][eta 00m:00s]
sqe_100read_4k: (groupid=0, jobs=56): err= 0: pid=18290: Tue Jan 21 21:49:01 2020
read: IOPS=784k, BW=3064MiB/s (3213MB/s)(539GiB/180002msec)
clat (usec): min=13, max=818, avg=70.97, stdev= 3.26
lat (usec): min=13, max=818, avg=71.01, stdev= 3.26
clat percentiles (usec):
| 1.00th=[ 65], 5.00th=[ 67], 10.00th=[ 68], 20.00th=[ 69],
| 30.00th=[ 70], 40.00th=[ 71], 50.00th=[ 72], 60.00th=[ 72],
| 70.00th=[ 73], 80.00th=[ 74], 90.00th=[ 75], 95.00th=[ 77],
| 99.00th=[ 80], 99.50th=[ 82], 99.90th=[ 90], 99.95th=[ 96],
| 99.99th=[ 123]
bw ( KiB/s): min=52192, max=63000, per=1.79%, avg=56077.44, stdev=164.76, samples=20146
iops : min=13048, max=15750, avg=14019.34, stdev=41.21, samples=20146
lat (usec) : 20=0.01%, 50=0.02%, 100=99.95%, 250=0.03%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
cpu : usr=1.93%, sys=3.07%, ctx=141207967, majf=0, minf=56
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=141207845,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3064MiB/s (3213MB/s), 3064MiB/s-3064MiB/s (3213MB/s-3213MB/s), io=539GiB (578GB), run=180002-180002msec
Disk stats (read/write):
nvme0n1: ios=141133135/0, merge=0/0, ticks=9700860/0, in_queue=0, util=100.00%
------
0% random / 0% read / 100% write / 4K
sudo fio -filename=/dev/nvme0n1p1 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=4k -size=1000G -numjobs=56 -runtime=180 -group_reporting -name=sqe_100write_4k
sqe_100write_4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.1
Starting 56 threads
Jobs: 56 (f=56): [W(56)][100.0%][r=0KiB/s,w=1898MiB/s][r=0,w=486k IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=56): err= 0: pid=18434: Tue Jan 21 21:53:02 2020
write: IOPS=485k, BW=1896MiB/s (1988MB/s)(333GiB/180002msec)
clat (usec): min=16, max=14090, avg=115.01, stdev=35.82
lat (usec): min=17, max=14090, avg=115.07, stdev=35.82
clat percentiles (usec):
| 1.00th=[ 104], 5.00th=[ 106], 10.00th=[ 108], 20.00th=[ 110],
| 30.00th=[ 110], 40.00th=[ 111], 50.00th=[ 112], 60.00th=[ 113],
| 70.00th=[ 114], 80.00th=[ 115], 90.00th=[ 117], 95.00th=[ 119],
| 99.00th=[ 269], 99.50th=[ 441], 99.90th=[ 603], 99.95th=[ 619],
| 99.99th=[ 660]
bw ( KiB/s): min=27280, max=35575, per=1.79%, avg=34700.64, stdev=580.95, samples=20153
iops : min= 6820, max= 8893, avg=8675.01, stdev=145.21, samples=20153
lat (usec) : 20=0.01%, 50=0.01%, 100=0.04%, 250=98.94%, 500=0.68%
lat (usec) : 750=0.34%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=0.89%, sys=2.29%, ctx=87357037, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,87355822,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=1896MiB/s (1988MB/s), 1896MiB/s-1896MiB/s (1988MB/s-1988MB/s), io=333GiB (358GB), run=180002-180002msec
Disk stats (read/write):
nvme0n1: ios=66/87306513, merge=0/1, ticks=3/9839825, in_queue=32, util=100.00%
------
100% random / 70% read / 30% write / 4K
sudo fio -filename=/dev/nvme0n1p1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=4k -size=1000G -numjobs=56 -runtime=180 -group_reporting -name=randrw_70read_4k
randrw_70read_4k: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.1
Starting 56 threads
Jobs: 56 (f=56): [m(56)][100.0%][r=2128MiB/s,w=908MiB/s][r=545k,w=233k IOPS][eta 00m:00s]
randrw_70read_4k: (groupid=0, jobs=56): err= 0: pid=18599: Tue Jan 21 21:59:13 2020
read: IOPS=447k, BW=1748MiB/s (1833MB/s)(307GiB/180001msec)
clat (usec): min=10, max=4400, avg=52.66, stdev=30.85
lat (usec): min=10, max=4400, avg=52.71, stdev=30.85
clat percentiles (usec):
| 1.00th=[ 19], 5.00th=[ 21], 10.00th=[ 24], 20.00th=[ 29],
| 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 47], 60.00th=[ 53],
| 70.00th=[ 61], 80.00th=[ 71], 90.00th=[ 88], 95.00th=[ 108],
| 99.00th=[ 172], 99.50th=[ 198], 99.90th=[ 255], 99.95th=[ 277],
| 99.99th=[ 326]
bw ( KiB/s): min=21742, max=45448, per=1.80%, avg=32166.15, stdev=4831.84, samples=20150
iops : min= 5435, max=11362, avg=8041.22, stdev=1207.96, samples=20150
write: IOPS=192k, BW=749MiB/s (786MB/s)(132GiB/180001msec)
clat (usec): min=10, max=4667, avg=166.93, stdev=161.98
lat (usec): min=10, max=4667, avg=167.01, stdev=161.98
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 27], 10.00th=[ 36], 20.00th=[ 49],
| 30.00th=[ 70], 40.00th=[ 116], 50.00th=[ 155], 60.00th=[ 188],
| 70.00th=[ 217], 80.00th=[ 243], 90.00th=[ 273], 95.00th=[ 306],
| 99.00th=[ 988], 99.50th=[ 1303], 99.90th=[ 1631], 99.95th=[ 1745],
| 99.99th=[ 1991]
bw ( KiB/s): min= 9440, max=20040, per=1.80%, avg=13786.14, stdev=2069.84, samples=20150
iops : min= 2360, max= 5010, avg=3446.22, stdev=517.47, samples=20150
lat (usec) : 20=3.32%, 50=42.14%, 100=31.08%, 250=18.16%, 500=4.64%
lat (usec) : 750=0.24%, 1000=0.12%
lat (msec) : 2=0.29%, 4=0.01%, 10=0.01%
cpu : usr=1.88%, sys=3.22%, ctx=115418845, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=80546341,34521876,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=1748MiB/s (1833MB/s), 1748MiB/s-1748MiB/s (1833MB/s-1833MB/s), io=307GiB (330GB), run=180001-180001msec
WRITE: bw=749MiB/s (786MB/s), 749MiB/s-749MiB/s (786MB/s-786MB/s), io=132GiB (141GB), run=180001-180001msec
Disk stats (read/write):
nvme0n1: ios=80432443/34472841, merge=0/0, ticks=4019815/5659680, in_queue=1180, util=100.00%