简介Storage测试
在谈到Storage时我们不得不谈到I/O读写,因为存储绕不开I/O读写,而怎么才能反映出一块存储盘的I/O读写呢。FIO工具就是因此应运而生的(博主认为FIO是真能贴近显示I/O读写性能的工具)，而关于FIO的简介在这里不在对大家进行过多的介绍。如果有不知道的同学可以让百度support下。
另外博主在这里介绍的Storage测试硬体环境是在centos/redhat系统下进行的，关于下Windows 、安卓等系统下storage测试博主在这里就不多加陈述。
如果在介绍过程中出现错误，请联系我帮忙指正,以防我误人子弟(-,-!)
硬体环境
1.基本的服务器硬件环境
2.两块及以上storage设备(SATA_SSD/HDD/NVME)
软体环境
1.Centos/redhat 系统
2.FIO Tool(version:2.7+)
3.The latest FW(习惯性操作)
4.issdcm/hdparm/nvme Tool
软体连接
FIO：http://brick.kernel.dk/snaps/
Issdcm:https://sourceforge.net/projects/hdparm/
nvme:https://download.csdn.net/download/qq_41901686/11630302
libaio引擎：yum install libaio*(如果是Debian就使用apt-get install libaio)
Centos/Redhat:OS自备
确认FIO 是否带有异步libaio引擎
Libaio: Linux专有的异步IO.Linux仅支持非buffered IO的队列行为.(linux特用的i/o异步)
#检查FIO引擎
fio --enghelp
#假如enghelp中没有libaio的话，请安装以下软体
[root@localhost \]#yum install libaio-devel
[root@localhost \]#yum install libaio*
FIO参数说明(详细的参照官方文档，这里简略的介绍常见参数)
FIO单盘压力测试案例
如下是FIO 配置文件stress.ini
#[root@localhost \]#fio stress.ini
[global]
ioengine=libaio
direct=1
time_based=1
ramp_time=75
group_reporting
log_avg_msec=1000
[Precondition]
numjobs=32
filename=/dev/nvme1n1
runtime=3600
iodepth=50
rw=randwrite
bs=16k
runtime=500
write_iops_log=4k_randwrite_32job_QD50
stonewall
[64KB Ran WR 32job QD100 First]
filename=/dev/nvme1n1
numjobs=32
write_iops_log=64k_randwrite_32job_QD100
runtime=259200
iodepth=100
rw=randwrite
bs=64k
Stonewall
FIO单盘性能测试案例(仅顺序读写)
#单盘顺序读写
#[root@localhost \]# fio seq_write_read.ini
[global]
direct=1
ioengine=libaio
time_based=1
end_fsync=1
group_reporting
log_avg_msec=500
bwavgtime=500
[Precondition]
numjobs=1
iodepth=32
rw=write
bs=512k
runtime=4h
stonewall
[512kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=512k
runtime=600
write_bw_log=512k_seq_wr_1job_QD32
stonewall
[512kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=512k
runtime=600
write_bw_log=512k_seq_rd_1job_QD32
stonewall
[128kB Seq WR 1job QD1]
numjobs=1
iodepth=1
rw=write
bs=128k
runtime=600
write_bw_log=128k_seq_wr_1job_QD1
stonewall
[128kB Seq WR 1job QD2]
numjobs=1
iodepth=2
rw=write
bs=128k
runtime=600
write_bw_log=128k_seq_wr_1job_QD2
stonewall
[128kB Seq WR 1job QD4]
numjobs=1
iodepth=4
rw=write
bs=128k
runtime=600
write_bw_log=128k_seq_wr_1job_QD4
stonewall
[128kB Seq WR 1job QD8]
numjobs=1
iodepth=8
rw=write
bs=128k
runtime=600
write_bw_log=128k_seq_wr_1job_QD8
stonewall
[128kB Seq WR 1job QD16]
numjobs=1
iodepth=16
rw=write
bs=128k
runtime=600
write_bw_log=128k_seq_wr_1job_QD16
stonewall
[128kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=128k
runtime=600
write_bw_log=128k_seq_wr_1job_QD32
stonewall
[128kB Seq RD 1job QD1]
numjobs=1
iodepth=1
rw=read
bs=128k
runtime=600
write_bw_log=128k_seq_rd_1job_QD1
stonewall
[128kB Seq RD 1job QD2]
numjobs=1
iodepth=2
rw=read
bs=128k
runtime=600
write_bw_log=128k_seq_rd_1job_QD2
stonewall
[128kB Seq RD 1job QD4]
numjobs=1
iodepth=4
rw=read
bs=128k
runtime=600
write_bw_log=128k_seq_rd_1job_QD4
stonewall
[128kB Seq RD 1job QD8]
numjobs=1
iodepth=8
rw=read
bs=128k
runtime=600
write_bw_log=128k_seq_rd_1job_QD8
stonewall
[128kB Seq RD 1job QD16]
numjobs=1
iodepth=16
rw=read
bs=128k
runtime=600
write_bw_log=128k_seq_rd_1job_QD16
stonewall
[128kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=128k
runtime=600
write_bw_log=128k_seq_rd_1job_QD32
stonewall
[64kB Seq WR 1job QD1]
numjobs=1
iodepth=1
rw=write
bs=64k
runtime=600
write_bw_log=64k_seq_wr_1job_QD1
stonewall
[64kB Seq WR 1job QD2]
numjobs=1
iodepth=2
rw=write
bs=64k
runtime=600
write_bw_log=64k_seq_wr_1job_QD2
stonewall
[64kB Seq WR 1job QD4]
numjobs=1
iodepth=4
rw=write
bs=64k
runtime=600
write_bw_log=64k_seq_wr_1job_QD4
stonewall
[64kB Seq WR 1job QD8]
numjobs=1
iodepth=8
rw=write
bs=64k
runtime=600
write_bw_log=64k_seq_wr_1job_QD8
stonewall
[64kB Seq WR 1job QD16]
numjobs=1
iodepth=16
rw=write
bs=64k
runtime=600
write_bw_log=64k_seq_wr_1job_QD16
stonewall
[64kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=64k
runtime=600
write_bw_log=64k_seq_wr_1job_QD32
stonewall
[64kB Seq RD 1job QD1]
numjobs=1
iodepth=1
rw=read
bs=64k
runtime=600
write_bw_log=64k_seq_rd_1job_QD1
stonewall
[64kB Seq RD 1job QD2]
numjobs=1
iodepth=2
rw=read
bs=64k
runtime=600
write_bw_log=64k_seq_rd_1job_QD2
stonewall
[64kB Seq RD 1job QD4]
numjobs=1
iodepth=4
rw=read
bs=64k
runtime=600
write_bw_log=64k_seq_rd_1job_QD4
stonewall
[64kB Seq RD 1job QD8]
numjobs=1
iodepth=8
rw=read
bs=64k
runtime=600
write_bw_log=64k_seq_rd_1job_QD8
stonewall
[64kB Seq RD 1job QD16]
numjobs=1
iodepth=16
rw=read
bs=64k
runtime=600
write_bw_log=64k_seq_rd_1job_QD16
stonewall
[64kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=64k
runtime=600
write_bw_log=64k_seq_rd_1job_QD32
stonewall
[32kB Seq WR 1job QD1]
numjobs=1
iodepth=1
rw=write
bs=32k
runtime=600
write_bw_log=32k_seq_wr_1job_QD1
stonewall
[32kB Seq WR 1job QD2]
numjobs=1
iodepth=2
rw=write
bs=32k
runtime=600
write_bw_log=32k_seq_wr_1job_QD2
stonewall
[32kB Seq WR 1job QD4]
numjobs=1
iodepth=4
rw=write
bs=32k
runtime=600
write_bw_log=32k_seq_wr_1job_QD4
stonewall
[32kB Seq WR 1job QD8]
numjobs=1
iodepth=8
rw=write
bs=32k
runtime=600
write_bw_log=32k_seq_wr_1job_QD8
stonewall
[32kB Seq WR 1job QD16]
numjobs=1
iodepth=16
rw=write
bs=32k
runtime=600
write_bw_log=32k_seq_wr_1job_QD16
stonewall
[32kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=32k
runtime=600
write_bw_log=32k_seq_wr_1job_QD32
stonewall
[32kB Seq RD 1job QD1]
numjobs=1
iodepth=1
rw=read
bs=32k
runtime=600
write_bw_log=32k_seq_rd_1job_QD1
stonewall
[32kB Seq RD 1job QD2]
numjobs=1
iodepth=2
rw=read
bs=32k
runtime=600
write_bw_log=32k_seq_rd_1job_QD2
stonewall
[32kB Seq RD 1job QD4]
numjobs=1
iodepth=4
rw=read
bs=32k
runtime=600
write_bw_log=32k_seq_rd_1job_QD4
stonewall
[32kB Seq RD 1job QD8]
numjobs=1
iodepth=8
rw=read
bs=32k
runtime=600
write_bw_log=32k_seq_rd_1job_QD8
stonewall
[32kB Seq RD 1job QD16]
numjobs=1
iodepth=16
rw=read
bs=32k
runtime=600
write_bw_log=32k_seq_rd_1job_QD16
stonewall
[32kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=32k
runtime=600
write_bw_log=32k_seq_rd_1job_QD32
stonewall
[16kB Seq WR 1job QD1]
numjobs=1
iodepth=1
rw=write
bs=16k
runtime=600
write_bw_log=16k_seq_wr_1job_QD1
stonewall
[16kB Seq WR 1job QD2]
numjobs=1
iodepth=2
rw=write
bs=16k
runtime=600
write_bw_log=16k_seq_wr_1job_QD2
stonewall
[16kB Seq WR 1job QD4]
numjobs=1
iodepth=4
rw=write
bs=16k
runtime=600
write_bw_log=16k_seq_wr_1job_QD4
stonewall
[16kB Seq WR 1job QD8]
numjobs=1
iodepth=8
rw=write
bs=16k
runtime=600
write_bw_log=16k_seq_wr_1job_QD8
stonewall
[16kB Seq WR 1job QD16]
numjobs=1
iodepth=16
rw=write
bs=16k
runtime=600
write_bw_log=16k_seq_wr_1job_QD16
stonewall
[16kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=16k
runtime=600
write_bw_log=16k_seq_wr_1job_QD32
stonewall
[16kB Seq RD 1job QD1]
numjobs=1
iodepth=1
rw=read
bs=16k
runtime=600
write_bw_log=16k_seq_rd_1job_QD1
stonewall
[16kB Seq RD 1job QD2]
numjobs=1
iodepth=2
rw=read
bs=16k
runtime=600
write_bw_log=16k_seq_rd_1job_QD2
stonewall
[16kB Seq RD 1job QD4]
numjobs=1
iodepth=4
rw=read
bs=16k
runtime=600
write_bw_log=16k_seq_rd_1job_QD4
stonewall
[16kB Seq RD 1job QD8]
numjobs=1
iodepth=8
rw=read
bs=16k
runtime=600
write_bw_log=16k_seq_rd_1job_QD8
stonewall
[16kB Seq RD 1job QD16]
numjobs=1
iodepth=16
rw=read
bs=16k
runtime=600
write_bw_log=16k_seq_rd_1job_QD16
stonewall
[16kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=16k
runtime=600
write_bw_log=16k_seq_rd_1job_QD32
stonewall
[8kB Seq WR 1job QD1]
numjobs=1
iodepth=1
rw=write
bs=8k
runtime=600
write_bw_log=8k_seq_wr_1job_QD1
stonewall
[8kB Seq WR 1job QD2]
numjobs=1
iodepth=2
rw=write
bs=8k
runtime=600
write_bw_log=8k_seq_wr_1job_QD2
stonewall
[8kB Seq WR 1job QD4]
numjobs=1
iodepth=4
rw=write
bs=8k
runtime=600
write_bw_log=8k_seq_wr_1job_QD4
stonewall
[8kB Seq WR 1job QD8]
numjobs=1
iodepth=8
rw=write
bs=8k
runtime=600
write_bw_log=8k_seq_wr_1job_QD8
stonewall
[8kB Seq WR 1job QD16]
numjobs=1
iodepth=16
rw=write
bs=8k
runtime=600
write_bw_log=8k_seq_wr_1job_QD16
stonewall
[8kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=8k
runtime=600
write_bw_log=8k_seq_wr_1job_QD32
stonewall
[8kB Seq RD 1job QD1]
numjobs=1
iodepth=1
rw=read
bs=8k
runtime=600
write_bw_log=8k_seq_rd_1job_QD1
stonewall
[8kB Seq RD 1job QD2]
numjobs=1
iodepth=2
rw=read
bs=8k
runtime=600
write_bw_log=8k_seq_rd_1job_QD2
stonewall
[8kB Seq RD 1job QD4]
numjobs=1
iodepth=4
rw=read
bs=8k
runtime=600
write_bw_log=8k_seq_rd_1job_QD4
stonewall
[8kB Seq RD 1job QD8]
numjobs=1
iodepth=8
rw=read
bs=8k
runtime=600
write_bw_log=8k_seq_rd_1job_QD8
stonewall
[8kB Seq RD 1job QD16]
numjobs=1
iodepth=16
rw=read
bs=8k
runtime=600
write_bw_log=8k_seq_rd_1job_QD16
stonewall
[8kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=8k
runtime=600
write_bw_log=8k_seq_rd_1job_QD32
stonewall
[4kB Seq WR 1job QD1]
numjobs=1
iodepth=1
rw=write
bs=4k
runtime=600
write_bw_log=4k_seq_wr_1job_QD1
stonewall
[4kB Seq WR 1job QD2]
numjobs=1
iodepth=2
rw=write
bs=4k
runtime=600
write_bw_log=4k_seq_wr_1job_QD2
stonewall
[4kB Seq WR 1job QD4]
numjobs=1
iodepth=4
rw=write
bs=4k
runtime=600
write_bw_log=4k_seq_wr_1job_QD4
stonewall
[4kB Seq WR 1job QD8]
numjobs=1
iodepth=8
rw=write
bs=4k
runtime=600
write_bw_log=4k_seq_wr_1job_QD8
stonewall
[4kB Seq WR 1job QD16]
numjobs=1
iodepth=16
rw=write
bs=4k
runtime=600
write_bw_log=4k_seq_wr_1job_QD16
stonewall
[4kB Seq WR 1job QD32]
numjobs=1
iodepth=32
rw=write
bs=4k
runtime=600
write_bw_log=4k_seq_wr_1job_QD32
stonewall
[4kB Seq RD 1job QD1]
numjobs=1
iodepth=1
rw=read
bs=4k
runtime=600
write_bw_log=4k_seq_rd_1job_QD1
stonewall
[4kB Seq RD 1job QD2]
numjobs=1
iodepth=2
rw=read
bs=4k
runtime=600
write_bw_log=4k_seq_rd_1job_QD2
stonewall
[4kB Seq RD 1job QD4]
numjobs=1
iodepth=4
rw=read
bs=4k
runtime=600
write_bw_log=4k_seq_rd_1job_QD4
stonewall
[4kB Seq RD 1job QD8]
numjobs=1
iodepth=8
rw=read
bs=4k
runtime=600
write_bw_log=4k_seq_rd_1job_QD8
stonewall
[4kB Seq RD 1job QD16]
numjobs=1
iodepth=16
rw=read
bs=4k
runtime=600
write_bw_log=4k_seq_rd_1job_QD16
stonewall
[4kB Seq RD 1job QD32]
numjobs=1
iodepth=32
rw=read
bs=4k
runtime=600
write_bw_log=4k_seq_rd_1job_QD32
stonewall
FIO Log说明
以下是运行顺序读写seq_write_read文件产生的log，借以说明fio log的含义。
Precondition: (g=0): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=32
512kB Seq WR 1job QD32: (g=1): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=32
512kB Seq RD 1job QD32: (g=2): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=32
128kB Seq WR 1job QD1: (g=3): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
128kB Seq WR 1job QD2: (g=4): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=2
128kB Seq WR 1job QD4: (g=5): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4
128kB Seq WR 1job QD8: (g=6): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8
128kB Seq WR 1job QD16: (g=7): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16
128kB Seq WR 1job QD32: (g=8): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
128kB Seq RD 1job QD1: (g=9): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1
128kB Seq RD 1job QD2: (g=10): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=2
128kB Seq RD 1job QD4: (g=11): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=4
128kB Seq RD 1job QD8: (g=12): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=8
128kB Seq RD 1job QD16: (g=13): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=16
128kB Seq RD 1job QD32: (g=14): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
64kB Seq WR 1job QD1: (g=15): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1
64kB Seq WR 1job QD2: (g=16): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=2
64kB Seq WR 1job QD4: (g=17): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=4
64kB Seq WR 1job QD8: (g=18): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8
64kB Seq WR 1job QD16: (g=19): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16
64kB Seq WR 1job QD32: (g=20): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=32
64kB Seq RD 1job QD1: (g=21): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1
64kB Seq RD 1job QD2: (g=22): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=2
64kB Seq RD 1job QD4: (g=23): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=4
64kB Seq RD 1job QD8: (g=24): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=8
64kB Seq RD 1job QD16: (g=25): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16
64kB Seq RD 1job QD32: (g=26): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=32
32kB Seq WR 1job QD1: (g=27): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
32kB Seq WR 1job QD2: (g=28): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=2
32kB Seq WR 1job QD4: (g=29): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=4
32kB Seq WR 1job QD8: (g=30): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=8
32kB Seq WR 1job QD16: (g=31): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=16
32kB Seq WR 1job QD32: (g=32): rw=write, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
32kB Seq RD 1job QD1: (g=33): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1
32kB Seq RD 1job QD2: (g=34): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=2
32kB Seq RD 1job QD4: (g=35): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=4
32kB Seq RD 1job QD8: (g=36): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=8
32kB Seq RD 1job QD16: (g=37): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=16
32kB Seq RD 1job QD32: (g=38): rw=read, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
16kB Seq WR 1job QD1: (g=39): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
16kB Seq WR 1job QD2: (g=40): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=2
16kB Seq WR 1job QD4: (g=41): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=4
16kB Seq WR 1job QD8: (g=42): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
16kB Seq WR 1job QD16: (g=43): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=16
16kB Seq WR 1job QD32: (g=44): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=32
16kB Seq RD 1job QD1: (g=45): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1
16kB Seq RD 1job QD2: (g=46): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=2
16kB Seq RD 1job QD4: (g=47): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=4
16kB Seq RD 1job QD8: (g=48): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=8
16kB Seq RD 1job QD16: (g=49): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=16
16kB Seq RD 1job QD32: (g=50): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=32
8kB Seq WR 1job QD1: (g=51): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
8kB Seq WR 1job QD2: (g=52): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=2
8kB Seq WR 1job QD4: (g=53): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=4
8kB Seq WR 1job QD8: (g=54): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=8
8kB Seq WR 1job QD16: (g=55): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
8kB Seq WR 1job QD32: (g=56): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32
8kB Seq RD 1job QD1: (g=57): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
8kB Seq RD 1job QD2: (g=58): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=2
8kB Seq RD 1job QD4: (g=59): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=4
8kB Seq RD 1job QD8: (g=60): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=8
8kB Seq RD 1job QD16: (g=61): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=16
8kB Seq RD 1job QD32: (g=62): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=32
4kB Seq WR 1job QD1: (g=63): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
4kB Seq WR 1job QD2: (g=64): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
4kB Seq WR 1job QD4: (g=65): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
4kB Seq WR 1job QD8: (g=66): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
4kB Seq WR 1job QD16: (g=67): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
4kB Seq WR 1job QD32: (g=68): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
4kB Seq RD 1job QD1: (g=69): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
4kB Seq RD 1job QD2: (g=70): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
4kB Seq RD 1job QD4: (g=71): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
4kB Seq RD 1job QD8: (g=72): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
4kB Seq RD 1job QD16: (g=73): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
4kB Seq RD 1job QD32: (g=74): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
fio-2.21
Starting 75 processes
Precondition: (groupid=0, jobs=1): err= 0: pid=2763: Mon Jul 15 21:38:51 2019
write: IOPS=501, BW=251MiB/s (263MB/s)(3525GiB/14400489msec)
slat (usec): min=13, max=830, avg=22.11, stdev= 3.05
clat (msec): min=2, max=2510, avg=63.81, stdev= 9.60
lat (msec): min=2, max=2511, avg=63.83, stdev= 9.60
clat percentiles (msec):
| 1.00th=[ 58], 5.00th=[ 60], 10.00th=[ 61], 20.00th=[ 62],
| 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 64], 60.00th=[ 65],
| 70.00th=[ 65], 80.00th=[ 67], 90.00th=[ 68], 95.00th=[ 69],
| 99.00th=[ 72], 99.50th=[ 73], 99.90th=[ 130], 99.95th=[ 265],
| 99.99th=[ 478]
bw ( KiB/s): min= 2808, max=312371, per=0.08%, avg=211893.75, stdev=27432.23
lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.34%, 100=99.54%
lat (msec) : 250=0.06%, 500=0.05%, 750=0.01%, >=2000=0.01%
cpu : usr=0.85%, sys=0.47%, ctx=7224082, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,7219129,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
512kB Seq WR 1job QD32: (groupid=1, jobs=1): err= 0: pid=3047: Mon Jul 15 21:38:51 2019
write: IOPS=509, BW=255MiB/s (267MB/s)(149GiB/600576msec)
slat (usec): min=15, max=112, avg=24.97, stdev= 4.25
clat (msec): min=1, max=685, avg=62.77, stdev=13.77
lat (msec): min=1, max=685, avg=62.79, stdev=13.77
clat percentiles (msec):
| 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 60], 20.00th=[ 61],
| 30.00th=[ 62], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 63],
| 70.00th=[ 64], 80.00th=[ 65], 90.00th=[ 67], 95.00th=[ 67],
| 99.00th=[ 68], 99.50th=[ 70], 99.90th=[ 210], 99.95th=[ 449],
| 99.99th=[ 619]
bw ( KiB/s): min=26514, max=288260, per=0.08%, avg=215169.44, stdev=31749.41
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.58%
lat (msec) : 100=99.24%, 250=0.08%, 500=0.04%, 750=0.04%
cpu : usr=1.00%, sys=0.49%, ctx=306018, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,305797,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
512kB Seq RD 1job QD32: (groupid=2, jobs=1): err= 0: pid=3061: Mon Jul 15 21:38:51 2019
read: IOPS=510, BW=255MiB/s (268MB/s)(150GiB/600060msec)
slat (usec): min=9, max=174, avg=12.45, stdev= 1.66
clat (msec): min=4, max=230, avg=62.63, stdev= 3.67
lat (msec): min=4, max=231, avg=62.64, stdev= 3.67
clat percentiles (msec):
| 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 60], 20.00th=[ 61],
| 30.00th=[ 62], 40.00th=[ 63], 50.00th=[ 63], 60.00th=[ 63],
| 70.00th=[ 64], 80.00th=[ 65], 90.00th=[ 67], 95.00th=[ 67],
| 99.00th=[ 68], 99.50th=[ 69], 99.90th=[ 104], 99.95th=[ 113],
| 99.99th=[ 176]
bw ( KiB/s): min=113777, max=278419, per=0.08%, avg=221064.34, stdev=24439.68
lat (msec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=99.83%, 250=0.15%
cpu : usr=0.14%, sys=0.72%, ctx=306707, majf=0, minf=4106
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=306521,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
128kB Seq WR 1job QD1: (groupid=3, jobs=1): err= 0: pid=3069: Mon Jul 15 21:38:51 2019
write: IOPS=2044, BW=256MiB/s (268MB/s)(150GiB/600007msec)
slat (usec): min=6, max=181, avg= 7.87, stdev= 0.63
clat (usec): min=270, max=99091, avg=480.61, stdev=211.63
lat (usec): min=277, max=99104, avg=488.56, stdev=211.63
clat percentiles (usec):
| 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 430], 20.00th=[ 434],
| 30.00th=[ 434], 40.00th=[ 454], 50.00th=[ 454], 60.00th=[ 454],
| 70.00th=[ 454], 80.00th=[ 474], 90.00th=[ 474], 95.00th=[ 932],
| 99.00th=[ 996], 99.50th=[ 996], 99.90th=[ 2064], 99.95th=[ 2064],
| 99.99th=[ 2608]
bw ( KiB/s): min=133995, max=278940, per=0.08%, avg=212975.21, stdev=27664.20
lat (usec) : 500=94.49%, 750=0.01%, 1000=5.19%
lat (msec) : 2=0.15%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.78%, sys=1.30%, ctx=1226847, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1226623,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
128kB Seq WR 1job QD2: (groupid=4, jobs=1): err= 0: pid=3081: Mon Jul 15 21:38:51 2019
write: IOPS=2045, BW=256MiB/s (268MB/s)(150GiB/600040msec)
slat (usec): min=6, max=145, avg= 7.69, stdev= 0.69
clat (usec): min=501, max=115327, avg=969.57, stdev=331.39
lat (usec): min=509, max=115334, avg=977.33, stdev=331.40
clat percentiles (usec):
| 1.00th=[ 836], 5.00th=[ 852], 10.00th=[ 876], 20.00th=[ 892],
| 30.00th=[ 892], 40.00th=[ 892], 50.00th=[ 916], 60.00th=[ 916],
| 70.00th=[ 932], 80.00th=[ 956], 90.00th=[ 1368], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1496], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 7648]
bw ( KiB/s): min=54280, max=262813, per=0.08%, avg=208796.38, stdev=21653.80
lat (usec) : 750=0.13%, 1000=88.86%
lat (msec) : 2=10.68%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.22%, sys=1.25%, ctx=1227390, majf=0, minf=12
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1227172,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
128kB Seq WR 1job QD4: (groupid=5, jobs=1): err= 0: pid=3091: Mon Jul 15 21:38:51 2019
write: IOPS=2044, BW=256MiB/s (268MB/s)(150GiB/600097msec)
slat (usec): min=6, max=144, avg= 7.69, stdev= 0.68
clat (usec): min=684, max=122483, avg=1948.01, stdev=452.36
lat (usec): min=712, max=122491, avg=1955.77, stdev=452.36
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1704], 10.00th=[ 1736], 20.00th=[ 1800],
| 30.00th=[ 1816], 40.00th=[ 1816], 50.00th=[ 1832], 60.00th=[ 1880],
| 70.00th=[ 1928], 80.00th=[ 2224], 90.00th=[ 2352], 95.00th=[ 2384],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3536], 99.95th=[ 3984],
| 99.99th=[13632]
bw ( KiB/s): min=132171, max=277506, per=0.08%, avg=215787.14, stdev=28511.73
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=77.97%, 4=21.98%, 10=0.02%, 20=0.02%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.23%, sys=1.24%, ctx=1227035, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1226804,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
128kB Seq WR 1job QD8: (groupid=6, jobs=1): err= 0: pid=3107: Mon Jul 15 21:38:51 2019
write: IOPS=2043, BW=255MiB/s (268MB/s)(150GiB/600064msec)
slat (usec): min=5, max=98, avg= 7.71, stdev= 0.71
clat (usec): min=865, max=145853, avg=3906.98, stdev=873.34
lat (usec): min=891, max=145861, avg=3914.76, stdev=873.34
clat percentiles (usec):
| 1.00th=[ 3376], 5.00th=[ 3472], 10.00th=[ 3568], 20.00th=[ 3600],
| 30.00th=[ 3664], 40.00th=[ 3696], 50.00th=[ 3888], 60.00th=[ 3984],
| 70.00th=[ 4128], 80.00th=[ 4192], 90.00th=[ 4320], 95.00th=[ 4384],
| 99.00th=[ 5216], 99.50th=[ 5280], 99.90th=[ 5920], 99.95th=[10816],
| 99.99th=[29568]
bw ( KiB/s): min=82777, max=277462, per=0.08%, avg=220889.09, stdev=25239.59
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=61.06%, 10=38.89%, 20=0.04%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.26%, sys=1.21%, ctx=1226196, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1225966,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
128kB Seq WR 1job QD16: (groupid=7, jobs=1): err= 0: pid=3120: Mon Jul 15 21:38:51 2019
write: IOPS=2044, BW=256MiB/s (268MB/s)(150GiB/600159msec)
slat (usec): min=5, max=117, avg= 8.08, stdev= 0.76
clat (usec): min=944, max=143187, avg=7816.54, stdev=963.65
lat (usec): min=952, max=143196, avg=7824.71, stdev=963.65
clat percentiles (usec):
| 1.00th=[ 6752], 5.00th=[ 7200], 10.00th=[ 7264], 20.00th=[ 7456],
| 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7776], 60.00th=[ 7840],
| 70.00th=[ 7904], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[ 9024], 99.50th=[ 9280], 99.90th=[10560], 99.95th=[16768],
| 99.99th=[43776]
bw ( KiB/s): min=133474, max=278328, per=0.08%, avg=212720.87, stdev=27551.94
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=99.89%, 20=0.08%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.33%, sys=1.23%, ctx=1227041, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1226814,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
128kB Seq WR 1job QD32: (groupid=8, jobs=1): err= 0: pid=3132: Mon Jul 15 21:38:51 2019
write: IOPS=2043, BW=255MiB/s (268MB/s)(150GiB/600187msec)
slat (usec): min=4, max=83, avg= 7.99, stdev= 0.77
clat (msec): min=1, max=143, avg=15.64, stdev= 1.49
lat (msec): min=1, max=143, avg=15.65, stdev= 1.49
clat percentiles (usec):
| 1.00th=[14016], 5.00th=[14528], 10.00th=[14784], 20.00th=[15168],
| 30.00th=[15296], 40.00th=[15424], 50.00th=[15680], 60.00th=[15680],
| 70.00th=[15808], 80.00th=[16192], 90.00th=[16512], 95.00th=[16768],
| 99.00th=[17280], 99.50th=[17536], 99.90th=[24448], 99.95th=[32640],
| 99.99th=[98816]
bw ( KiB/s): min=132431, max=278847, per=0.08%, avg=212791.67, stdev=27587.68
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=99.84%, 50=0.13%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.38%, sys=1.17%, ctx=1226987, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,1226779,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
128kB Seq RD 1job QD1: (groupid=9, jobs=1): err= 0: pid=3143: Mon Jul 15 21:38:51 2019
read: IOPS=2045, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=5, max=62, avg= 5.79, stdev= 0.49
clat (usec): min=257, max=100457, avg=482.54, stdev=213.19
lat (usec): min=263, max=100470, avg=488.41, stdev=213.19
clat percentiles (usec):
| 1.00th=[ 398], 5.00th=[ 414], 10.00th=[ 434], 20.00th=[ 450],
| 30.00th=[ 450], 40.00th=[ 450], 50.00th=[ 450], 60.00th=[ 450],
| 70.00th=[ 462], 80.00th=[ 466], 90.00th=[ 486], 95.00th=[ 932],
| 99.00th=[ 988], 99.50th=[ 1004], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2576]
bw ( KiB/s): min=129564, max=277116, per=0.08%, avg=213208.79, stdev=27787.08
lat (usec) : 500=92.29%, 750=2.20%, 1000=4.86%
lat (msec) : 2=0.49%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.23%, sys=1.41%, ctx=1227375, majf=0, minf=43
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1227165,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
128kB Seq RD 1job QD2: (groupid=10, jobs=1): err= 0: pid=3151: Mon Jul 15 21:38:51 2019
read: IOPS=2046, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=4, max=69, avg= 5.51, stdev= 0.57
clat (usec): min=510, max=34710, avg=971.22, stdev=242.00
lat (usec): min=516, max=34715, avg=976.81, stdev=242.00
clat percentiles (usec):
| 1.00th=[ 820], 5.00th=[ 852], 10.00th=[ 868], 20.00th=[ 892],
| 30.00th=[ 908], 40.00th=[ 908], 50.00th=[ 908], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 956], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1480], 99.50th=[ 1512], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 3088]
bw ( KiB/s): min=137384, max=278680, per=0.08%, avg=209468.09, stdev=22985.19
lat (usec) : 750=0.01%, 1000=88.84%
lat (msec) : 2=10.83%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=0.58%, sys=1.45%, ctx=1228037, majf=0, minf=75
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1227833,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
128kB Seq RD 1job QD4: (groupid=11, jobs=1): err= 0: pid=3160: Mon Jul 15 21:38:51 2019
read: IOPS=2046, BW=256MiB/s (268MB/s)(150GiB/600002msec)
slat (usec): min=4, max=68, avg= 5.52, stdev= 0.58
clat (usec): min=578, max=51787, avg=1948.77, stdev=353.40
lat (usec): min=646, max=51793, avg=1954.37, stdev=353.41
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1720], 10.00th=[ 1752], 20.00th=[ 1784],
| 30.00th=[ 1816], 40.00th=[ 1816], 50.00th=[ 1832], 60.00th=[ 1864],
| 70.00th=[ 1928], 80.00th=[ 2256], 90.00th=[ 2352], 95.00th=[ 2416],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3536], 99.95th=[ 3984],
| 99.99th=[10048]
bw ( KiB/s): min=135560, max=278068, per=0.08%, avg=221476.57, stdev=24142.65
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=77.97%, 4=21.99%, 10=0.02%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.62%, sys=1.41%, ctx=1227905, majf=0, minf=138
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1227700,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
128kB Seq RD 1job QD8: (groupid=12, jobs=1): err= 0: pid=3173: Mon Jul 15 21:38:51 2019
read: IOPS=2045, BW=256MiB/s (268MB/s)(150GiB/600004msec)
slat (usec): min=4, max=72, avg= 5.47, stdev= 0.58
clat (usec): min=652, max=126479, avg=3904.95, stdev=585.00
lat (usec): min=725, max=126485, avg=3910.50, stdev=585.00
clat percentiles (usec):
| 1.00th=[ 3376], 5.00th=[ 3440], 10.00th=[ 3568], 20.00th=[ 3600],
| 30.00th=[ 3664], 40.00th=[ 3696], 50.00th=[ 3856], 60.00th=[ 3984],
| 70.00th=[ 4128], 80.00th=[ 4192], 90.00th=[ 4320], 95.00th=[ 4384],
| 99.00th=[ 5216], 99.50th=[ 5280], 99.90th=[ 5856], 99.95th=[ 5920],
| 99.99th=[17536]
bw ( KiB/s): min=113394, max=276945, per=0.08%, avg=219137.50, stdev=27151.19
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=60.68%, 10=39.29%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.59%, sys=1.44%, ctx=1227523, majf=0, minf=266
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1227311,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
128kB Seq RD 1job QD16: (groupid=13, jobs=1): err= 0: pid=3182: Mon Jul 15 21:38:51 2019
read: IOPS=2046, BW=256MiB/s (268MB/s)(150GiB/600008msec)
slat (usec): min=4, max=71, avg= 5.47, stdev= 0.60
clat (usec): min=1220, max=106243, avg=7813.79, stdev=672.67
lat (usec): min=1291, max=106250, avg=7819.34, stdev=672.67
clat percentiles (usec):
| 1.00th=[ 6752], 5.00th=[ 7200], 10.00th=[ 7264], 20.00th=[ 7456],
| 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7840], 60.00th=[ 7840],
| 70.00th=[ 7904], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[ 9024], 99.50th=[ 9280], 99.90th=[ 9664], 99.95th=[10688],
| 99.99th=[48896]
bw ( KiB/s): min=126435, max=275291, per=0.08%, avg=216581.12, stdev=28394.01
lat (msec) : 2=0.01%, 4=0.01%, 10=99.93%, 20=0.05%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.60%, sys=1.42%, ctx=1227867, majf=0, minf=523
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1227661,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
128kB Seq RD 1job QD32: (groupid=14, jobs=1): err= 0: pid=3190: Mon Jul 15 21:38:51 2019
read: IOPS=2046, BW=256MiB/s (268MB/s)(150GiB/600016msec)
slat (usec): min=3, max=64, avg= 5.43, stdev= 0.61
clat (msec): min=2, max=102, avg=15.63, stdev= 1.04
lat (msec): min=2, max=102, avg=15.64, stdev= 1.04
clat percentiles (usec):
| 1.00th=[14016], 5.00th=[14528], 10.00th=[14784], 20.00th=[15168],
| 30.00th=[15296], 40.00th=[15424], 50.00th=[15680], 60.00th=[15680],
| 70.00th=[15808], 80.00th=[16192], 90.00th=[16512], 95.00th=[16768],
| 99.00th=[17280], 99.50th=[17536], 99.90th=[18560], 99.95th=[23936],
| 99.99th=[58112]
bw ( KiB/s): min=134256, max=276346, per=0.08%, avg=213466.60, stdev=27867.99
lat (msec) : 4=0.01%, 10=0.01%, 20=99.91%, 50=0.06%, 100=0.03%
lat (msec) : 250=0.01%
cpu : usr=0.62%, sys=1.40%, ctx=1227874, majf=0, minf=1036
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=1227695,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
64kB Seq WR 1job QD1: (groupid=15, jobs=1): err= 0: pid=3199: Mon Jul 15 21:38:51 2019
write: IOPS=4087, BW=255MiB/s (268MB/s)(150GiB/600007msec)
slat (usec): min=4, max=163, avg= 5.52, stdev= 0.59
clat (usec): min=147, max=115759, avg=238.48, stdev=158.21
lat (usec): min=153, max=115772, avg=244.09, stdev=158.21
clat percentiles (usec):
| 1.00th=[ 195], 5.00th=[ 213], 10.00th=[ 215], 20.00th=[ 215],
| 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 235],
| 70.00th=[ 235], 80.00th=[ 235], 90.00th=[ 235], 95.00th=[ 255],
| 99.00th=[ 740], 99.50th=[ 756], 99.90th=[ 772], 99.95th=[ 1832],
| 99.99th=[ 1880]
bw ( KiB/s): min=88622, max=276594, per=0.08%, avg=209458.00, stdev=23859.26
lat (usec) : 250=94.88%, 500=2.37%, 750=1.94%, 1000=0.73%
lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=0.98%, sys=2.22%, ctx=2452532, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,2452267,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
64kB Seq WR 1job QD2: (groupid=16, jobs=1): err= 0: pid=3212: Mon Jul 15 21:38:51 2019
write: IOPS=4089, BW=256MiB/s (268MB/s)(150GiB/600060msec)
slat (usec): min=4, max=121, avg= 5.40, stdev= 0.60
clat (usec): min=261, max=87688, avg=483.00, stdev=203.94
lat (usec): min=267, max=87694, avg=488.48, stdev=203.94
clat percentiles (usec):
| 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 434], 20.00th=[ 434],
| 30.00th=[ 434], 40.00th=[ 454], 50.00th=[ 454], 60.00th=[ 454],
| 70.00th=[ 454], 80.00th=[ 474], 90.00th=[ 478], 95.00th=[ 940],
| 99.00th=[ 996], 99.50th=[ 996], 99.90th=[ 2064], 99.95th=[ 2064],
| 99.99th=[ 2608]
bw ( KiB/s): min=128521, max=276816, per=0.08%, avg=221378.65, stdev=24084.15
lat (usec) : 500=94.48%, 750=0.01%, 1000=5.19%
lat (msec) : 2=0.15%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.84%, sys=2.15%, ctx=2454173, majf=0, minf=10
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,2453922,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
64kB Seq WR 1job QD4: (groupid=17, jobs=1): err= 0: pid=3224: Mon Jul 15 21:38:51 2019
write: IOPS=4089, BW=256MiB/s (268MB/s)(150GiB/600211msec)
slat (usec): min=4, max=163, avg= 5.38, stdev= 0.61
clat (usec): min=260, max=66982, avg=971.90, stdev=262.92
lat (usec): min=265, max=66987, avg=977.35, stdev=262.92
clat percentiles (usec):
| 1.00th=[ 836], 5.00th=[ 852], 10.00th=[ 876], 20.00th=[ 892],
| 30.00th=[ 900], 40.00th=[ 900], 50.00th=[ 916], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 956], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1496], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 7712]
bw ( KiB/s): min=131875, max=276073, per=0.08%, avg=213425.43, stdev=27936.99
lat (usec) : 500=0.01%, 750=0.06%, 1000=88.92%
lat (msec) : 2=10.69%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.84%, sys=2.13%, ctx=2454534, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,2454274,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
64kB Seq WR 1job QD8: (groupid=18, jobs=1): err= 0: pid=3240: Mon Jul 15 21:38:51 2019
write: IOPS=4086, BW=255MiB/s (268MB/s)(150GiB/600023msec)
slat (usec): min=4, max=109, avg= 5.44, stdev= 0.62
clat (usec): min=773, max=160880, avg=1951.32, stdev=539.89
lat (usec): min=797, max=160885, avg=1956.83, stdev=539.89
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1704], 10.00th=[ 1736], 20.00th=[ 1800],
| 30.00th=[ 1816], 40.00th=[ 1816], 50.00th=[ 1832], 60.00th=[ 1880],
| 70.00th=[ 1944], 80.00th=[ 2224], 90.00th=[ 2352], 95.00th=[ 2384],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3536], 99.95th=[ 4016],
| 99.99th=[18048]
bw ( KiB/s): min=131519, max=277462, per=0.08%, avg=213340.76, stdev=28075.46
lat (usec) : 1000=0.01%
lat (msec) : 2=77.96%, 4=21.98%, 10=0.03%, 20=0.02%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.91%, sys=2.10%, ctx=2452517, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,2452278,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
64kB Seq WR 1job QD16: (groupid=19, jobs=1): err= 0: pid=3254: Mon Jul 15 21:38:51 2019
write: IOPS=4088, BW=256MiB/s (268MB/s)(150GiB/600106msec)
slat (usec): min=3, max=104, avg= 5.52, stdev= 0.63
clat (usec): min=358, max=73099, avg=3906.64, stdev=554.04
lat (usec): min=382, max=73105, avg=3912.24, stdev=554.04
clat percentiles (usec):
| 1.00th=[ 3376], 5.00th=[ 3472], 10.00th=[ 3568], 20.00th=[ 3600],
| 30.00th=[ 3664], 40.00th=[ 3696], 50.00th=[ 3888], 60.00th=[ 3984],
| 70.00th=[ 4128], 80.00th=[ 4192], 90.00th=[ 4320], 95.00th=[ 4384],
| 99.00th=[ 5216], 99.50th=[ 5344], 99.90th=[ 5920], 99.95th=[ 7008],
| 99.99th=[21888]
bw ( KiB/s): min=118690, max=276552, per=0.08%, avg=209860.58, stdev=24205.99
lat (usec) : 500=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=60.65%, 10=39.30%, 20=0.04%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.94%, sys=2.11%, ctx=2453765, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,2453511,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
64kB Seq WR 1job QD32: (groupid=20, jobs=1): err= 0: pid=3267: Mon Jul 15 21:38:51 2019
write: IOPS=4088, BW=256MiB/s (268MB/s)(150GiB/600059msec)
slat (usec): min=3, max=110, avg= 5.61, stdev= 0.65
clat (usec): min=1289, max=95408, avg=7819.32, stdev=809.51
lat (usec): min=1294, max=95414, avg=7825.02, stdev=809.51
clat percentiles (usec):
| 1.00th=[ 6752], 5.00th=[ 7200], 10.00th=[ 7264], 20.00th=[ 7456],
| 30.00th=[ 7648], 40.00th=[ 7776], 50.00th=[ 7776], 60.00th=[ 7840],
| 70.00th=[ 7904], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[ 9024], 99.50th=[ 9280], 99.90th=[10688], 99.95th=[17280],
| 99.99th=[36096]
bw ( KiB/s): min=133604, max=276985, per=0.08%, avg=216715.20, stdev=28339.87
lat (msec) : 2=0.01%, 4=0.01%, 10=99.87%, 20=0.10%, 50=0.02%
lat (msec) : 100=0.01%
cpu : usr=1.94%, sys=2.16%, ctx=2453764, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,2453527,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
64kB Seq RD 1job QD1: (groupid=21, jobs=1): err= 0: pid=3278: Mon Jul 15 21:38:51 2019
read: IOPS=4091, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=4, max=120, avg= 4.56, stdev= 0.56
clat (usec): min=131, max=106435, avg=239.27, stdev=147.45
lat (usec): min=145, max=106444, avg=243.90, stdev=147.46
clat percentiles (usec):
| 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 199], 20.00th=[ 211],
| 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 233], 60.00th=[ 233],
| 70.00th=[ 235], 80.00th=[ 235], 90.00th=[ 249], 95.00th=[ 249],
| 99.00th=[ 748], 99.50th=[ 764], 99.90th=[ 780], 99.95th=[ 1832],
| 99.99th=[ 1880]
bw ( KiB/s): min=104533, max=272929, per=0.08%, avg=218938.21, stdev=23460.87
lat (usec) : 250=96.58%, 500=0.67%, 750=1.87%, 1000=0.80%
lat (msec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=0.45%, sys=2.32%, ctx=2454854, majf=0, minf=26
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=2454612,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
64kB Seq RD 1job QD2: (groupid=22, jobs=1): err= 0: pid=3287: Mon Jul 15 21:38:51 2019
read: IOPS=4091, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=3, max=48, avg= 4.33, stdev= 0.52
clat (usec): min=304, max=59305, avg=483.89, stdev=193.58
lat (usec): min=308, max=59309, avg=488.30, stdev=193.59
clat percentiles (usec):
| 1.00th=[ 402], 5.00th=[ 414], 10.00th=[ 418], 20.00th=[ 450],
| 30.00th=[ 450], 40.00th=[ 450], 50.00th=[ 450], 60.00th=[ 454],
| 70.00th=[ 458], 80.00th=[ 466], 90.00th=[ 490], 95.00th=[ 940],
| 99.00th=[ 988], 99.50th=[ 1012], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2576]
bw ( KiB/s): min=141164, max=277939, per=0.08%, avg=219584.54, stdev=26532.83
lat (usec) : 500=91.13%, 750=3.36%, 1000=4.86%
lat (msec) : 2=0.48%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.22%, sys=2.33%, ctx=2455170, majf=0, minf=42
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=2454926,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
64kB Seq RD 1job QD4: (groupid=23, jobs=1): err= 0: pid=3296: Mon Jul 15 21:38:51 2019
read: IOPS=4092, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=3, max=53, avg= 4.24, stdev= 0.47
clat (usec): min=533, max=50947, avg=972.60, stdev=253.17
lat (usec): min=537, max=50952, avg=976.92, stdev=253.17
clat percentiles (usec):
| 1.00th=[ 820], 5.00th=[ 860], 10.00th=[ 868], 20.00th=[ 892],
| 30.00th=[ 908], 40.00th=[ 908], 50.00th=[ 908], 60.00th=[ 924],
| 70.00th=[ 940], 80.00th=[ 956], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1480], 99.50th=[ 1512], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 3088]
bw ( KiB/s): min=136602, max=277203, per=0.08%, avg=217231.87, stdev=28161.77
lat (usec) : 750=0.01%, 1000=88.86%
lat (msec) : 2=10.81%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.28%, sys=2.22%, ctx=2455638, majf=0, minf=76
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=2455394,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
64kB Seq RD 1job QD8: (groupid=24, jobs=1): err= 0: pid=3308: Mon Jul 15 21:38:51 2019
read: IOPS=4090, BW=256MiB/s (268MB/s)(150GiB/600002msec)
slat (usec): min=3, max=255, avg= 4.30, stdev= 0.75
clat (usec): min=428, max=124598, avg=1950.62, stdev=430.99
lat (usec): min=435, max=124608, avg=1954.99, stdev=430.99
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1720], 10.00th=[ 1752], 20.00th=[ 1784],
| 30.00th=[ 1816], 40.00th=[ 1816], 50.00th=[ 1832], 60.00th=[ 1864],
| 70.00th=[ 1928], 80.00th=[ 2256], 90.00th=[ 2352], 95.00th=[ 2416],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3536], 99.95th=[ 3984],
| 99.99th=[10176]
bw ( KiB/s): min=128521, max=276088, per=0.08%, avg=214374.55, stdev=28339.21
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=77.96%, 4=21.99%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=1.24%, sys=2.29%, ctx=2454817, majf=0, minf=139
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=2454602,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
64kB Seq RD 1job QD16: (groupid=25, jobs=1): err= 0: pid=3315: Mon Jul 15 21:38:51 2019
read: IOPS=4092, BW=256MiB/s (268MB/s)(150GiB/600004msec)
slat (usec): min=3, max=54, avg= 4.23, stdev= 0.48
clat (usec): min=749, max=46089, avg=3904.71, stdev=466.96
lat (usec): min=804, max=46094, avg=3909.01, stdev=466.96
clat percentiles (usec):
| 1.00th=[ 3376], 5.00th=[ 3472], 10.00th=[ 3568], 20.00th=[ 3600],
| 30.00th=[ 3632], 40.00th=[ 3696], 50.00th=[ 3856], 60.00th=[ 3984],
| 70.00th=[ 4128], 80.00th=[ 4192], 90.00th=[ 4320], 95.00th=[ 4384],
| 99.00th=[ 5216], 99.50th=[ 5280], 99.90th=[ 5856], 99.95th=[ 5920],
| 99.99th=[12480]
bw ( KiB/s): min=133474, max=279154, per=0.08%, avg=211161.74, stdev=25790.01
lat (usec) : 750=0.01%
lat (msec) : 2=0.01%, 4=60.51%, 10=39.47%, 20=0.01%, 50=0.01%
cpu : usr=1.23%, sys=2.27%, ctx=2455798, majf=0, minf=266
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=2455556,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
64kB Seq RD 1job QD32: (groupid=26, jobs=1): err= 0: pid=3324: Mon Jul 15 21:38:51 2019
read: IOPS=4091, BW=256MiB/s (268MB/s)(150GiB/600008msec)
slat (usec): min=2, max=50, avg= 4.18, stdev= 0.44
clat (usec): min=454, max=92384, avg=7815.82, stdev=719.42
lat (usec): min=461, max=92389, avg=7820.08, stdev=719.43
clat percentiles (usec):
| 1.00th=[ 6752], 5.00th=[ 7200], 10.00th=[ 7264], 20.00th=[ 7520],
| 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7840], 60.00th=[ 7840],
| 70.00th=[ 7904], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
| 99.00th=[ 9024], 99.50th=[ 9280], 99.90th=[ 9664], 99.95th=[10688],
| 99.99th=[49408]
bw ( KiB/s): min=138337, max=265922, per=0.08%, avg=214318.61, stdev=22332.48
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=99.92%, 20=0.05%, 50=0.02%
lat (msec) : 100=0.01%
cpu : usr=1.18%, sys=2.29%, ctx=2455290, majf=0, minf=522
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=2455085,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
32kB Seq WR 1job QD1: (groupid=27, jobs=1): err= 0: pid=3332: Mon Jul 15 21:38:51 2019
write: IOPS=8157, BW=255MiB/s (267MB/s)(149GiB/600007msec)
slat (usec): min=4, max=92, avg= 4.48, stdev= 0.53
clat (usec): min=72, max=93575, avg=117.43, stdev=124.00
lat (usec): min=90, max=93584, avg=121.99, stdev=124.00
clat percentiles (usec):
| 1.00th=[ 94], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95],
| 30.00th=[ 113], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 115],
| 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 116], 95.00th=[ 116],
| 99.00th=[ 620], 99.50th=[ 636], 99.90th=[ 636], 99.95th=[ 660],
| 99.99th=[ 1736]
bw ( KiB/s): min=116780, max=277614, per=0.08%, avg=219675.92, stdev=26025.82
lat (usec) : 100=29.46%, 250=69.15%, 500=0.01%, 750=1.33%, 1000=0.01%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.43%, sys=4.10%, ctx=4894911, majf=0, minf=12
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4894638,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
32kB Seq WR 1job QD2: (groupid=28, jobs=1): err= 0: pid=3345: Mon Jul 15 21:38:51 2019
write: IOPS=8176, BW=256MiB/s (268MB/s)(150GiB/600037msec)
slat (usec): min=3, max=201, avg= 4.26, stdev= 0.52
clat (usec): min=25, max=86860, avg=239.72, stdev=139.43
lat (usec): min=118, max=86864, avg=244.05, stdev=139.43
clat percentiles (usec):
| 1.00th=[ 195], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215],
| 30.00th=[ 217], 40.00th=[ 217], 50.00th=[ 217], 60.00th=[ 235],
| 70.00th=[ 235], 80.00th=[ 237], 90.00th=[ 237], 95.00th=[ 255],
| 99.00th=[ 740], 99.50th=[ 756], 99.90th=[ 780], 99.95th=[ 1832],
| 99.99th=[ 1880]
bw ( KiB/s): min=128977, max=277441, per=0.08%, avg=217428.51, stdev=28033.54
lat (usec) : 50=0.01%, 250=94.86%, 500=2.38%, 750=1.94%, 1000=0.73%
lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.09%, sys=3.90%, ctx=4906232, majf=0, minf=12
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4905968,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
32kB Seq WR 1job QD4: (groupid=29, jobs=1): err= 0: pid=3357: Mon Jul 15 21:38:51 2019
write: IOPS=8177, BW=256MiB/s (268MB/s)(150GiB/600054msec)
slat (usec): min=3, max=124, avg= 4.25, stdev= 0.53
clat (usec): min=274, max=104027, avg=484.21, stdev=198.67
lat (usec): min=280, max=104031, avg=488.53, stdev=198.67
clat percentiles (usec):
| 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 434], 20.00th=[ 438],
| 30.00th=[ 438], 40.00th=[ 458], 50.00th=[ 458], 60.00th=[ 458],
| 70.00th=[ 458], 80.00th=[ 478], 90.00th=[ 478], 95.00th=[ 940],
| 99.00th=[ 996], 99.50th=[ 996], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2608]
bw ( KiB/s): min=160343, max=265338, per=0.08%, avg=213321.56, stdev=22207.03
lat (usec) : 500=94.47%, 750=0.02%, 1000=5.20%
lat (msec) : 2=0.15%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=3.01%, sys=3.96%, ctx=4907463, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4907200,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
32kB Seq WR 1job QD8: (groupid=30, jobs=1): err= 0: pid=3427: Mon Jul 15 21:38:51 2019
write: IOPS=8174, BW=255MiB/s (268MB/s)(150GiB/600039msec)
slat (usec): min=3, max=129, avg= 4.26, stdev= 0.53
clat (usec): min=554, max=114499, avg=973.69, stdev=318.75
lat (usec): min=561, max=114508, avg=978.03, stdev=318.75
clat percentiles (usec):
| 1.00th=[ 836], 5.00th=[ 860], 10.00th=[ 876], 20.00th=[ 900],
| 30.00th=[ 900], 40.00th=[ 900], 50.00th=[ 916], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 956], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1496], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 9280]
bw ( KiB/s): min=133037, max=275797, per=0.08%, avg=214998.76, stdev=28468.94
lat (usec) : 750=0.03%, 1000=88.92%
lat (msec) : 2=10.70%, 4=0.33%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=3.06%, sys=3.93%, ctx=4905412, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4905150,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
32kB Seq WR 1job QD16: (groupid=31, jobs=1): err= 0: pid=3440: Mon Jul 15 21:38:51 2019
write: IOPS=8176, BW=256MiB/s (268MB/s)(150GiB/600046msec)
slat (usec): min=2, max=156, avg= 4.29, stdev= 0.53
clat (usec): min=816, max=80993, avg=1951.84, stdev=397.37
lat (usec): min=821, max=80998, avg=1956.20, stdev=397.37
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1704], 10.00th=[ 1736], 20.00th=[ 1800],
| 30.00th=[ 1816], 40.00th=[ 1816], 50.00th=[ 1832], 60.00th=[ 1880],
| 70.00th=[ 1944], 80.00th=[ 2256], 90.00th=[ 2352], 95.00th=[ 2384],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3568], 99.95th=[ 4016],
| 99.99th=[14016]
bw ( KiB/s): min=133474, max=275056, per=0.08%, avg=220490.12, stdev=25423.37
lat (usec) : 1000=0.01%
lat (msec) : 2=77.94%, 4=22.00%, 10=0.04%, 20=0.02%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.03%, sys=3.97%, ctx=4906396, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4906130,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
32kB Seq WR 1job QD32: (groupid=32, jobs=1): err= 0: pid=3453: Mon Jul 15 21:38:51 2019
write: IOPS=8175, BW=255MiB/s (268MB/s)(150GiB/600032msec)
slat (usec): min=2, max=268, avg= 4.34, stdev= 0.58
clat (usec): min=1127, max=66493, avg=3908.77, stdev=514.86
lat (usec): min=1132, max=66498, avg=3913.18, stdev=514.86
clat percentiles (usec):
| 1.00th=[ 3376], 5.00th=[ 3472], 10.00th=[ 3568], 20.00th=[ 3600],
| 30.00th=[ 3664], 40.00th=[ 3696], 50.00th=[ 3888], 60.00th=[ 3984],
| 70.00th=[ 4128], 80.00th=[ 4192], 90.00th=[ 4320], 95.00th=[ 4384],
| 99.00th=[ 5216], 99.50th=[ 5344], 99.90th=[ 5984], 99.95th=[ 9024],
| 99.99th=[18304]
bw ( KiB/s): min=144423, max=275855, per=0.08%, avg=212019.73, stdev=26858.23
lat (msec) : 2=0.01%, 4=60.60%, 10=39.36%, 20=0.04%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.06%, sys=3.98%, ctx=4906101, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,4905853,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
32kB Seq RD 1job QD1: (groupid=33, jobs=1): err= 0: pid=3465: Mon Jul 15 21:38:51 2019
read: IOPS=8183, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=3, max=59, avg= 3.91, stdev= 0.35
clat (usec): min=62, max=88697, avg=117.67, stdev=96.53
lat (usec): min=87, max=88705, avg=121.65, stdev=96.53
clat percentiles (usec):
| 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 97],
| 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 111], 60.00th=[ 111],
| 70.00th=[ 112], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 135],
| 99.00th=[ 620], 99.50th=[ 628], 99.90th=[ 660], 99.95th=[ 660],
| 99.99th=[ 1736]
bw ( KiB/s): min=134321, max=276268, per=0.08%, avg=218497.62, stdev=27581.31
lat (usec) : 100=47.16%, 250=51.46%, 500=0.01%, 750=1.34%, 1000=0.01%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.94%, sys=4.08%, ctx=4910403, majf=0, minf=20
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=4910132,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
32kB Seq RD 1job QD2: (groupid=34, jobs=1): err= 0: pid=3475: Mon Jul 15 21:38:51 2019
read: IOPS=8183, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=141, avg= 3.68, stdev= 0.51
clat (usec): min=83, max=58565, avg=240.11, stdev=136.83
lat (usec): min=139, max=58569, avg=243.87, stdev=136.83
clat percentiles (usec):
| 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 201], 20.00th=[ 213],
| 30.00th=[ 213], 40.00th=[ 215], 50.00th=[ 233], 60.00th=[ 235],
| 70.00th=[ 239], 80.00th=[ 241], 90.00th=[ 249], 95.00th=[ 253],
| 99.00th=[ 748], 99.50th=[ 764], 99.90th=[ 780], 99.95th=[ 1832],
| 99.99th=[ 1880]
bw ( KiB/s): min=125393, max=277225, per=0.08%, avg=215641.04, stdev=28418.01
lat (usec) : 100=0.01%, 250=91.24%, 500=6.00%, 750=1.85%, 1000=0.82%
lat (msec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.40%, sys=4.12%, ctx=4910150, majf=0, minf=26
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=4909871,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
32kB Seq RD 1job QD4: (groupid=35, jobs=1): err= 0: pid=3483: Mon Jul 15 21:38:51 2019
read: IOPS=8184, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=137, avg= 3.68, stdev= 0.51
clat (usec): min=260, max=50629, avg=484.45, stdev=183.74
lat (usec): min=264, max=50633, avg=488.21, stdev=183.74
clat percentiles (usec):
| 1.00th=[ 406], 5.00th=[ 414], 10.00th=[ 418], 20.00th=[ 442],
| 30.00th=[ 450], 40.00th=[ 454], 50.00th=[ 454], 60.00th=[ 458],
| 70.00th=[ 458], 80.00th=[ 470], 90.00th=[ 494], 95.00th=[ 940],
| 99.00th=[ 988], 99.50th=[ 1004], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2576]
bw ( KiB/s): min=135820, max=275627, per=0.08%, avg=212777.14, stdev=27323.20
lat (usec) : 500=92.08%, 750=2.41%, 1000=4.81%
lat (msec) : 2=0.53%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.47%, sys=4.05%, ctx=4910922, majf=0, minf=42
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=4910654,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
32kB Seq RD 1job QD8: (groupid=36, jobs=1): err= 0: pid=3494: Mon Jul 15 21:38:51 2019
read: IOPS=8182, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=48, avg= 3.68, stdev= 0.50
clat (usec): min=543, max=90455, avg=973.44, stdev=288.33
lat (usec): min=547, max=90465, avg=977.20, stdev=288.34
clat percentiles (usec):
| 1.00th=[ 828], 5.00th=[ 860], 10.00th=[ 868], 20.00th=[ 884],
| 30.00th=[ 900], 40.00th=[ 908], 50.00th=[ 916], 60.00th=[ 916],
| 70.00th=[ 948], 80.00th=[ 964], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1512], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 3088]
bw ( KiB/s): min=108192, max=278004, per=0.08%, avg=209229.50, stdev=22818.21
lat (usec) : 750=0.01%, 1000=88.09%
lat (msec) : 2=11.57%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.44%, sys=4.07%, ctx=4909649, majf=0, minf=74
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=4909395,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
32kB Seq RD 1job QD16: (groupid=37, jobs=1): err= 0: pid=3502: Mon Jul 15 21:38:51 2019
read: IOPS=8185, BW=256MiB/s (268MB/s)(150GiB/600002msec)
slat (usec): min=2, max=134, avg= 3.72, stdev= 0.49
clat (usec): min=717, max=49948, avg=1950.28, stdev=331.89
lat (usec): min=751, max=49957, avg=1954.09, stdev=331.89
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1720], 10.00th=[ 1752], 20.00th=[ 1784],
| 30.00th=[ 1816], 40.00th=[ 1832], 50.00th=[ 1832], 60.00th=[ 1880],
| 70.00th=[ 1928], 80.00th=[ 2256], 90.00th=[ 2352], 95.00th=[ 2416],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3536], 99.95th=[ 3984],
| 99.99th=[ 9024]
bw ( KiB/s): min=134386, max=276816, per=0.08%, avg=221612.74, stdev=24004.50
lat (usec) : 750=0.01%
lat (msec) : 2=77.97%, 4=22.00%, 10=0.03%, 20=0.01%, 50=0.01%
cpu : usr=2.46%, sys=4.10%, ctx=4911755, majf=0, minf=138
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=4911483,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
32kB Seq RD 1job QD32: (groupid=38, jobs=1): err= 0: pid=3511: Mon Jul 15 21:38:51 2019
read: IOPS=8183, BW=256MiB/s (268MB/s)(150GiB/600004msec)
slat (usec): min=1, max=45, avg= 3.70, stdev= 0.49
clat (usec): min=419, max=93344, avg=3905.87, stdev=540.43
lat (usec): min=423, max=93353, avg=3909.65, stdev=540.43
clat percentiles (usec):
| 1.00th=[ 3376], 5.00th=[ 3472], 10.00th=[ 3568], 20.00th=[ 3600],
| 30.00th=[ 3664], 40.00th=[ 3728], 50.00th=[ 3856], 60.00th=[ 3984],
| 70.00th=[ 4128], 80.00th=[ 4192], 90.00th=[ 4320], 95.00th=[ 4384],
| 99.00th=[ 5216], 99.50th=[ 5280], 99.90th=[ 5856], 99.95th=[ 5920],
| 99.99th=[12736]
bw ( KiB/s): min=130802, max=278875, per=0.08%, avg=219507.29, stdev=26726.54
lat (usec) : 500=0.01%, 750=0.01%
lat (msec) : 2=0.01%, 4=60.72%, 10=39.25%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.52%, sys=4.01%, ctx=4910525, majf=0, minf=267
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=4910306,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
16kB Seq WR 1job QD1: (groupid=39, jobs=1): err= 0: pid=3520: Mon Jul 15 21:38:51 2019
write: IOPS=15.7k, BW=245MiB/s (257MB/s)(144GiB/600013msec)
slat (usec): min=3, max=124, avg= 3.88, stdev= 0.41
clat (usec): min=8, max=96324, avg=59.24, stdev=78.33
lat (usec): min=59, max=96333, avg=63.21, stdev=78.33
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56],
| 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 57],
| 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 57], 95.00th=[ 57],
| 99.00th=[ 123], 99.50th=[ 151], 99.90th=[ 498], 99.95th=[ 556],
| 99.99th=[ 2640]
bw ( KiB/s): min=130639, max=259562, per=0.08%, avg=208053.31, stdev=26816.50
lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=98.76%, 250=1.09%
lat (usec) : 500=0.05%, 750=0.06%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.29%, sys=7.41%, ctx=9407110, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,9406808,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
16kB Seq WR 1job QD2: (groupid=40, jobs=1): err= 0: pid=3532: Mon Jul 15 21:38:51 2019
write: IOPS=16.2k, BW=253MiB/s (265MB/s)(148GiB/600831msec)
slat (usec): min=3, max=228, avg= 3.70, stdev= 0.54
clat (usec): min=6, max=91778, avg=119.00, stdev=158.95
lat (usec): min=59, max=91782, avg=122.79, stdev=158.95
clat percentiles (usec):
| 1.00th=[ 95], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 99],
| 30.00th=[ 111], 40.00th=[ 113], 50.00th=[ 115], 60.00th=[ 116],
| 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116],
| 99.00th=[ 620], 99.50th=[ 636], 99.90th=[ 636], 99.95th=[ 884],
| 99.99th=[ 1752]
bw ( KiB/s): min=133507, max=278991, per=0.08%, avg=212194.37, stdev=28298.31
lat (usec) : 10=0.01%, 50=0.01%, 100=24.92%, 250=73.51%, 500=0.17%
lat (usec) : 750=1.34%, 1000=0.01%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=5.64%, sys=7.24%, ctx=9730042, majf=0, minf=10
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,9729770,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
16kB Seq WR 1job QD4: (groupid=41, jobs=1): err= 0: pid=3543: Mon Jul 15 21:38:51 2019
write: IOPS=16.3k, BW=255MiB/s (268MB/s)(150GiB/600159msec)
slat (usec): min=2, max=110, avg= 3.66, stdev= 0.54
clat (usec): min=53, max=47724, avg=240.47, stdev=139.80
lat (usec): min=61, max=47728, avg=244.20, stdev=139.80
clat percentiles (usec):
| 1.00th=[ 197], 5.00th=[ 215], 10.00th=[ 217], 20.00th=[ 217],
| 30.00th=[ 217], 40.00th=[ 217], 50.00th=[ 217], 60.00th=[ 237],
| 70.00th=[ 237], 80.00th=[ 237], 90.00th=[ 237], 95.00th=[ 258],
| 99.00th=[ 740], 99.50th=[ 756], 99.90th=[ 780], 99.95th=[ 1832],
| 99.99th=[ 2352]
bw ( KiB/s): min=129205, max=277279, per=0.08%, avg=217174.41, stdev=28244.37
lat (usec) : 100=0.01%, 250=94.87%, 500=2.35%, 750=1.96%, 1000=0.73%
lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=5.46%, sys=7.45%, ctx=9806381, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,9806207,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
16kB Seq WR 1job QD8: (groupid=42, jobs=1): err= 0: pid=3559: Mon Jul 15 21:38:51 2019
write: IOPS=16.3k, BW=255MiB/s (268MB/s)(150GiB/600025msec)
slat (usec): min=2, max=91, avg= 3.69, stdev= 0.52
clat (usec): min=310, max=114371, avg=485.03, stdev=216.56
lat (usec): min=314, max=114380, avg=488.80, stdev=216.57
clat percentiles (usec):
| 1.00th=[ 418], 5.00th=[ 418], 10.00th=[ 438], 20.00th=[ 438],
| 30.00th=[ 438], 40.00th=[ 458], 50.00th=[ 458], 60.00th=[ 458],
| 70.00th=[ 458], 80.00th=[ 478], 90.00th=[ 478], 95.00th=[ 940],
| 99.00th=[ 996], 99.50th=[ 996], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2608]
bw ( KiB/s): min=128905, max=274141, per=0.08%, avg=217415.57, stdev=28111.73
lat (usec) : 500=94.47%, 750=0.01%, 1000=5.19%
lat (msec) : 2=0.16%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=5.64%, sys=7.40%, ctx=9809159, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,9808965,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
16kB Seq WR 1job QD16: (groupid=43, jobs=1): err= 0: pid=3572: Mon Jul 15 21:38:51 2019
write: IOPS=16.3k, BW=255MiB/s (268MB/s)(150GiB/600054msec)
slat (usec): min=2, max=144, avg= 3.67, stdev= 0.54
clat (usec): min=607, max=113636, avg=974.72, stdev=300.63
lat (usec): min=611, max=113640, avg=978.47, stdev=300.63
clat percentiles (usec):
| 1.00th=[ 836], 5.00th=[ 860], 10.00th=[ 876], 20.00th=[ 900],
| 30.00th=[ 900], 40.00th=[ 900], 50.00th=[ 916], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 956], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1496], 99.90th=[ 2576], 99.95th=[ 3056],
| 99.99th=[ 9792]
bw ( KiB/s): min=135918, max=275927, per=0.08%, avg=214689.37, stdev=28464.03
lat (usec) : 750=0.05%, 1000=88.90%
lat (msec) : 2=10.68%, 4=0.35%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=5.61%, sys=7.30%, ctx=9806051, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,9805864,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
16kB Seq WR 1job QD32: (groupid=44, jobs=1): err= 0: pid=3584: Mon Jul 15 21:38:51 2019
write: IOPS=16.1k, BW=252MiB/s (264MB/s)(148GiB/600521msec)
slat (usec): min=2, max=163, avg= 3.70, stdev= 0.54
clat (msec): min=1, max=781, avg= 1.98, stdev= 1.78
lat (msec): min=1, max=781, avg= 1.98, stdev= 1.78
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1704], 10.00th=[ 1736], 20.00th=[ 1800],
| 30.00th=[ 1816], 40.00th=[ 1816], 50.00th=[ 1832], 60.00th=[ 1880],
| 70.00th=[ 1944], 80.00th=[ 2256], 90.00th=[ 2352], 95.00th=[ 2416],
| 99.00th=[ 3376], 99.50th=[ 3536], 99.90th=[ 9024], 99.95th=[16768],
| 99.99th=[24704]
bw ( KiB/s): min= 1740, max=279690, per=0.08%, avg=217805.80, stdev=27091.00
lat (msec) : 2=77.71%, 4=21.89%, 10=0.31%, 20=0.06%, 50=0.03%
lat (msec) : 500=0.01%, 1000=0.01%
cpu : usr=5.47%, sys=7.39%, ctx=9686583, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,9686446,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
16kB Seq RD 1job QD1: (groupid=45, jobs=1): err= 0: pid=3594: Mon Jul 15 21:38:51 2019
read: IOPS=16.3k, BW=254MiB/s (267MB/s)(149GiB/600001msec)
slat (usec): min=3, max=134, avg= 3.51, stdev= 0.53
clat (usec): min=9, max=81055, avg=57.30, stdev=58.36
lat (usec): min=57, max=81064, avg=60.89, stdev=58.36
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54],
| 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 54],
| 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 56], 95.00th=[ 58],
| 99.00th=[ 95], 99.50th=[ 139], 99.90th=[ 564], 99.95th=[ 572],
| 99.99th=[ 1640]
bw ( KiB/s): min=135494, max=272065, per=0.08%, avg=217179.45, stdev=27280.17
lat (usec) : 10=0.01%, 20=0.01%, 50=0.01%, 100=99.08%, 250=0.49%
lat (usec) : 500=0.20%, 750=0.20%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.87%, sys=7.49%, ctx=9767018, majf=0, minf=14
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=9766702,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
16kB Seq RD 1job QD2: (groupid=46, jobs=1): err= 0: pid=3605: Mon Jul 15 21:38:51 2019
read: IOPS=16.4k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=159, avg= 3.34, stdev= 0.51
clat (usec): min=33, max=33961, avg=118.21, stdev=86.46
lat (usec): min=57, max=33964, avg=121.62, stdev=86.47
clat percentiles (usec):
| 1.00th=[ 96], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97],
| 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 112],
| 70.00th=[ 113], 80.00th=[ 133], 90.00th=[ 137], 95.00th=[ 137],
| 99.00th=[ 620], 99.50th=[ 636], 99.90th=[ 652], 99.95th=[ 660],
| 99.99th=[ 1736]
bw ( KiB/s): min=136830, max=278166, per=0.08%, avg=215749.88, stdev=28414.89
lat (usec) : 50=0.01%, 100=33.90%, 250=64.72%, 500=0.01%, 750=1.34%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=5.06%, sys=7.31%, ctx=9823200, majf=0, minf=19
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=9822944,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
16kB Seq RD 1job QD4: (groupid=47, jobs=1): err= 0: pid=3614: Mon Jul 15 21:38:51 2019
read: IOPS=16.4k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=66, avg= 3.34, stdev= 0.50
clat (usec): min=100, max=58651, avg=240.52, stdev=144.27
lat (usec): min=105, max=58654, avg=243.93, stdev=144.27
clat percentiles (usec):
| 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 205], 20.00th=[ 213],
| 30.00th=[ 215], 40.00th=[ 217], 50.00th=[ 221], 60.00th=[ 235],
| 70.00th=[ 237], 80.00th=[ 241], 90.00th=[ 249], 95.00th=[ 253],
| 99.00th=[ 748], 99.50th=[ 764], 99.90th=[ 780], 99.95th=[ 1832],
| 99.99th=[ 1880]
bw ( KiB/s): min=132203, max=276825, per=0.08%, avg=212700.71, stdev=27285.58
lat (usec) : 250=90.26%, 500=6.98%, 750=1.87%, 1000=0.80%
lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=4.94%, sys=7.43%, ctx=9818082, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=9817831,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
16kB Seq RD 1job QD8: (groupid=48, jobs=1): err= 0: pid=3625: Mon Jul 15 21:38:51 2019
read: IOPS=16.4k, BW=256MiB/s (268MB/s)(150GiB/600029msec)
slat (usec): min=2, max=61, avg= 3.37, stdev= 0.51
clat (usec): min=224, max=104907, avg=484.91, stdev=207.84
lat (usec): min=228, max=104915, avg=488.36, stdev=207.84
clat percentiles (usec):
| 1.00th=[ 406], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 442],
| 30.00th=[ 450], 40.00th=[ 454], 50.00th=[ 454], 60.00th=[ 458],
| 70.00th=[ 462], 80.00th=[ 470], 90.00th=[ 494], 95.00th=[ 940],
| 99.00th=[ 988], 99.50th=[ 1004], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2576]
bw ( KiB/s): min=105353, max=277906, per=0.08%, avg=209280.06, stdev=22956.82
lat (usec) : 250=0.01%, 500=92.26%, 750=2.23%, 1000=4.83%
lat (msec) : 2=0.52%, 4=0.16%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=5.06%, sys=7.40%, ctx=9818905, majf=0, minf=43
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=9818630,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
16kB Seq RD 1job QD16: (groupid=49, jobs=1): err= 0: pid=3634: Mon Jul 15 21:38:51 2019
read: IOPS=16.4k, BW=256MiB/s (268MB/s)(150GiB/600035msec)
slat (usec): min=2, max=33, avg= 3.42, stdev= 0.51
clat (usec): min=580, max=47930, avg=973.34, stdev=245.05
lat (usec): min=587, max=47938, avg=976.84, stdev=245.05
clat percentiles (usec):
| 1.00th=[ 820], 5.00th=[ 860], 10.00th=[ 868], 20.00th=[ 892],
| 30.00th=[ 908], 40.00th=[ 908], 50.00th=[ 908], 60.00th=[ 916],
| 70.00th=[ 948], 80.00th=[ 964], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1512], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 3088]
bw ( KiB/s): min=131519, max=275382, per=0.08%, avg=216680.28, stdev=28333.25
lat (usec) : 750=0.01%, 1000=88.74%
lat (msec) : 2=10.93%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=5.03%, sys=7.52%, ctx=9823020, majf=0, minf=74
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=9822733,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
16kB Seq RD 1job QD32: (groupid=50, jobs=1): err= 0: pid=3644: Mon Jul 15 21:38:51 2019
read: IOPS=16.4k, BW=256MiB/s (268MB/s)(150GiB/600002msec)
slat (usec): min=1, max=96, avg= 3.35, stdev= 0.50
clat (usec): min=1172, max=49217, avg=1950.96, stdev=358.52
lat (usec): min=1176, max=49220, avg=1954.38, stdev=358.52
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1704], 10.00th=[ 1752], 20.00th=[ 1784],
| 30.00th=[ 1800], 40.00th=[ 1832], 50.00th=[ 1832], 60.00th=[ 1880],
| 70.00th=[ 1928], 80.00th=[ 2256], 90.00th=[ 2352], 95.00th=[ 2416],
| 99.00th=[ 2480], 99.50th=[ 3408], 99.90th=[ 3536], 99.95th=[ 3984],
| 99.99th=[ 6880]
bw ( KiB/s): min=134321, max=275630, per=0.08%, avg=213477.51, stdev=27856.76
lat (msec) : 2=77.97%, 4=22.00%, 10=0.03%, 20=0.01%, 50=0.01%
cpu : usr=5.11%, sys=7.33%, ctx=9821632, majf=0, minf=138
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=9821413,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
8kB Seq WR 1job QD1: (groupid=51, jobs=1): err= 0: pid=3654: Mon Jul 15 21:38:51 2019
write: IOPS=21.3k, BW=166MiB/s (174MB/s)(97.4GiB/600004msec)
slat (usec): min=3, max=47, avg= 3.68, stdev= 0.49
clat (usec): min=15, max=95185, avg=42.64, stdev=43.13
lat (usec): min=44, max=95194, avg=46.39, stdev=43.13
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41],
| 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 42],
| 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 42], 95.00th=[ 43],
| 99.00th=[ 44], 99.50th=[ 125], 99.90th=[ 159], 99.95th=[ 175],
| 99.99th=[ 207]
bw ( KiB/s): min=69899, max=173821, per=0.08%, avg=136724.71, stdev=16095.82
lat (usec) : 20=0.01%, 50=99.18%, 100=0.14%, 250=0.68%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.83%, sys=9.95%, ctx=12763666, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,12763371,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
8kB Seq WR 1job QD2: (groupid=52, jobs=1): err= 0: pid=3668: Mon Jul 15 21:38:51 2019
write: IOPS=32.7k, BW=255MiB/s (268MB/s)(150GiB/600014msec)
slat (usec): min=2, max=136, avg= 3.44, stdev= 0.55
clat (usec): min=14, max=25827, avg=57.17, stdev=56.28
lat (usec): min=43, max=25831, avg=60.68, stdev=56.28
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52],
| 30.00th=[ 52], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 53],
| 70.00th=[ 55], 80.00th=[ 56], 90.00th=[ 56], 95.00th=[ 57],
| 99.00th=[ 74], 99.50th=[ 494], 99.90th=[ 580], 99.95th=[ 580],
| 99.99th=[ 1672]
bw ( KiB/s): min=153007, max=271694, per=0.08%, avg=218703.79, stdev=23265.39
lat (usec) : 20=0.01%, 50=0.15%, 100=98.98%, 250=0.27%, 500=0.10%
lat (usec) : 750=0.48%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=10.61%, sys=14.45%, ctx=19595878, majf=0, minf=13
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,19595820,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
8kB Seq WR 1job QD4: (groupid=53, jobs=1): err= 0: pid=3681: Mon Jul 15 21:38:51 2019
write: IOPS=32.6k, BW=255MiB/s (267MB/s)(149GiB/600044msec)
slat (usec): min=2, max=306, avg= 3.44, stdev= 0.56
clat (usec): min=23, max=222301, avg=118.53, stdev=152.04
lat (usec): min=44, max=222304, avg=122.05, stdev=152.04
clat percentiles (usec):
| 1.00th=[ 98], 5.00th=[ 100], 10.00th=[ 101], 20.00th=[ 105],
| 30.00th=[ 106], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 112],
| 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 117],
| 99.00th=[ 620], 99.50th=[ 636], 99.90th=[ 644], 99.95th=[ 652],
| 99.99th=[ 1736]
bw ( KiB/s): min=77427, max=276569, per=0.08%, avg=219216.91, stdev=26458.57
lat (usec) : 50=0.01%, 100=1.05%, 250=97.54%, 500=0.01%, 750=1.35%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=10.62%, sys=14.48%, ctx=19572714, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,19573450,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
8kB Seq WR 1job QD8: (groupid=54, jobs=1): err= 0: pid=3698: Mon Jul 15 21:38:51 2019
write: IOPS=32.7k, BW=255MiB/s (268MB/s)(150GiB/600007msec)
slat (usec): min=2, max=154, avg= 3.44, stdev= 0.55
clat (usec): min=41, max=114258, avg=240.70, stdev=158.96
lat (usec): min=59, max=114267, avg=244.23, stdev=158.96
clat percentiles (usec):
| 1.00th=[ 201], 5.00th=[ 209], 10.00th=[ 217], 20.00th=[ 217],
| 30.00th=[ 221], 40.00th=[ 221], 50.00th=[ 223], 60.00th=[ 227],
| 70.00th=[ 233], 80.00th=[ 237], 90.00th=[ 237], 95.00th=[ 251],
| 99.00th=[ 748], 99.50th=[ 756], 99.90th=[ 772], 99.95th=[ 1848],
| 99.99th=[ 2352]
bw ( KiB/s): min=131942, max=275062, per=0.08%, avg=210891.97, stdev=26016.94
lat (usec) : 50=0.01%, 100=0.01%, 250=94.94%, 500=2.29%, 750=1.99%
lat (usec) : 1000=0.68%
lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=10.66%, sys=14.50%, ctx=19607835, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,19608592,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
8kB Seq WR 1job QD16: (groupid=55, jobs=1): err= 0: pid=3712: Mon Jul 15 21:38:51 2019
write: IOPS=32.7k, BW=256MiB/s (268MB/s)(150GiB/600030msec)
slat (usec): min=2, max=149, avg= 3.43, stdev= 0.55
clat (usec): min=86, max=33470, avg=485.14, stdev=178.56
lat (usec): min=120, max=33474, avg=488.65, stdev=178.56
clat percentiles (usec):
| 1.00th=[ 418], 5.00th=[ 422], 10.00th=[ 434], 20.00th=[ 442],
| 30.00th=[ 446], 40.00th=[ 454], 50.00th=[ 458], 60.00th=[ 458],
| 70.00th=[ 462], 80.00th=[ 478], 90.00th=[ 482], 95.00th=[ 940],
| 99.00th=[ 996], 99.50th=[ 996], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 3536]
bw ( KiB/s): min=162387, max=261842, per=0.08%, avg=211478.86, stdev=21810.34
lat (usec) : 100=0.01%, 250=0.01%, 500=94.48%, 750=0.01%, 1000=5.07%
lat (msec) : 2=0.27%, 4=0.17%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=10.74%, sys=14.47%, ctx=19622846, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,19623713,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
8kB Seq WR 1job QD32: (groupid=56, jobs=1): err= 0: pid=3724: Mon Jul 15 21:38:51 2019
write: IOPS=32.7k, BW=255MiB/s (268MB/s)(150GiB/600016msec)
slat (usec): min=1, max=368, avg= 3.44, stdev= 0.56
clat (usec): min=558, max=48272, avg=975.15, stdev=280.09
lat (usec): min=567, max=48275, avg=978.66, stdev=280.09
clat percentiles (usec):
| 1.00th=[ 836], 5.00th=[ 852], 10.00th=[ 868], 20.00th=[ 892],
| 30.00th=[ 900], 40.00th=[ 908], 50.00th=[ 916], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 964], 90.00th=[ 1384], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1496], 99.90th=[ 2576], 99.95th=[ 3056],
| 99.99th=[ 9280]
bw ( KiB/s): min=130411, max=278536, per=0.08%, avg=215437.38, stdev=28576.48
lat (usec) : 750=0.01%, 1000=88.96%
lat (msec) : 2=10.68%, 4=0.33%, 10=0.02%, 20=0.01%, 50=0.01%
cpu : usr=10.67%, sys=14.48%, ctx=19606367, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,19607401,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
8kB Seq RD 1job QD1: (groupid=57, jobs=1): err= 0: pid=3736: Mon Jul 15 21:38:51 2019
read: IOPS=22.4k, BW=175MiB/s (184MB/s)(103GiB/600001msec)
slat (usec): min=3, max=187, avg= 3.39, stdev= 0.51
clat (usec): min=0, max=81053, avg=40.60, stdev=32.13
lat (usec): min=42, max=81061, avg=44.05, stdev=32.13
clat percentiles (usec):
| 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39],
| 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 40],
| 70.00th=[ 40], 80.00th=[ 40], 90.00th=[ 40], 95.00th=[ 41],
| 99.00th=[ 49], 99.50th=[ 109], 99.90th=[ 141], 99.95th=[ 153],
| 99.99th=[ 306]
bw ( KiB/s): min=91568, max=183217, per=0.08%, avg=145957.80, stdev=18554.73
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=99.01%, 100=0.35%
lat (usec) : 250=0.62%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=2.47%, sys=10.16%, ctx=13451703, majf=0, minf=13
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=13451398,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
8kB Seq RD 1job QD2: (groupid=58, jobs=1): err= 0: pid=3743: Mon Jul 15 21:38:51 2019
read: IOPS=32.7k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=119, avg= 3.20, stdev= 0.42
clat (usec): min=5, max=33380, avg=57.27, stdev=57.42
lat (usec): min=41, max=33383, avg=60.55, stdev=57.42
clat percentiles (usec):
| 1.00th=[ 47], 5.00th=[ 49], 10.00th=[ 50], 20.00th=[ 51],
| 30.00th=[ 51], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 52],
| 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 69], 95.00th=[ 75],
| 99.00th=[ 78], 99.50th=[ 478], 99.90th=[ 580], 99.95th=[ 588],
| 99.99th=[ 1656]
bw ( KiB/s): min=133539, max=277728, per=0.08%, avg=209779.91, stdev=23582.38
lat (usec) : 10=0.01%, 20=0.01%, 50=8.93%, 100=90.15%, 250=0.23%
lat (usec) : 500=0.22%, 750=0.45%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=10.16%, sys=14.17%, ctx=19646639, majf=0, minf=14
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=19646601,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
8kB Seq RD 1job QD4: (groupid=59, jobs=1): err= 0: pid=3751: Mon Jul 15 21:38:51 2019
read: IOPS=32.7k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=219, avg= 3.20, stdev= 0.44
clat (usec): min=16, max=50425, avg=118.34, stdev=87.39
lat (usec): min=42, max=50428, avg=121.62, stdev=87.39
clat percentiles (usec):
| 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97],
| 30.00th=[ 98], 40.00th=[ 98], 50.00th=[ 119], 60.00th=[ 120],
| 70.00th=[ 121], 80.00th=[ 125], 90.00th=[ 126], 95.00th=[ 127],
| 99.00th=[ 588], 99.50th=[ 628], 99.90th=[ 644], 99.95th=[ 660],
| 99.99th=[ 1736]
bw ( KiB/s): min=163034, max=272913, per=0.08%, avg=219048.62, stdev=23217.67
lat (usec) : 20=0.01%, 50=0.01%, 100=47.64%, 250=50.96%, 500=0.16%
lat (usec) : 750=1.21%, 1000=0.01%
lat (msec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=10.16%, sys=14.26%, ctx=19643852, majf=0, minf=18
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=19644227,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
8kB Seq RD 1job QD8: (groupid=60, jobs=1): err= 0: pid=3763: Mon Jul 15 21:38:51 2019
read: IOPS=32.7k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=145, avg= 3.20, stdev= 0.43
clat (usec): min=39, max=91949, avg=240.60, stdev=144.96
lat (usec): min=43, max=91957, avg=243.88, stdev=144.96
clat percentiles (usec):
| 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 201], 20.00th=[ 221],
| 30.00th=[ 221], 40.00th=[ 223], 50.00th=[ 227], 60.00th=[ 227],
| 70.00th=[ 229], 80.00th=[ 229], 90.00th=[ 251], 95.00th=[ 251],
| 99.00th=[ 740], 99.50th=[ 756], 99.90th=[ 780], 99.95th=[ 1832],
| 99.99th=[ 1864]
bw ( KiB/s): min=122900, max=279300, per=0.08%, avg=219858.76, stdev=26384.22
lat (usec) : 50=0.01%, 100=0.01%, 250=87.70%, 500=9.53%, 750=2.11%
lat (usec) : 1000=0.56%
lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=10.12%, sys=14.22%, ctx=19637670, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=19638068,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
8kB Seq RD 1job QD16: (groupid=61, jobs=1): err= 0: pid=3770: Mon Jul 15 21:38:51 2019
read: IOPS=32.7k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=2, max=168, avg= 3.18, stdev= 0.42
clat (usec): min=184, max=59135, avg=485.05, stdev=192.35
lat (usec): min=207, max=59139, avg=488.31, stdev=192.35
clat percentiles (usec):
| 1.00th=[ 402], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 450],
| 30.00th=[ 454], 40.00th=[ 454], 50.00th=[ 454], 60.00th=[ 454],
| 70.00th=[ 454], 80.00th=[ 478], 90.00th=[ 482], 95.00th=[ 932],
| 99.00th=[ 996], 99.50th=[ 1004], 99.90th=[ 2064], 99.95th=[ 2096],
| 99.99th=[ 2576]
bw ( KiB/s): min=145156, max=275148, per=0.08%, avg=217816.29, stdev=27844.55
lat (usec) : 250=0.01%, 500=91.02%, 750=3.47%, 1000=4.93%
lat (msec) : 2=0.43%, 4=0.15%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=10.00%, sys=14.23%, ctx=19637761, majf=0, minf=44
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=19638084,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
8kB Seq RD 1job QD32: (groupid=62, jobs=1): err= 0: pid=3777: Mon Jul 15 21:38:51 2019
read: IOPS=32.7k, BW=256MiB/s (268MB/s)(150GiB/600001msec)
slat (usec): min=1, max=359, avg= 3.20, stdev= 0.44
clat (usec): min=548, max=68106, avg=973.71, stdev=269.28
lat (usec): min=554, max=68109, avg=976.99, stdev=269.28
clat percentiles (usec):
| 1.00th=[ 828], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 884],
| 30.00th=[ 908], 40.00th=[ 908], 50.00th=[ 908], 60.00th=[ 916],
| 70.00th=[ 940], 80.00th=[ 964], 90.00th=[ 1368], 95.00th=[ 1432],
| 99.00th=[ 1496], 99.50th=[ 1512], 99.90th=[ 2544], 99.95th=[ 2576],
| 99.99th=[ 3088]
bw ( KiB/s): min=130395, max=278416, per=0.08%, avg=215383.05, stdev=28494.59
lat (usec) : 750=0.02%, 1000=88.56%
lat (msec) : 2=11.09%, 4=0.32%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=10.05%, sys=14.30%, ctx=19640671, majf=0, minf=75
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=19641150,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
4kB Seq WR 1job QD1: (groupid=63, jobs=1): err= 0: pid=3825: Mon Jul 15 21:38:51 2019
write: IOPS=25.7k, BW=100MiB/s (105MB/s)(58.8GiB/600006msec)
slat (usec): min=3, max=51, avg= 3.32, stdev= 0.49
clat (usec): min=1, max=98510, avg=34.97, stdev=40.76
lat (usec): min=36, max=98519, avg=38.35, stdev=40.77
clat percentiles (usec):
| 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34],
| 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 35],
| 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 35], 95.00th=[ 35],
| 99.00th=[ 36], 99.50th=[ 37], 99.90th=[ 161], 99.95th=[ 177],
| 99.99th=[ 199]
bw ( KiB/s): min=52382, max=104879, per=0.08%, avg=83382.66, stdev=10500.72
lat (usec) : 2=0.01%, 20=0.01%, 50=99.64%, 100=0.01%, 250=0.35%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.03%, sys=11.31%, ctx=15407852, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,15407540,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
4kB Seq WR 1job QD2: (groupid=64, jobs=1): err= 0: pid=3837: Mon Jul 15 21:38:51 2019
write: IOPS=45.4k, BW=178MiB/s (186MB/s)(104GiB/600012msec)
slat (usec): min=2, max=170, avg= 3.27, stdev= 0.49
clat (usec): min=5, max=61773, avg=40.09, stdev=38.28
lat (usec): min=36, max=61777, avg=43.42, stdev=38.28
clat percentiles (usec):
| 1.00th=[ 37], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38],
| 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 39],
| 70.00th=[ 39], 80.00th=[ 39], 90.00th=[ 39], 95.00th=[ 39],
| 99.00th=[ 41], 99.50th=[ 165], 99.90th=[ 302], 99.95th=[ 310],
| 99.99th=[ 334]
bw ( KiB/s): min=92749, max=185498, per=0.08%, avg=145333.29, stdev=15495.09
lat (usec) : 10=0.01%, 20=0.01%, 50=99.14%, 100=0.01%, 250=0.68%
lat (usec) : 500=0.17%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=14.19%, sys=19.92%, ctx=27267213, majf=0, minf=11
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,27267599,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
4kB Seq WR 1job QD4: (groupid=65, jobs=1): err= 0: pid=3847: Mon Jul 15 21:38:51 2019
write: IOPS=52.5k, BW=205MiB/s (215MB/s)(120GiB/600038msec)
slat (usec): min=2, max=197, avg= 3.28, stdev= 0.50
clat (usec): min=16, max=30629, avg=72.28, stdev=78.51
lat (usec): min=36, max=30641, avg=75.62, stdev=78.51
clat percentiles (usec):
| 1.00th=[ 66], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67],
| 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 68],
| 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 68], 95.00th=[ 68],
| 99.00th=[ 195], 99.50th=[ 203], 99.90th=[ 1656], 99.95th=[ 1656],
| 99.99th=[ 1656]
bw ( KiB/s): min=126344, max=214539, per=0.08%, avg=177145.84, stdev=18637.06
lat (usec) : 20=0.01%, 50=0.01%, 100=98.43%, 250=1.35%, 500=0.02%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.18%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=16.50%, sys=22.97%, ctx=31490067, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,31492754,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
4kB Seq WR 1job QD8: (groupid=66, jobs=1): err= 0: pid=3862: Mon Jul 15 21:38:51 2019
write: IOPS=52.7k, BW=206MiB/s (216MB/s)(121GiB/600019msec)
slat (usec): min=2, max=165, avg= 3.27, stdev= 0.49
clat (usec): min=27, max=114281, avg=147.97, stdev=131.10
lat (usec): min=34, max=114289, avg=151.31, stdev=131.10
clat percentiles (usec):
| 1.00th=[ 139], 5.00th=[ 139], 10.00th=[ 139], 20.00th=[ 139],
| 30.00th=[ 139], 40.00th=[ 139], 50.00th=[ 139], 60.00th=[ 139],
| 70.00th=[ 139], 80.00th=[ 139], 90.00th=[ 141], 95.00th=[ 141],
| 99.00th=[ 270], 99.50th=[ 274], 99.90th=[ 1720], 99.95th=[ 1736],
| 99.99th=[ 1736]
bw ( KiB/s): min=107608, max=216609, per=0.08%, avg=172130.85, stdev=22174.14
lat (usec) : 50=0.01%, 100=0.01%, 250=98.05%, 500=1.56%, 750=0.02%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.36%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=16.62%, sys=22.98%, ctx=31598958, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,31602294,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
4kB Seq WR 1job QD16: (groupid=67, jobs=1): err= 0: pid=3873: Mon Jul 15 21:38:51 2019
write: IOPS=52.8k, BW=206MiB/s (216MB/s)(121GiB/600014msec)
slat (usec): min=2, max=289, avg= 3.27, stdev= 0.48
clat (usec): min=62, max=155151, avg=299.39, stdev=220.98
lat (usec): min=65, max=155154, avg=302.73, stdev=220.98
clat percentiles (usec):
| 1.00th=[ 282], 5.00th=[ 282], 10.00th=[ 282], 20.00th=[ 282],
| 30.00th=[ 282], 40.00th=[ 282], 50.00th=[ 282], 60.00th=[ 282],
| 70.00th=[ 282], 80.00th=[ 282], 90.00th=[ 282], 95.00th=[ 322],
| 99.00th=[ 442], 99.50th=[ 1864], 99.90th=[ 1880], 99.95th=[ 1880],
| 99.99th=[ 3312]
bw ( KiB/s): min=99204, max=217821, per=0.08%, avg=169947.39, stdev=20389.97
lat (usec) : 100=0.01%, 250=0.01%, 500=99.18%, 750=0.07%, 1000=0.01%
lat (msec) : 2=0.70%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 250=0.01%
cpu : usr=16.66%, sys=22.92%, ctx=31648899, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,31651991,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
4kB Seq WR 1job QD32: (groupid=68, jobs=1): err= 0: pid=3884: Mon Jul 15 21:38:51 2019
write: IOPS=52.7k, BW=206MiB/s (216MB/s)(121GiB/600093msec)
slat (usec): min=1, max=266, avg= 3.30, stdev= 0.51
clat (usec): min=161, max=54322, avg=603.69, stdev=260.89
lat (usec): min=165, max=54325, avg=607.06, stdev=260.89
clat percentiles (usec):
| 1.00th=[ 564], 5.00th=[ 564], 10.00th=[ 564], 20.00th=[ 564],
| 30.00th=[ 564], 40.00th=[ 564], 50.00th=[ 572], 60.00th=[ 572],
| 70.00th=[ 572], 80.00th=[ 572], 90.00th=[ 580], 95.00th=[ 708],
| 99.00th=[ 2128], 99.50th=[ 2160], 99.90th=[ 2160], 99.95th=[ 2608],
| 99.99th=[ 8032]
bw ( KiB/s): min=108029, max=208631, per=0.08%, avg=171983.97, stdev=18359.01
lat (usec) : 250=0.01%, 500=0.01%, 750=97.07%, 1000=1.44%
lat (msec) : 2=0.06%, 4=1.40%, 10=0.02%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=16.69%, sys=23.02%, ctx=31593527, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=0,31597690,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
4kB Seq RD 1job QD1: (groupid=69, jobs=1): err= 0: pid=3948: Mon Jul 15 21:38:51 2019
read: IOPS=26.4k, BW=103MiB/s (108MB/s)(60.3GiB/600001msec)
slat (usec): min=3, max=70, avg= 3.28, stdev= 0.47
clat (usec): min=1, max=81076, avg=34.03, stdev=27.11
lat (usec): min=35, max=81085, avg=37.37, stdev=27.11
clat percentiles (usec):
| 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33],
| 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 34],
| 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 34], 95.00th=[ 35],
| 99.00th=[ 36], 99.50th=[ 38], 99.90th=[ 137], 99.95th=[ 143],
| 99.99th=[ 189]
bw ( KiB/s): min=53857, max=107698, per=0.08%, avg=86619.93, stdev=11188.63
lat (usec) : 2=0.01%, 20=0.01%, 50=99.62%, 100=0.05%, 250=0.32%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=3.06%, sys=11.54%, ctx=15812711, majf=0, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=15812402,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
4kB Seq RD 1job QD2: (groupid=70, jobs=1): err= 0: pid=4099: Mon Jul 15 21:38:51 2019
read: IOPS=47.6k, BW=186MiB/s (195MB/s)(109GiB/600001msec)
slat (usec): min=2, max=200, avg= 3.16, stdev= 0.52
clat (usec): min=4, max=51496, avg=38.28, stdev=39.58
lat (usec): min=34, max=51499, avg=41.50, stdev=39.58
clat percentiles (usec):
| 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 36],
| 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 36],
| 70.00th=[ 37], 80.00th=[ 37], 90.00th=[ 38], 95.00th=[ 38],
| 99.00th=[ 42], 99.50th=[ 141], 99.90th=[ 788], 99.95th=[ 812],
| 99.99th=[ 828]
bw ( KiB/s): min=96790, max=194419, per=0.08%, avg=152525.17, stdev=17081.67
lat (usec) : 10=0.01%, 20=0.01%, 50=99.12%, 100=0.03%, 250=0.68%
lat (usec) : 500=0.02%, 750=0.01%, 1000=0.14%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=14.67%, sys=20.34%, ctx=28524110, majf=0, minf=12
IO depths : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=28531219,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=2
4kB Seq RD 1job QD4: (groupid=71, jobs=1): err= 0: pid=4126: Mon Jul 15 21:38:51 2019
read: IOPS=61.4k, BW=240MiB/s (252MB/s)(141GiB/600001msec)
slat (usec): min=2, max=179, avg= 3.16, stdev= 0.48
clat (usec): min=11, max=41004, avg=61.32, stdev=70.13
lat (usec): min=35, max=41007, avg=64.55, stdev=70.13
clat percentiles (usec):
| 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 58],
| 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 59],
| 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 59], 95.00th=[ 60],
| 99.00th=[ 167], 99.50th=[ 183], 99.90th=[ 282], 99.95th=[ 1368],
| 99.99th=[ 2576]
bw ( KiB/s): min=166181, max=253379, per=0.08%, avg=207832.16, stdev=21861.60
lat (usec) : 20=0.01%, 50=0.02%, 100=98.53%, 250=1.35%, 500=0.05%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.05%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=18.99%, sys=26.33%, ctx=36845435, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=36859094,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
4kB Seq RD 1job QD8: (groupid=72, jobs=1): err= 0: pid=4151: Mon Jul 15 21:38:51 2019
read: IOPS=61.6k, BW=241MiB/s (252MB/s)(141GiB/600001msec)
slat (usec): min=2, max=82, avg= 3.16, stdev= 0.42
clat (usec): min=30, max=81001, avg=126.08, stdev=106.16
lat (usec): min=36, max=81009, avg=129.30, stdev=106.16
clat percentiles (usec):
| 1.00th=[ 119], 5.00th=[ 119], 10.00th=[ 120], 20.00th=[ 120],
| 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 121],
| 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 122], 95.00th=[ 123],
| 99.00th=[ 245], 99.50th=[ 247], 99.90th=[ 940], 99.95th=[ 2640],
| 99.99th=[ 2640]
bw ( KiB/s): min=125515, max=254169, per=0.08%, avg=206495.51, stdev=24863.66
lat (usec) : 50=0.01%, 100=0.01%, 250=99.64%, 500=0.25%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.09%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=19.13%, sys=26.24%, ctx=36943140, majf=0, minf=17
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=36961618,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
4kB Seq RD 1job QD16: (groupid=73, jobs=1): err= 0: pid=4179: Mon Jul 15 21:38:51 2019
read: IOPS=61.8k, BW=242MiB/s (253MB/s)(142GiB/600001msec)
slat (usec): min=2, max=244, avg= 3.14, stdev= 0.42
clat (usec): min=63, max=40467, avg=255.00, stdev=138.78
lat (usec): min=66, max=40470, avg=258.21, stdev=138.78
clat percentiles (usec):
| 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 245],
| 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245],
| 70.00th=[ 245], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 274],
| 99.00th=[ 378], 99.50th=[ 430], 99.90th=[ 2768], 99.95th=[ 2768],
| 99.99th=[ 2768]
bw ( KiB/s): min=127429, max=256000, per=0.08%, avg=205053.23, stdev=26171.17
lat (usec) : 100=0.01%, 250=92.95%, 500=6.69%, 750=0.16%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.17%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=19.22%, sys=26.26%, ctx=37084484, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=37098659,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
4kB Seq RD 1job QD32: (groupid=74, jobs=1): err= 0: pid=4188: Mon Jul 15 21:38:51 2019
read: IOPS=61.8k, BW=241MiB/s (253MB/s)(141GiB/600001msec)
slat (usec): min=1, max=108, avg= 3.15, stdev= 0.40
clat (usec): min=338, max=57967, avg=514.02, stdev=220.07
lat (usec): min=342, max=57970, avg=517.23, stdev=220.07
clat percentiles (usec):
| 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 494], 20.00th=[ 494],
| 30.00th=[ 494], 40.00th=[ 494], 50.00th=[ 494], 60.00th=[ 494],
| 70.00th=[ 494], 80.00th=[ 494], 90.00th=[ 498], 95.00th=[ 620],
| 99.00th=[ 700], 99.50th=[ 892], 99.90th=[ 3024], 99.95th=[ 3024],
| 99.99th=[ 3504]
bw ( KiB/s): min=216112, max=255162, per=0.10%, avg=247303.18, stdev=3364.16
lat (usec) : 500=90.05%, 750=9.07%, 1000=0.48%
lat (msec) : 2=0.04%, 4=0.35%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=19.04%, sys=26.40%, ctx=37070920, majf=0, minf=41
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued rwt: total=37081350,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=3525GiB (3785GB), run=14400489-14400489msec
Run status group 1 (all jobs):
WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=149GiB (160GB), run=600576-600576msec
Run status group 2 (all jobs):
READ: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600060-600060msec
Run status group 3 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600007-600007msec
Run status group 4 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600040-600040msec
Run status group 5 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600097-600097msec
Run status group 6 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600064-600064msec
Run status group 7 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600159-600159msec
Run status group 8 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600187-600187msec
Run status group 9 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 10 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 11 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600002-600002msec
Run status group 12 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600004-600004msec
Run status group 13 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600008-600008msec
Run status group 14 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600016-600016msec
Run status group 15 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600007-600007msec
Run status group 16 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600060-600060msec
Run status group 17 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600211-600211msec
Run status group 18 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600023-600023msec
Run status group 19 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600106-600106msec
Run status group 20 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600059-600059msec
Run status group 21 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 22 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 23 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 24 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600002-600002msec
Run status group 25 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600004-600004msec
Run status group 26 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600008-600008msec
Run status group 27 (all jobs):
WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=149GiB (160GB), run=600007-600007msec
Run status group 28 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600037-600037msec
Run status group 29 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600054-600054msec
Run status group 30 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600039-600039msec
Run status group 31 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600046-600046msec
Run status group 32 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600032-600032msec
Run status group 33 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 34 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 35 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 36 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 37 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600002-600002msec
Run status group 38 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600004-600004msec
Run status group 39 (all jobs):
WRITE: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=144GiB (154GB), run=600013-600013msec
Run status group 40 (all jobs):
WRITE: bw=253MiB/s (265MB/s), 253MiB/s-253MiB/s (265MB/s-265MB/s), io=148GiB (159GB), run=600831-600831msec
Run status group 41 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600159-600159msec
Run status group 42 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600025-600025msec
Run status group 43 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600054-600054msec
Run status group 44 (all jobs):
WRITE: bw=252MiB/s (264MB/s), 252MiB/s-252MiB/s (264MB/s-264MB/s), io=148GiB (159GB), run=600521-600521msec
Run status group 45 (all jobs):
READ: bw=254MiB/s (267MB/s), 254MiB/s-254MiB/s (267MB/s-267MB/s), io=149GiB (160GB), run=600001-600001msec
Run status group 46 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 47 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 48 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600029-600029msec
Run status group 49 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600035-600035msec
Run status group 50 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600002-600002msec
Run status group 51 (all jobs):
WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=97.4GiB (105GB), run=600004-600004msec
Run status group 52 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600014-600014msec
Run status group 53 (all jobs):
WRITE: bw=255MiB/s (267MB/s), 255MiB/s-255MiB/s (267MB/s-267MB/s), io=149GiB (160GB), run=600044-600044msec
Run status group 54 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600007-600007msec
Run status group 55 (all jobs):
WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600030-600030msec
Run status group 56 (all jobs):
WRITE: bw=255MiB/s (268MB/s), 255MiB/s-255MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600016-600016msec
Run status group 57 (all jobs):
READ: bw=175MiB/s (184MB/s), 175MiB/s-175MiB/s (184MB/s-184MB/s), io=103GiB (110GB), run=600001-600001msec
Run status group 58 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 59 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 60 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 61 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 62 (all jobs):
READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB), run=600001-600001msec
Run status group 63 (all jobs):
WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=58.8GiB (63.1GB), run=600006-600006msec
Run status group 64 (all jobs):
WRITE: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=104GiB (112GB), run=600012-600012msec
Run status group 65 (all jobs):
WRITE: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=120GiB (129GB), run=600038-600038msec
Run status group 66 (all jobs):
WRITE: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=121GiB (129GB), run=600019-600019msec
Run status group 67 (all jobs):
WRITE: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=121GiB (130GB), run=600014-600014msec
Run status group 68 (all jobs):
WRITE: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=121GiB (129GB), run=600093-600093msec
Run status group 69 (all jobs):
READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=60.3GiB (64.8GB), run=600001-600001msec
Run status group 70 (all jobs):
READ: bw=186MiB/s (195MB/s), 186MiB/s-186MiB/s (195MB/s-195MB/s), io=109GiB (117GB), run=600001-600001msec
Run status group 71 (all jobs):
READ: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=141GiB (151GB), run=600001-600001msec
Run status group 72 (all jobs):
READ: bw=241MiB/s (252MB/s), 241MiB/s-241MiB/s (252MB/s-252MB/s), io=141GiB (151GB), run=600001-600001msec
Run status group 73 (all jobs):
READ: bw=242MiB/s (253MB/s), 242MiB/s-242MiB/s (253MB/s-253MB/s), io=142GiB (152GB), run=600001-600001msec
Run status group 74 (all jobs):
READ: bw=241MiB/s (253MB/s), 241MiB/s-241MiB/s (253MB/s-253MB/s), io=141GiB (152GB), run=600001-600001msec
Disk stats (read/write):
sdf: ios=414739529/397065979, merge=215/136, ticks=244047993/704528527, in_queue=947057936, util=98.74%
read : io=50010MB, bw=78.246KB/s, iops=13679, runt=180000msec
—fio做了50010MB的IO，速率78.246MB/s，总IOPS 13679 ，运行了3分钟。
slat (usec): min=3, max=335, avg= 9.73, stdev= 5.76
—slat (提交延迟submission latency):盘需要多久将IO提交到kernel做处理? 因子------------>>kernel---------->>i/o完成(具体什么意思参看其它大神解释)
连接:https://blog.csdn.net/beckdon/article/details/44591273
clat (usec): min=1, max=18600, avg=51.29, stdev=16.79
—clat(完成延迟completion latency)：kernel完成i/o动作的延迟
lat (usec): min=44, max=18627, avg=61.33, stdev=17.91
—lat(响应延迟 latency)：lat = slat + clat
举例：SAS硬盘，占测试的所有4个盘总IO的0.36%。
bw (KB /s): min= 71, max= 251, per=0.36%, avg=154.84, stdev=18.29
lat (usec) : 2= 0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=51.41%
lat (usec) : 100=48.53%, 250=0.06%, 500=0.01%, 1000=0.01%
lat (msec) : 2= 0.01%, 4=0.01%, 10=0.01%, 20=0.01%
—这组数据表示latency的分布，说明了51.41%的request延迟小于50微秒，48.53%的延迟小于100微秒(但是大于50微秒)，以此类推
clat percentiles (usec):
| 1.00th=[ 3952], 5.00th=[ 5792], 10.00th=[ 7200], 20.00th=[ 8896],
| 30.00th=[10304], 40.00th=[11456], 50.00th=[12608], 60.00th=[13760],
| 70.00th=[15168], 80.00th=[16768], 90.00th=[18816], 95.00th=[20608],
| 99.00th=[23424], 99.50th=[24192], 99.90th=[26752], 99.95th=[28032],
| 99.99th=[30080] bw (KB /s): min=52536, max=75504, per=67.14%, avg=63316.81, stdev=4057.09
—表示95%的IO请求的响应时间是小于等于 20.608 ms;表示60%的IO请求的响应时间是小于等于 13.76 ms
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
—Fio有一个iodepth设置，用来控制同一时刻发送给OS多少个IO。这完全是纯应用层面的行为，和盘的IO queue不是一回事。这里iodepth设成1，所以IO depth在全部时间都是1
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
—submit和complete代表同一时间段内fio发送上去和已完成的IO数量。对于产生这个输出的垃圾回收测试用例来说,iodepth是默认值1，所以100%的IO在同一时刻发送1次，放在1-4栏位里。通常来说，只有iodepth大于1才需要关注这一部分数据。
issued : total=r=0/w=736475126/d=0, short=r=0/w=0/d=0
—由于我的设置是随机写因此在这里r=0
MIXED(unified_rw_reporting参数运行测试): io=12497MB, aggrb=42653KB/s, minb=277KB/s, maxb=41711KB/s, mint=300000msec, maxt=300012msec
—io=表示总共完成的IO数量。在基于时间的测试中这是一个变量，在基于容量的测试中，这个值能匹配size参数。aggrb是所有进程/设备的汇总带宽。minb/maxb表示测量到的最小/最大带宽。mint/maxt表示测试的最短和最长耗时。和io=参数类似，时间值对于基于时间的测试应该能匹配runtime参数，对于基于容量的测试是一个变量。
Disk stats (read/write):
nvme0n1: ios=2515/2068995521, merge=0/0, ticks=619/3061444881, in_queue=3066967667, util=99.90%
—ios: 所有group总共执行的IO数.;merge= 总共发生的IO合并数;ticks= Number of ticks we kept the disk busy.;io_queue= 花费在队列上的总共时间.util= 磁盘利用率.
正常情况下BW 随bs的增加而增加；IOPS随bs的增加而减少
FIO硬盘性能测试思导图
1. 开机进OS之后查看message/dmesg/sel/mcelog/是否报错 (确保硬件无问题)
2. 将CPU打成performance状态(SUT处于性能状态，有的需要去BIOS设置)
3. 检查待测盘的上电时间(对于上电时间久的盘，不建议做性能测试待测盘)
4. 对磁盘进行安全擦除(如果对于一些有OP需求的建议先进行OP操作)
5. 检查samrt log中关键type是否增加如Media error/CRC error等
6. 绑核/绑中断/锁频跑测试(博主推荐绑核跑测试但不要把盘绑到core0或者core上),
7. 测试前收集samrt log
8. 测试后收集samrt log
9. 检查samrt log中关键type是否增加 and 系统log是否报错
10. 收集相应的测试log
11. 与SPEC文件作对比分析(iops:磁盘的每秒读写次数，这个是随机读写考察的重点 ;bw:磁盘的吞吐量，这个是顺序读写考察的重点，类似于下载速度)
硬盘测试总结
对于硬盘测试来说，现在没有详细统一的测试模板，这是因为对应于不同的客户我们要制定不同的测试方案，这也是由于市场的因素所决定的。
假如客户的服务器是用于存储数据的,那么我们制定性能测试方案的时候要更加细致的进行参数划分比如跑哪些block size 跑哪些 iodepth,跑多少job；而对于服务器是用于运维的者对硬盘的技术要求不高，我们一般跑相关硬盘SPEC文件要的参数即可。
综上所述，虽然我们没有详细的技术文件模板支撑我们的测试，但是我们知道FIO tool的关键参数用法，那我我们就可以做到以不变应万变来应对千千万万的技术要求。
2871
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
