fio模拟mysql写入速度_fio存储性能压测

一、环境说明：

此次演示是dellr620 物理服务器，系统为centos7.6 x86_64位

硬件相关的信息如下：

[root@mysql-redis scripts]# sh hw_msg1.sh

+--------------------------------------------------------------+

| This Machine's Hyper-Threading is Enabled(recommend disable) |

+--------------------------------------------------------------+

Systembit : 64

MEM info : 20*4096 MB

Disk_totle :

Pro_SN_name : Product Name: PowerEdge R620 Serial Number: 5RV2962

System name : \S

Board_SN name: Product Name: Serial Number:

CPU model : 32 Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz

CPU_phy_num : 2

CPU_core_num : 8

CPU_logic_num: 16

Netcard info : Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe

+++++++++++++++++++++++++++++++++++

<>

Virtual Drive: 0 (Target Id: 0)

Name :

RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0

Size : 893.75 GB

State : Optimal

Strip Size : 64 KB

Number Of Drives : 1

--

Virtual Drive Information:

Virtual Drive: 1 (Target Id: 1)

Name :

RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0

Size : 223.0 GB

State : Optimal

Strip Size : 64 KB

Number Of Drives : 1

+++++++++++++++++++++++++++++++++++

<>

Device Id: 3

PD Type: SATA

Raw Size: 894.252 GB [0x6fc81ab0 Sectors]

Inquiry Data: 56AB609UKCDU TOSHIBA Q300. SAFM12.3

Device Speed: 6.0Gb/s

Drive Temperature : N/A

Device Id: 0

PD Type: SATA

Raw Size: 223.570 GB [0x1bf244b0 Sectors]

Inquiry Data: PHDV723500E8240AGN INTEL SSDSC2BB240G7 N2010112

Device Speed: 6.0Gb/s

Drive Temperature :25C (77.00 F)

二、安装fio

1.redhat linux系统直接yum 安装

yum install fio.x86_64

centos7.6系统上直接yum 安装的话，默认的版本是fio.3.7

2.源码编译安装

为了测试异步io，需要安装libaio

yum install libaio-devel

wget -P /data/soft http://github.com/axboe/fio/archive/fio-3.7.tar.gz

tar xf /data/soft/fio.3.7.tar.gz -C /usr/local/

cd /usr/local/fio.3.7

./configure

make && make install

三、重要参数介绍

Fio各个版本的参数不尽相同，这里只列出日常较为广发使用的重要参数

filename=/dev/sdb1 或者/data/soft/test1： 指定测试硬盘名称或者测试文件名称(切忌选择系统设备测试，会导致系统奔溃的)

direct=1: 测试过程绕过机器自带的缓冲区，使得测试的结果更准确。

rw=randwrite: 随机写。 rw可取值：randwrite/randread/read/write/randrw/trim/randtrim/trimwrite ;分别是随机写，随机读，顺序读，顺序写，混合随机读写，

bs=16k : 单次I/O块大小为16KB,单次I/O大小对IOPS的影响比较大，一般来说要想得到最大的IOPS，bs越小越好。当然，这也和文件系统最小快的大小有关。一般linux下的文件分区最小快的大小为512B,1KB,2KB,4KB,8KB等可以根据文件系统类型进行选择。

size=5g : 测试文件大小为5GB

ioengine=libaio: I/O引擎使用libaio,libaio是异步的I/O引擎，一次提交一批I/O,然后等待这批I/O完成，这种方式减少了交互的次数，效率更好。FIO 至少支持13种不通的I/O引擎。sync,mmap,libaio,network等

iodepth=4 : I/O队列深度，主要是根据设备的并行度来调整。通常有两种I/O访问方式。也就是同步I/O，和异步I/O.同步I/O一次只能发送一个I/o请求，等待内核完成才可以返回。这对于单个线程来说iodepth总是小于1.想要iodepth大于1，可以通过多线程并发执行做到。异步I/O一次提交多个I/O请求。等I/O完成或者间隔一段时间收割一次。iodepth一般用在异步I/O模型(libaio),用于指定一次发起多少个I/O请求。Flash存储一般有多个并行的单元。支持多个I/O并发执行。较合适异步的I/O

-numjobs=4: 使用4个线程对硬盘设备进行压测 对于Raid设备，如raid1,raid5或者并行高的设备，可以考虑适当加大测试线程数-numjobs来测试。

runtime=1000: 测试时间为1000s.和参数size共同起作用，属于双限制，即达到任何一个限制都停止测试。

rwmixread=70: 在混合读写模式下，读占70%

group_reporting: 设置显示结果，指定group_reporting 将汇总每个进程的信息。

-name iops_randwrite： 改测试命名为 iops_randwrite

--output TestResult.log: 测试结果指定输出到文件TestResult.log 中

四、测试io性能

可以使用fio模拟测试各种不同类型的io请求，入随机写，顺序写，随机读，顺序读。混合随机读写

4.1随机写：

time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite --output TestResult.log

命令介绍：

采用4个线程对 /data/soft/test1文件做持续时间为60s，队列深度为1，块大小为4kb的direct y异步随机写(libaio)测试，该测试命名为iops_randwrite.输出结果不安4个jobs展示，而是按照group 组汇总 ，这样得出的在此压力下从此文件系统的随机写的IOPS。对于Raid设备，如raid1,raid5或者并行高的设备，可以考虑适当加大测试线程数-numjobs来测试。

--output TestResult.log 输出到指定的日志

下面是随机写具体执行过程：

[root@mysql-redis soft]# time fio -rw=randwrite -bs=4k -runtime=60 -iodepth 1 -numjobs=4 -size=5G -filename /data/soft/test1 -ioengine libaio -direct=1 -group_reporting -name iops_randwrite

iops_randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1

...

fio-3.7

Starting 4 processes

iops_randwrite: Laying out IO file (1 file / 5120MiB)

Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=68.2MiB/s][r=0,w=17.5k IOPS][eta 00m:00s]

iops_randwrite: (groupid=0, jobs=4): err= 0: pid=13505: Thu May 21 18:50:33 2020

write: IOPS=39.2k, BW=153MiB/s (160MB/s)(9182MiB/60001msec)

slat (usec): min=6, max=27813, avg=17.42, stdev=36.21

clat (nsec): min=1177, max=123217k, avg=82183.17, stdev=323538.76

lat (usec): min=53, max=123230, avg=99.84, stdev=325.69

clat percentiles (usec):

| 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 57], 20.00th=[ 62],

| 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 79],

| 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 103], 95.00th=[ 117],

| 99.00th=[ 161], 99.50th=[ 206], 99.90th=[ 330], 99.95th=[ 619],

| 99.99th=[14615]

bw ( KiB/s): min=17536, max=45304, per=25.12%, avg=39354.01, stdev=5145.44, samples=476

iops : min= 4384, max=11326, avg=9838.50, stdev=1286.36, samples=476

lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.22%

lat (usec) : 100=84.07%, 250=11.50%, 500=0.12%, 750=0.03%, 1000=0.01%

lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%

lat (msec) : 250=0.01%

cpu : usr=4.33%, sys=19.34%, ctx=2824889, majf=0, minf=1185

IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=0,2350467,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9182MiB (9628MB), run=60001-60001msec

Disk stats (read/write):

sda: ios=0/2349456, merge=0/0, ticks=0/167181, in_queue=166516, util=96.46%

real 1m0.706s

user 0m11.167s

sys 0m47.017s

4.2随机读：

time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread

关于命令参数说明：

命令介绍：采用2个线程对 /data/soft/test1文件做持续时间为60s，队列深度为1，块大小为4kb的direct y异步随机读(libaio)测试，该测试命名为iops_randread.输出结果不安2个jobs展示，而是按照group 组汇总 ，这样得出的在此压力下从此文件系统的随机读的IOPS

下面是随机读具体执行过程：

[root@mysql-redis39 ~]# time fio -readonly -rw=randread -bs=4k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randread

iops_randread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1

...

fio-3.7

Starting 2 processes

Jobs: 2 (f=2): [r(2)][100.0%][r=36.7MiB/s,w=0KiB/s][r=9389,w=0 IOPS][eta 00m:00s]

iops_randread: (groupid=0, jobs=2): err= 0: pid=24566: Fri May 22 06:21:46 2020

read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec)

slat (usec): min=7, max=1480, avg=23.09, stdev= 8.09

clat (usec): min=8, max=123984, avg=185.97, stdev=570.65

lat (usec): min=136, max=123999, avg=209.50, stdev=570.70

clat percentiles (usec):

| 1.00th=[ 139], 5.00th=[ 141], 10.00th=[ 145], 20.00th=[ 169],

| 30.00th=[ 174], 40.00th=[ 178], 50.00th=[ 182], 60.00th=[ 188],

| 70.00th=[ 198], 80.00th=[ 204], 90.00th=[ 212], 95.00th=[ 219],

| 99.00th=[ 255], 99.50th=[ 273], 99.90th=[ 322], 99.95th=[ 537],

| 99.99th=[ 1549]

bw ( KiB/s): min=13824, max=19616, per=49.99%, avg=18713.87, stdev=1060.66, samples=238

iops : min= 3456, max= 4904, avg=4678.43, stdev=265.16, samples=238

lat (usec) : 10=0.01%, 100=0.01%, 250=98.87%, 500=1.08%, 750=0.02%

lat (usec) : 1000=0.01%

lat (msec) : 2=0.02%, 4=0.01%, 20=0.01%, 250=0.01%

cpu : usr=4.11%, sys=15.44%, ctx=561546, majf=0, minf=498

IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=561503,0,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=2193MiB (2300MB), run=60001-60001msec

Disk stats (read/write):

sda: ios=560393/2, merge=0/0, ticks=102314/0, in_queue=101910, util=97.80%

real 1m0.706s

user 0m5.649s

sys 0m19.196s

以上需要关注的指标：

read: IOPS=9358, BW=36.6MiB/s (38.3MB/s)(2193MiB/60001msec) io=2193MiB (2300MB) 这些参数可以了解到iops和通途量

lat (usec): min=136, max=123999, avg=209.50, stdev=570.70 这些参数也要注意：io延迟平均在209.5微妙。

4.3顺序写：

time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write

说明：采用4个线程对/data/soft/test3文件做持续60s，队列深度为1的块大小为1M的direct异步顺序写(libaio)压力测试该测试命名为bw_write.输出结果不安4个job分别展示，而是按照group汇总展示。

[root@mysql-redis ~]# time fio -rw=write -bs=1m -runtime=60 -iodepth 1 -numjobs=5 -size=5G -filename /data/soft/test3 -ioengine libaio -direct=1 -group_reporting -name bw_write

bw_write: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1

...

fio-3.7

Starting 5 processes

bw_write: Laying out IO file (1 file / 5120MiB)

Jobs: 5 (f=5): [W(5)][100.0%][r=0KiB/s,w=184MiB/s][r=0,w=184 IOPS][eta 00m:00s]

bw_write: (groupid=0, jobs=5): err= 0: pid=26278: Fri May 22 06:44:38 2020

write: IOPS=319, BW=320MiB/s (335MB/s)(18.7GiB/60009msec)

slat (usec): min=56, max=303, avg=140.02, stdev=29.86

clat (msec): min=3, max=1092, avg=15.49, stdev=67.68

lat (msec): min=3, max=1093, avg=15.63, stdev=67.68

4.4顺序读：

time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read

[root@mysql-redis ~]# time fio -readonly -rw=read -bs=1m -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name bw_read

bw_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1

...

fio-3.7

Starting 2 processes

Jobs: 2 (f=2): [R(2)][100.0%][r=259MiB/s,w=0KiB/s][r=259,w=0 IOPS][eta 00m:00s]

bw_read: (groupid=0, jobs=2): err= 0: pid=26789: Fri May 22 06:51:01 2020

read: IOPS=255, BW=256MiB/s (268MB/s)(10.0GiB/40034msec)

slat (usec): min=95, max=1149, avg=245.64, stdev=42.23

clat (msec): min=4, max=130, avg= 7.57, stdev= 2.98

lat (msec): min=4, max=130, avg= 7.81, stdev= 2.98

4.5混合读写

time fio -rw=randrw -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

命令介绍：

2个线程对/data/soft/test2 文件做持续时间为60s，队列深度为1，块大小为16kb的direct异步混合读写的(libaio)压力测试，读写比伪7:3，测试被命名为iops_randrw

rwmixwrite=30 意思是在混合读写的模式下，写占30%。

-rwmixread=70 意思是在混合读写的模式下，读占70%。

混合读写具体输出结果：

[root@mysql-redis ~]# time fio -rw=randrw -rwmixread=70 -bs=16k -runtime=60 -iodepth 1 -numjobs=2 -filename /data/soft/test2 -ioengine libaio -direct=1 -group_reporting -name iops_randrw

iops_randrw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1

...

fio-3.7

Starting 2 processes

Jobs: 2 (f=2): [m(2)][100.0%][r=62.0MiB/s,w=27.2MiB/s][r=4030,w=1740 IOPS][eta 00m:00s]

iops_randrw: (groupid=0, jobs=2): err= 0: pid=27095: Fri May 22 06:55:08 2020

read: IOPS=3931, BW=61.4MiB/s (64.4MB/s)(3686MiB/60001msec)

slat (usec): min=8, max=6353, avg=36.20, stdev=19.77

clat (nsec): min=1648, max=123466k, avg=346175.56, stdev=725094.33

lat (usec): min=12, max=123516, avg=382.99, stdev=725.55

clat percentiles (usec):

| 1.00th=[ 215], 5.00th=[ 241], 10.00th=[ 255], 20.00th=[ 277],

| 30.00th=[ 293], 40.00th=[ 310], 50.00th=[ 322], 60.00th=[ 334],

| 70.00th=[ 351], 80.00th=[ 375], 90.00th=[ 412], 95.00th=[ 449],

| 99.00th=[ 1221], 99.50th=[ 1467], 99.90th=[ 1631], 99.95th=[ 1827],

| 99.99th=[16712]

bw ( KiB/s): min=23328, max=33792, per=49.97%, avg=31431.39, stdev=1649.11, samples=238

iops : min= 1458, max= 2112, avg=1964.44, stdev=103.06, samples=238

write: IOPS=1686, BW=26.4MiB/s (27.6MB/s)(1581MiB/60001msec)

slat (usec): min=12, max=2005, avg=42.06, stdev=16.09

clat (usec): min=3, max=123270, avg=225.73, stdev=432.84

lat (usec): min=86, max=123320, avg=268.44, stdev=433.42

clat percentiles (usec):

| 1.00th=[ 78], 5.00th=[ 85], 10.00th=[ 100], 20.00th=[ 129],

| 30.00th=[ 149], 40.00th=[ 174], 50.00th=[ 202], 60.00th=[ 233],

| 70.00th=[ 265], 80.00th=[ 306], 90.00th=[ 359], 95.00th=[ 408],

| 99.00th=[ 510], 99.50th=[ 1090], 99.90th=[ 1549], 99.95th=[ 1614],

| 99.99th=[ 5080]

bw ( KiB/s): min= 9152, max=15329, per=49.97%, avg=13484.37, stdev=892.91, samples=238

iops : min= 572, max= 958, avg=842.76, stdev=55.80, samples=238

lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%, 100=3.06%

lat (usec) : 250=22.22%, 500=72.91%, 750=0.76%, 1000=0.05%

lat (msec) : 2=0.96%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%

lat (msec) : 250=0.01%

cpu : usr=4.64%, sys=15.91%, ctx=346233, majf=0, minf=266

IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=235884,101198,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):

READ: bw=61.4MiB/s (64.4MB/s), 61.4MiB/s-61.4MiB/s (64.4MB/s-64.4MB/s), io=3686MiB (3865MB), run=60001-60001msec

WRITE: bw=26.4MiB/s (27.6MB/s), 26.4MiB/s-26.4MiB/s (27.6MB/s-27.6MB/s), io=1581MiB (1658MB), run=60001-60001msec

Disk stats (read/write):

sda: ios=274477/101294, merge=0/0, ticks=88860/19119, in_queue=107576, util=95.82%

real 1m0.690s

user 0m6.300s

sys 0m19.718s

4.6SSD性能测试配置文件

配置文件内容如下：

[root@mysql-redis scripts]# cat /data/soft/ssd-test.fio

##2020-05-24

[global]

bs=4k

ioengine=libaio

iodepth=4

numjobs=1

group_reporting

size=5g

direct=1

runtime=60

directory=/data/soft

filename=ssd.test.file

#inflate-log=/data/soft/TestResult.log

#output=/data/soft/TestResult.log

[seq-read]

name=iops_seqread

rw=read

stonewall

[rand-read]

name=iops_randread

rw=randread

stonewall

[seq-write]

name=iops_seqwrite

rw=write

stonewall

[rand-write]

name=iops_randwrite

rw=randwrite

stonewall

[randrw]

name=iops_randrw

rwmixread=70

rw=randrw

stonewall

下面是fio测试指定配置文件输出的内容：

[root@mysql-redis soft]# fio ssd-test.fio

iops_seqread: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4

iops_randread: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4

iops_seqwrite: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4

iops_randwrite: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4

iops_randrw: (g=4): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4

fio-3.7

Starting 5 processes

iops_seqread: Laying out IO file (1 file / 5120MiB)

Jobs: 1 (f=1): [_(4),m(1)][90.5%][r=47.8MiB/s,w=20.7MiB/s][r=12.2k,w=5308 IOPS][eta 00m:21s]

iops_seqread: (groupid=0, jobs=1): err= 0: pid=2142: Sun May 24 20:06:20 2020

read: IOPS=57.5k, BW=225MiB/s (236MB/s)(5120MiB/22782msec)

slat (usec): min=3, max=1943, avg= 5.13, stdev= 2.21

clat (usec): min=25, max=12467, avg=63.13, stdev=202.65

lat (usec): min=40, max=12472, avg=68.41, stdev=202.72

clat percentiles (usec):

| 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 56],

| 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 57], 60.00th=[ 57],

| 70.00th=[ 57], 80.00th=[ 58], 90.00th=[ 59], 95.00th=[ 62],

| 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 2073], 99.95th=[ 5080],

| 99.99th=[10814]

bw ( KiB/s): min=43579, max=193812, per=75.80%, avg=174446.82, stdev=47419.87, samples=45

iops : min=10894, max=48453, avg=43611.36, stdev=11855.06, samples=45

lat (usec) : 50=3.39%, 100=96.25%, 250=0.08%, 500=0.07%, 750=0.04%

lat (usec) : 1000=0.02%

lat (msec) : 2=0.06%, 4=0.04%, 10=0.05%, 20=0.01%

cpu : usr=19.83%, sys=39.29%, ctx=1222141, majf=0, minf=114

IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=4

iops_randread: (groupid=1, jobs=1): err= 0: pid=2170: Sun May 24 20:06:20 2020

read: IOPS=21.7k, BW=84.9MiB/s (89.0MB/s)(5094MiB/60001msec)

slat (usec): min=3, max=1068, avg=12.62, stdev= 6.14

clat (usec): min=42, max=38568, avg=167.76, stdev=315.76

lat (usec): min=62, max=38588, avg=180.75, stdev=316.07

clat percentiles (usec):

| 1.00th=[ 130], 5.00th=[ 135], 10.00th=[ 137], 20.00th=[ 141],

| 30.00th=[ 143], 40.00th=[ 147], 50.00th=[ 151], 60.00th=[ 155],

| 70.00th=[ 161], 80.00th=[ 172], 90.00th=[ 192], 95.00th=[ 206],

| 99.00th=[ 251], 99.50th=[ 273], 99.90th=[ 3982], 99.95th=[ 7308],

| 99.99th=[14091]

bw ( KiB/s): min= 5397, max=74196, per=79.00%, avg=68682.74, stdev=15713.88, samples=119

iops : min= 1349, max=18549, avg=17170.32, stdev=3928.48, samples=119

lat (usec) : 50=0.01%, 100=0.02%, 250=98.94%, 500=0.82%, 750=0.02%

lat (usec) : 1000=0.02%

lat (msec) : 2=0.03%, 4=0.04%, 10=0.07%, 20=0.02%, 50=0.01%

cpu : usr=11.52%, sys=37.46%, ctx=668232, majf=0, minf=319

IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=1304064,0,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=4

iops_seqwrite: (groupid=2, jobs=1): err= 0: pid=2231: Sun May 24 20:06:20 2020

write: IOPS=47.1k, BW=184MiB/s (193MB/s)(5120MiB/27848msec)

slat (usec): min=4, max=139, avg= 7.50, stdev= 2.96

clat (usec): min=27, max=15374, avg=76.14, stdev=278.53

lat (usec): min=49, max=15381, avg=83.78, stdev=279.26

clat percentiles (usec):

| 1.00th=[ 56], 5.00th=[ 59], 10.00th=[ 60], 20.00th=[ 61],

| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 67],

| 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 73], 95.00th=[ 74],

| 99.00th=[ 81], 99.50th=[ 89], 99.90th=[ 3326], 99.95th=[ 3392],

| 99.99th=[15139]

bw ( KiB/s): min= 3769, max=175262, per=80.83%, avg=152166.87, stdev=43009.10, samples=55

iops : min= 942, max=43815, avg=38041.35, stdev=10752.22, samples=55

lat (usec) : 50=0.24%, 100=99.43%, 250=0.07%, 500=0.01%, 750=0.01%

lat (usec) : 1000=0.01%

lat (msec) : 2=0.01%, 4=0.21%, 10=0.01%, 20=0.03%

cpu : usr=11.79%, sys=43.48%, ctx=652945, majf=0, minf=95

IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=4

iops_randwrite: (groupid=3, jobs=1): err= 0: pid=2262: Sun May 24 20:06:20 2020

write: IOPS=47.2k, BW=185MiB/s (194MB/s)(5120MiB/27743msec)

slat (usec): min=4, max=415, avg= 7.49, stdev= 2.82

clat (usec): min=26, max=21032, avg=75.63, stdev=276.91

lat (usec): min=48, max=21058, avg=83.27, stdev=277.60

clat percentiles (usec):

| 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 62],

| 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 67],

| 70.00th=[ 68], 80.00th=[ 70], 90.00th=[ 73], 95.00th=[ 74],

| 99.00th=[ 80], 99.50th=[ 86], 99.90th=[ 3326], 99.95th=[ 3392],

| 99.99th=[15139]

bw ( KiB/s): min= 3175, max=142274, per=68.21%, avg=128901.20, stdev=33671.41, samples=55

iops : min= 793, max=35568, avg=32224.95, stdev=8417.81, samples=55

lat (usec) : 50=0.21%, 100=99.50%, 250=0.04%, 500=0.01%, 750=0.01%

lat (usec) : 1000=0.01%

lat (msec) : 2=0.01%, 4=0.20%, 10=0.01%, 20=0.02%, 50=0.01%

cpu : usr=12.97%, sys=43.05%, ctx=654916, majf=0, minf=258

IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=4

iops_randrw: (groupid=4, jobs=1): err= 0: pid=2292: Sun May 24 20:06:20 2020

read: IOPS=11.3k, BW=44.3MiB/s (46.5MB/s)(2659MiB/60001msec)

slat (usec): min=4, max=594, avg=13.08, stdev= 6.88

clat (usec): min=61, max=45073, avg=244.45, stdev=441.39

lat (usec): min=134, max=45115, avg=257.84, stdev=441.61

clat percentiles (usec):

| 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 176],

| 30.00th=[ 190], 40.00th=[ 202], 50.00th=[ 212], 60.00th=[ 221],

| 70.00th=[ 233], 80.00th=[ 258], 90.00th=[ 330], 95.00th=[ 359],

| 99.00th=[ 474], 99.50th=[ 1270], 99.90th=[ 6521], 99.95th=[10028],

| 99.99th=[17695]

bw ( KiB/s): min= 3072, max=51824, per=99.92%, avg=45336.55, stdev=11250.66, samples=119

iops : min= 768, max=12958, avg=11334.12, stdev=2812.66, samples=119

write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(1139MiB/60001msec)

slat (usec): min=6, max=874, avg=16.94, stdev= 7.42

clat (usec): min=40, max=31265, avg=192.80, stdev=413.19

lat (usec): min=56, max=31306, avg=210.09, stdev=413.32

clat percentiles (usec):

| 1.00th=[ 52], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 78],

| 30.00th=[ 145], 40.00th=[ 169], 50.00th=[ 188], 60.00th=[ 202],

| 70.00th=[ 212], 80.00th=[ 227], 90.00th=[ 251], 95.00th=[ 277],

| 99.00th=[ 619], 99.50th=[ 1205], 99.90th=[ 5997], 99.95th=[ 9634],

| 99.99th=[16581]

bw ( KiB/s): min= 1184, max=22176, per=99.94%, avg=19428.64, stdev=4847.41, samples=119

iops : min= 296, max= 5544, avg=4857.13, stdev=1211.85, samples=119

lat (usec) : 50=0.12%, 100=7.22%, 250=74.36%, 500=17.22%, 750=0.24%

lat (usec) : 1000=0.10%

lat (msec) : 2=0.52%, 4=0.06%, 10=0.11%, 20=0.04%, 50=0.01%

cpu : usr=7.88%, sys=29.46%, ctx=340101, majf=0, minf=374

IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%

issued rwts: total=680630,291634,0,0 short=0,0,0,0 dropped=0,0,0,0

latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):

READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=5120MiB (5369MB), run=22782-22782msec

Run status group 1 (all jobs):

READ: bw=84.9MiB/s (89.0MB/s), 84.9MiB/s-84.9MiB/s (89.0MB/s-89.0MB/s), io=5094MiB (5341MB), run=60001-60001msec

Run status group 2 (all jobs):

WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5120MiB (5369MB), run=27848-27848msec

Run status group 3 (all jobs):

WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5120MiB (5369MB), run=27743-27743msec

Run status group 4 (all jobs):

READ: bw=44.3MiB/s (46.5MB/s), 44.3MiB/s-44.3MiB/s (46.5MB/s-46.5MB/s), io=2659MiB (2788MB), run=60001-60001msec

WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=1139MiB (1195MB), run=60001-60001msec

Disk stats (read/write):

sda: ios=3294785/2912828, merge=0/3, ticks=430527/238028, in_queue=666582, util=99.09%

[root@mysql-redis soft]#

五、总结

关于fio测试硬盘的建议：

一、建议使用顺序I/O和较大的blocksize 来测试设备的通吐量和延迟

二、建议使用随机I/O和较小的blocksize来测试设备的IOPS和延迟

三、在配置numjobs和iodepth 测试底层存储性能时，建议要深入了解应用到底采用的是同步的io还是异步的io(是多进程并发i/o请求还是一次提交一批的i/o的请求)

提示：对fio压测 磁盘性能感兴趣的话，可以参考下ezfio(直接在github.com上搜索ezfio)

重要提示：fio是 支持对裸设备直接进行I/O测试的，在测试裸设备时，一定要避开系统所在的裸设备，否则会导致系统奔溃。

例如：

[root@mysql-redis soft]# df -h / /boot

文件系统 容量 已用 可用 已用% 挂载点

/dev/sdb5 207G 7.5G 200G 4% /

/dev/sdb1 297M 120M 177M 41% /boot

/dev/sdb /dev/sdb5 /dev/sdb1 这些设备是绝对禁止进行fio测试的，

同时在测试其他的设备裸设备时，一定要提前确定好此裸设备上是否有重要 的数据，一定要重要的设备提前做好备份。