FIO测试IOPS指导

FIO是测试IOPS的非常好的工具，用来对硬件进行压力测试和验证，支持13种不同的I/O引擎，包括:sync,mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio 等等。
FIO用法:
随机读：(可直接用，向磁盘写一个2G文件，10线程，随机读1分钟，给出结果)
fio -filename=/tmp/test_randread -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest
说明：
filename=/dev/sdb1 #测试文件名称，通常选择需要测试的盘的data目录。
direct=1 #测试过程绕过机器自带的buffer。使测试结果更真实。
rw=randwrite #测试随机写的I/O
rw=randrw #测试随机写和读的I/O
rw参数说明：
read #顺序读
write #顺序写
rw,readwrite #顺序混合读写
randwrite #随机写
randread #随机读
randrw #随机混合读写
rwmixwrite #混合读写模式下写所占比例
rwmixread #混合读写模式读所占比例
bs=16k #单次io的块文件大小为16k
bsrange=512-2048 #同上，提定数据块的大小范围
size=5g #本次的测试文件大小为5g，以每次4k的io进行测试。
numjobs=30 #本次的测试线程为30.
runtime=1000 #测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止。
ioengine=psync #io引擎使用pync方式
rwmixwrite=30 #在混合读写的模式下，写占30%
group_reporting #关于显示结果的，汇总每个进程的信息。
此外：
lockmem=1g #只使用1g内存进行测试。
zero_buffers #用0初始化系统buffer。
nrfiles=8 #每个进程生成文件的数量。

简单介绍fio的参数：
ioengine: 负载引擎，我们一般使用libaio，发起异步IO请求。
bs: IO大小
direct: 直写，绕过操作系统Cache。因为我们测试的是硬盘，而不是操作系统的Cache，所以设置为1。
rw: 读写模式，有顺序写write、顺序读read、随机写randwrite、随机读randread等。
size: 寻址空间，IO会落在 [0, size)这个区间的硬盘空间上。这是一个可以影响IOPS的参数。一般设置为硬盘的大小。
filename: 测试对象
iodepth: 队列深度，只有使用libaio时才有意义。这是一个可以影响IOPS的参数。
runtime: 测试时长

测试结果参数说明：
bw：磁盘的吞吐量，这个是顺序读写考察的重点，类似于下载速度。
iops：磁盘的每秒读写次数，这个是随机读写考察的重点
io总的输入输出量
bw：带宽 KB/s
iops：每秒钟的IO数
runt：总运行时间
lat (msec)：延迟(毫秒)
msec： 毫秒
usec： 微秒

测试案例：
顺序读：
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest

随机写：
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest

顺序写：
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest

混合随机读写：
fio -filename=/dev/sdb1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest -ioscheduler=noop

后面附赠几个实际测试案例：
随机写4k案例：
[root@JSNJ-CloudJP-rac01 tmp]# /tmp/fio -filename=/dev/sdl -direct=1 -iodepth 128 -rw=randwrite -ioengine=libaio -bs=4k -runtime=30 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0K/241.9M /s] [0 /61.1K iops] [eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=52698
write: io=7606.4MB, bw=259586KB/s, iops=64896 , runt= 30005msec
slat (usec): min=5 , max=258 , avg= 9.79, stdev= 4.67
clat (usec): min=289 , max=42764 , avg=1961.58, stdev=2280.11
lat (usec): min=298 , max=42775 , avg=1971.51, stdev=2280.15
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 796], 10.00th=[ 892], 20.00th=[ 1048],
| 30.00th=[ 1176], 40.00th=[ 1304], 50.00th=[ 1432], 60.00th=[ 1592],
| 70.00th=[ 1768], 80.00th=[ 2008], 90.00th=[ 2576], 95.00th=[ 4640],
| 99.00th=[14016], 99.50th=[16512], 99.90th=[21632], 99.95th=[23424],
| 99.99th=[27776]
bw (KB/s) : min=211440, max=340944, per=100.00%, avg=259595.82, stdev=25087.73
lat (usec) : 500=0.12%, 750=3.26%, 1000=13.50%
lat (msec) : 2=62.90%, 4=14.63%, 10=3.15%, 20=2.26%, 50=0.18%
cpu : usr=13.32%, sys=63.54%, ctx=249330, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=0/w=1947220/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=7606.4MB, aggrb=259586KB/s, minb=259586KB/s, maxb=259586KB/s, mint=30005msec, maxt=30005msec

Disk stats (read/write):
sdl: ios=84/1941416, merge=0/0, ticks=10/3491174, in_queue=3497733, util=99.82%

随机读4k案例：
[root@JSNJ-CloudJP-rac01 tmp]# /tmp/fio -filename=/dev/sdl -direct=1 -iodepth 128 -rw=randread -ioengine=libaio -bs=4k -runtime=30 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [300.2M/0K /s] [77.6K/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=72456
read : io=10064MB, bw=343482KB/s, iops=85870 , runt= 30003msec
slat (usec): min=5 , max=166 , avg= 8.07, stdev= 4.02
clat (usec): min=80 , max=10600 , avg=1481.69, stdev=1418.22
lat (usec): min=87 , max=10607 , avg=1489.89, stdev=1418.07
clat percentiles (usec):
| 1.00th=[ 223], 5.00th=[ 358], 10.00th=[ 410], 20.00th=[ 478],
| 30.00th=[ 540], 40.00th=[ 620], 50.00th=[ 740], 60.00th=[ 988],
| 70.00th=[ 1592], 80.00th=[ 2864], 90.00th=[ 3824], 95.00th=[ 4320],
| 99.00th=[ 5728], 99.50th=[ 6304], 99.90th=[ 7264], 99.95th=[ 7520],
| 99.99th=[ 8768]
bw (KB/s) : min=279648, max=362952, per=100.00%, avg=343499.20, stdev=17476.77
lat (usec) : 100=0.01%, 250=1.34%, 500=22.13%, 750=27.18%, 1000=9.73%
lat (msec) : 2=13.28%, 4=18.33%, 10=8.00%, 20=0.01%
cpu : usr=8.31%, sys=74.61%, ctx=144615, majf=0, minf=153
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=2576370/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=10064MB, aggrb=343481KB/s, minb=343481KB/s, maxb=343481KB/s, mint=30003msec, maxt=30003msec

Disk stats (read/write):
sdl: ios=2568969/0, merge=0/0, ticks=3584118/0, in_queue=3592496, util=99.83%
[root@JSNJ-CloudJP-rac01 tmp]#

随机写1024k案例：
[root@JSNJ-CloudJP-rac01 tmp]# /tmp/fio -filename=/dev/sdl -direct=1 -iodepth 128 -rw=randwrite -ioengine=libaio -bs=1024k -runtime=30 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0K/935.0M /s] [0 /935 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=8340
write: io=29669MB, bw=994079KB/s, iops=970 , runt= 30562msec
slat (usec): min=97 , max=54550 , avg=769.20, stdev=2544.51
clat (msec): min=2 , max=1618 , avg=131.00, stdev=164.95
lat (msec): min=3 , max=1619 , avg=131.77, stdev=164.96
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 14],
| 30.00th=[ 24], 40.00th=[ 39], 50.00th=[ 63], 60.00th=[ 99],
| 70.00th=[ 151], 80.00th=[ 219], 90.00th=[ 359], 95.00th=[ 494],
| 99.00th=[ 717], 99.50th=[ 799], 99.90th=[ 1090], 99.95th=[ 1237],
| 99.99th=[ 1565]
bw (KB/s) : min=239831, max=2231433, per=100.00%, avg=996745.37, stdev=268800.16
lat (msec) : 4=0.34%, 10=13.72%, 20=13.05%, 50=18.52%, 100=14.75%
lat (msec) : 250=22.67%, 500=12.08%, 750=4.09%, 1000=0.62%, 2000=0.16%
cpu : usr=34.79%, sys=4.20%, ctx=4278, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=0/w=29669/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
WRITE: io=29669MB, aggrb=994079KB/s, minb=994079KB/s, maxb=994079KB/s, mint=30562msec, maxt=30562msec

Disk stats (read/write):
sdl: ios=84/32474, merge=0/26842, ticks=14/4013897, in_queue=4025740, util=99.47%
[root@JSNJ-CloudJP-rac01 tmp]#

随机读1024k案例：
[root@JSNJ-CloudJP-rac01 tmp]# /tmp/fio -filename=/dev/sdl -direct=1 -iodepth 128 -rw=randread -ioengine=libaio -bs=1024k -runtime=30 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [3383M/0K /s] [3383 /0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=18723
read : io=102400MB, bw=4491.4MB/s, iops=4491 , runt= 22801msec
slat (usec): min=42 , max=7651 , avg=56.03, stdev=68.81
clat (msec): min=1 , max=774 , avg=28.40, stdev=59.63
lat (msec): min=1 , max=774 , avg=28.46, stdev=59.63
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 1928], 10.00th=[ 2128], 20.00th=[ 2512],
| 30.00th=[ 2960], 40.00th=[ 3664], 50.00th=[ 5600], 60.00th=[11456],
| 70.00th=[19584], 80.00th=[33536], 90.00th=[79360], 95.00th=[136192],
| 99.00th=[284672], 99.50th=[382976], 99.90th=[610304], 99.95th=[675840],
| 99.99th=[733184]
bw (MB/s) : min= 2064, max= 9256, per=100.00%, avg=4627.87, stdev=1705.67
lat (msec) : 2=6.62%, 4=36.42%, 10=14.84%, 20=12.53%, 50=15.43%
lat (msec) : 100=6.29%, 250=6.57%, 500=1.00%, 750=0.27%, 1000=0.01%
cpu : usr=0.96%, sys=25.25%, ctx=68484, majf=0, minf=32793
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=102400/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=102400MB, aggrb=4491.4MB/s, minb=4491.4MB/s, maxb=4491.4MB/s, mint=22801msec, maxt=22801msec

Disk stats (read/write):
sdl: ios=102735/0, merge=101407/0, ticks=2867137/0, in_queue=2875590, util=99.63%
[root@JSNJ-CloudJP-rac01 tmp]#

随机混合读写4k案例：
[root@JSNJ-CloudJP-rac01 tmp]# /tmp/fio -filename=/dev/sdl -direct=1 -iodepth 128 -rw=randrw -ioengine=libaio -bs=4k -runtime=30 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
Jobs: 1 (f=1): [m] [100.0% done] [131.9M/132.8M /s] [33.8K/33.1K iops] [eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=27760
read : io=4830.4MB, bw=164853KB/s, iops=41213 , runt= 30004msec
slat (usec): min=5 , max=165 , avg= 8.84, stdev= 5.98
clat (usec): min=78 , max=29136 , avg=1439.47, stdev=847.49
lat (usec): min=90 , max=29143 , avg=1448.44, stdev=847.32
clat percentiles (usec):
| 1.00th=[ 221], 5.00th=[ 524], 10.00th=[ 724], 20.00th=[ 964],
| 30.00th=[ 1096], 40.00th=[ 1192], 50.00th=[ 1272], 60.00th=[ 1352],
| 70.00th=[ 1464], 80.00th=[ 1720], 90.00th=[ 2256], 95.00th=[ 3088],
| 99.00th=[ 4704], 99.50th=[ 5280], 99.90th=[ 7520], 99.95th=[ 9792],
| 99.99th=[15424]
bw (KB/s) : min=127200, max=175072, per=100.00%, avg=164863.27, stdev=9572.64
write: io=4823.5MB, bw=164619KB/s, iops=41154 , runt= 30004msec
slat (usec): min=5 , max=232 , avg=10.33, stdev= 6.62
clat (usec): min=328 , max=29983 , avg=1647.41, stdev=983.11
lat (usec): min=341 , max=29989 , avg=1657.87, stdev=982.89
clat percentiles (usec):
| 1.00th=[ 596], 5.00th=[ 796], 10.00th=[ 932], 20.00th=[ 1144],
| 30.00th=[ 1272], 40.00th=[ 1368], 50.00th=[ 1448], 60.00th=[ 1528],
| 70.00th=[ 1672], 80.00th=[ 1896], 90.00th=[ 2384], 95.00th=[ 3216],
| 99.00th=[ 5408], 99.50th=[ 6432], 99.90th=[11968], 99.95th=[14144],
| 99.99th=[20352]
bw (KB/s) : min=127992, max=174016, per=100.00%, avg=164626.58, stdev=9406.53
lat (usec) : 100=0.01%, 250=0.67%, 500=1.77%, 750=4.83%, 1000=10.17%
lat (msec) : 2=67.24%, 4=12.84%, 10=2.36%, 20=0.11%, 50=0.01%
cpu : usr=11.90%, sys=79.89%, ctx=41574, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=1236565/w=1234808/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=4830.4MB, aggrb=164853KB/s, minb=164853KB/s, maxb=164853KB/s, mint=30004msec, maxt=30004msec
WRITE: io=4823.5MB, aggrb=164619KB/s, minb=164619KB/s, maxb=164619KB/s, mint=30004msec, maxt=30004msec

Disk stats (read/write):
sdl: ios=1233437/1231638, merge=0/0, ticks=1225549/1477516, in_queue=2705356, util=99.79%
[root@JSNJ-CloudJP-rac01 tmp]#

随机混合读写1024k案例：
[root@JSNJ-CloudJP-rac01 tmp]# /tmp/fio -filename=/dev/sdl -direct=1 -iodepth 128 -rw=randrw -ioengine=libaio -bs=1024k -runtime=30 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=1M-1M/1M-1M, ioengine=libaio, iodepth=128
2.0.8
Starting 1 process
Jobs: 1 (f=1): [m] [100.0% done] [932.0M/970.0M /s] [932 /970 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=38116
read : io=30461MB, bw=1001.3MB/s, iops=1001 , runt= 30424msec
slat (usec): min=38 , max=25457 , avg=154.38, stdev=806.30
clat (msec): min=1 , max=1405 , avg= 7.92, stdev=26.59
lat (msec): min=1 , max=1405 , avg= 8.07, stdev=26.59
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7],
| 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 16],
| 99.00th=[ 26], 99.50th=[ 28], 99.90th=[ 429], 99.95th=[ 627],
| 99.99th=[ 1319]
bw (MB/s) : min= 491, max= 2480, per=100.00%, avg=1032.50, stdev=277.28
write: io=30188MB, bw=992.25MB/s, iops=992 , runt= 30424msec
slat (usec): min=98 , max=32326 , avg=485.17, stdev=837.63
clat (msec): min=1 , max=2777 , avg=120.31, stdev=155.84
lat (msec): min=2 , max=2777 , avg=120.80, stdev=155.85
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 9],
| 30.00th=[ 18], 40.00th=[ 38], 50.00th=[ 67], 60.00th=[ 101],
| 70.00th=[ 147], 80.00th=[ 204], 90.00th=[ 314], 95.00th=[ 408],
| 99.00th=[ 668], 99.50th=[ 791], 99.90th=[ 1418], 99.95th=[ 1647],
| 99.99th=[ 2442]
bw (MB/s) : min= 503, max= 2211, per=100.00%, avg=1019.10, stdev=251.33
lat (msec) : 2=0.16%, 4=11.61%, 10=43.49%, 20=9.36%, 50=7.80%
lat (msec) : 100=7.45%, 250=12.63%, 500=6.10%, 750=1.07%, 1000=0.19%
lat (msec) : 2000=0.12%, >=2000=0.01%
cpu : usr=33.45%, sys=10.19%, ctx=17267, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=30461/w=30188/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=30461MB, aggrb=1001.3MB/s, minb=1001.3MB/s, maxb=1001.3MB/s, mint=30424msec, maxt=30424msec
WRITE: io=30188MB, aggrb=992.25MB/s, minb=992.25MB/s, maxb=992.25MB/s, mint=30424msec, maxt=30424msec

Disk stats (read/write):
sdl: ios=32449/32013, merge=28543/28352, ticks=206505/3764242, in_queue=3974680, util=99.45%