fio

最新推荐文章于 2021-08-27 10:39:01 发布

a545578125

最新推荐文章于 2021-08-27 10:39:01 发布

阅读量943

点赞数

分类专栏： oracle db

本文链接：https://blog.csdn.net/a545578125/article/details/49886809

版权

oracle db 专栏收录该内容

78 篇文章 2 订阅

订阅专栏

一，FIO安装
wget http://brick.kernel.dk/snaps/fio-2.0.7.tar.gz
yum install libaio-devel
tar -zxvf fio-2.0.7.tar.gz
cd fio-2.0.7
make
make install

说明：
filename=/dev/sdb1       测试文件名称，通常选择需要测试的盘的data目录。
direct=1                 测试过程绕过机器自带的buffer。使测试结果更真实。
rw=randwrite             测试随机写的I/O
rw=randrw                测试随机写和读的I/O
bs=16k                   单次io的块文件大小为16k
bsrange=512-2048         同上，提定数据块的大小范围
size=5g    本次的测试文件大小为5g，以每次4k的io进行测试。
numjobs=30               本次的测试线程为30.
runtime=1000             测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止。
ioengine=psync           io引擎使用pync方式
rwmixwrite=30            在混合读写的模式下，写占30%
group_reporting          关于显示结果的，汇总每个进程的信息。

此外
lockmem=1g               只使用1g内存进行测试。
zero_buffers             用0初始化系统buffer。
nrfiles=8                每个进程生成文件的数量。

阿里云普通机器测试结果：

[root@SIP-DG-DB testio]# fio -filename=/oracle/testio/1_4000_883154058.dbf -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=100 -group_reporting -name=mytest

mytest: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 10 threads
mytest: Laying out IO file(s) (1 file(s) / 2048MB)
Jobs: 10 (f=10): [rrrrrrrrrr] [100.0% done] [23632KB/0KB/0KB /s] [1477/0/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=10): err= 0: pid=10135: Tue Nov 17 12:40:16 2015
read : io=1563.1MB, bw=15929KB/s, iops=995, runt=100536msec
    clat (usec): min=668, max=1359.4K, avg=10041.30, stdev=74308.92
     lat (usec): min=669, max=1359.4K, avg=10041.53, stdev=74308.92
    clat percentiles (usec):
     | 1.00th=[ 1368], 5.00th=[ 1720], 10.00th=[ 1960], 20.00th=[ 2288],
     | 30.00th=[ 2544], 40.00th=[ 2800], 50.00th=[ 3024], 60.00th=[ 3312],
     | 70.00th=[ 3664], 80.00th=[ 4256], 90.00th=[ 7584], 95.00th=[14656],
     | 99.00th=[40704], 99.50th=[501760], 99.90th=[1155072], 99.95th=[1236992],
     | 99.99th=[1318912]
    bw (KB /s): min=   16, max= 4790, per=12.77%, avg=2034.64, stdev=1255.47
    lat (usec) : 750=0.01%, 1000=0.04%
    lat (msec) : 2=10.86%, 4=65.73%, 10=15.29%, 20=5.43%, 50=1.77%
    lat (msec) : 100=0.12%, 250=0.08%, 500=0.15%, 750=0.03%, 1000=0.22%
    lat (msec) : 2000=0.26%
cpu          : usr=0.05%, sys=0.18%, ctx=100155, majf=0, minf=50
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=100092/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=1563.1MB, aggrb=15929KB/s, minb=15929KB/s, maxb=15929KB/s, mint=100536msec, maxt=100536msec

Disk stats (read/write):
xvdb: ios=101624/24515, merge=2/160, ticks=1019970/574278, in_queue=1605730, util=100.00%
[root@SIP-DG-DB testio]#

[root@SIP-DG-DB testio]# fio -filename=/oracle/testio/1_4000_883154058.dbf -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done] [0KB/7264KB/0KB /s] [0/454/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=10290: Tue Nov 17 12:46:09 2015
write: io=776480KB, bw=7758.9KB/s, iops=484, runt=100077msec
    clat (usec): min=938, max=2243.6K, avg=61835.85, stdev=260320.68
     lat (usec): min=939, max=2243.6K, avg=61839.04, stdev=260320.90
    clat percentiles (usec):
     | 1.00th=[ 1096], 5.00th=[ 1176], 10.00th=[ 1224], 20.00th=[ 1320],
     | 30.00th=[ 1416], 40.00th=[ 1544], 50.00th=[ 1768], 60.00th=[ 2128],
     | 70.00th=[ 2448], 80.00th=[ 2800], 90.00th=[ 3792], 95.00th=[659456],
     | 99.00th=[1368064], 99.50th=[1531904], 99.90th=[1794048], 99.95th=[1859584],
     | 99.99th=[2179072]
    bw (KB /s): min=    8, max= 2133, per=3.43%, avg=265.88, stdev=282.83
    lat (usec) : 1000=0.02%
    lat (msec) : 2=56.39%, 4=34.35%, 10=3.57%, 20=0.22%, 50=0.02%
    lat (msec) : 100=0.01%, 250=0.03%, 500=0.07%, 750=0.63%, 1000=1.34%
    lat (msec) : 2000=3.34%, >=2000=0.02%
cpu          : usr=0.02%, sys=0.05%, ctx=97137, majf=0, minf=8
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=48530/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=776480KB, aggrb=7758KB/s, minb=7758KB/s, maxb=7758KB/s, mint=100077msec, maxt=100077msec

Disk stats (read/write):
xvdb: ios=239/113070, merge=0/15, ticks=3696/253713, in_queue=257364, util=99.52%
[root@SIP-DG-DB testio]#

[root@SIP-DG-DB testio]# fio -filename=/oracle/testio/1_4000_883154058.dbf -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [2080KB/2224KB/0KB /s] [130/139/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=11527: Tue Nov 17 13:32:18 2015
read : io=249312KB, bw=2490.6KB/s, iops=155, runt=100104msec
    clat (usec): min=520, max=482883, avg=95783.79, stdev=94938.05
     lat (usec): min=521, max=482883, avg=95784.11, stdev=94938.06
    clat percentiles (usec):
     | 1.00th=[ 732], 5.00th=[ 1352], 10.00th=[ 2352], 20.00th=[ 8256],
     | 30.00th=[13248], 40.00th=[21632], 50.00th=[68096], 60.00th=[119296],
     | 70.00th=[150528], 80.00th=[187392], 90.00th=[226304], 95.00th=[261120],
     | 99.00th=[354304], 99.50th=[391168], 99.90th=[448512], 99.95th=[460800],
     | 99.99th=[481280]
    bw (KB /s): min=   20, max= 288, per=3.36%, avg=83.77, stdev=34.29
write: io=219696KB, bw=2194.7KB/s, iops=137, runt=100104msec
    clat (msec): min=1, max=474, avg=109.91, stdev=106.19
     lat (msec): min=1, max=474, avg=109.92, stdev=106.19
    clat percentiles (usec):
     | 1.00th=[ 1256], 5.00th=[ 1480], 10.00th=[ 1736], 20.00th=[ 2576],
     | 30.00th=[ 5728], 40.00th=[16768], 50.00th=[110080], 60.00th=[150528],
     | 70.00th=[181248], 80.00th=[209920], 90.00th=[250880], 95.00th=[288768],
     | 99.00th=[362496], 99.50th=[399360], 99.90th=[452608], 99.95th=[456704],
     | 99.99th=[464896]
    bw (KB /s): min=   19, max= 290, per=3.60%, avg=78.96, stdev=44.75
    lat (usec) : 750=0.58%, 1000=0.85%
    lat (msec) : 2=9.21%, 4=9.34%, 10=9.00%, 20=11.09%, 50=6.40%
    lat (msec) : 100=5.39%, 250=40.00%, 500=8.12%
cpu          : usr=0.01%, sys=0.02%, ctx=51370, majf=0, minf=7
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=15582/w=13731/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=249312KB, aggrb=2490KB/s, minb=2490KB/s, maxb=2490KB/s, mint=100104msec, maxt=100104msec
WRITE: io=219696KB, aggrb=2194KB/s, minb=2194KB/s, maxb=2194KB/s, mint=100104msec, maxt=100104msec

Disk stats (read/write):
xvdb: ios=15970/48482, merge=0/39, ticks=232192/340517, in_queue=572709, util=99.99%

[root@SIP-DG-DB ~]# fio -filename=/oracle/testio/1_4000_883154058.dbf -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% done] [28672KB/0KB/0KB /s] [1792/0/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=12280: Tue Nov 17 14:00:22 2015
read : io=1950.8MB, bw=19974KB/s, iops=1248, runt=100006msec
    clat (msec): min=1, max=1816, avg=24.03, stdev=149.04
     lat (msec): min=1, max=1816, avg=24.03, stdev=149.04
    clat percentiles (msec):
     | 1.00th=[    3], 5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    7], 60.00th=[    7],
     | 70.00th=[    8], 80.00th=[    9], 90.00th=[   11], 95.00th=[   15],
     | 99.00th=[ 1221], 99.50th=[ 1401], 99.90th=[ 1467], 99.95th=[ 1483],
     | 99.99th=[ 1516]
    bw (KB /s): min=   10, max= 3001, per=5.55%, avg=1107.58, stdev=1048.92
    lat (msec) : 2=0.23%, 4=9.33%, 10=80.11%, 20=7.16%, 50=1.49%
    lat (msec) : 100=0.19%, 250=0.14%, 500=0.12%, 750=0.05%, 2000=1.18%
cpu          : usr=0.02%, sys=0.07%, ctx=126049, majf=0, minf=129
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=124848/w=0/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=1950.8MB, aggrb=19974KB/s, minb=19974KB/s, maxb=19974KB/s, mint=100006msec, maxt=100006msec

Disk stats (read/write):
xvdb: ios=133818/13243, merge=513/30468, ticks=3327304/855913, in_queue=4183275, util=100.00%
[root@SIP-DG-DB ~]# fio -filename=/oracle/testio/1_4000_883154058.dbf -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW] [100.0% done] [0KB/8160KB/0KB /s] [0/510/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=12386: Tue Nov 17 14:02:34 2015
write: io=819664KB, bw=8191.7KB/s, iops=511, runt=100061msec
    clat (usec): min=930, max=2216.1K, avg=58572.92, stdev=262850.66
     lat (usec): min=934, max=2216.1K, avg=58576.02, stdev=262850.91
    clat percentiles (usec):
     | 1.00th=[ 996], 5.00th=[ 1048], 10.00th=[ 1096], 20.00th=[ 1176],
     | 30.00th=[ 1272], 40.00th=[ 1448], 50.00th=[ 1752], 60.00th=[ 2096],
     | 70.00th=[ 2320], 80.00th=[ 2640], 90.00th=[ 3440], 95.00th=[11712],
     | 99.00th=[1449984], 99.50th=[1630208], 99.90th=[1908736], 99.95th=[1974272],
     | 99.99th=[2146304]
    bw (KB /s): min=    7, max= 2363, per=3.42%, avg=280.07, stdev=299.58
    lat (usec) : 1000=1.09%
    lat (msec) : 2=55.62%, 4=35.65%, 10=2.57%, 20=0.23%, 50=0.05%
    lat (msec) : 100=0.02%, 250=0.01%, 500=0.03%, 750=0.35%, 1000=1.04%
    lat (msec) : 2000=3.30%, >=2000=0.04%
cpu          : usr=0.02%, sys=0.05%, ctx=102565, majf=0, minf=14
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=51229/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: io=819664KB, aggrb=8191KB/s, minb=8191KB/s, maxb=8191KB/s, mint=100061msec, maxt=100061msec

Disk stats (read/write):

xvdb: ios=2264/133530, merge=0/9, ticks=11035/283243, in_queue=294219, util=99.72%

阿里云普通云优化盘

[root@iZ259jzdb59Z test]# fio -filename=1_16_889610461.dbf -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.10
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [31984KB/22793KB/0KB /s] [1999/1424/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=4999: Tue Jan 5 17:18:31 2016
read : io=2237.7MB, bw=22911KB/s, iops=1431, runt=100012msec
    clat (usec): min=437, max=85080, avg=9879.15, stdev=10917.82
     lat (usec): min=437, max=85081, avg=9879.55, stdev=10917.86
    clat percentiles (usec):
     | 1.00th=[ 502], 5.00th=[ 540], 10.00th=[ 564], 20.00th=[ 612],
     | 30.00th=[ 668], 40.00th=[ 716], 50.00th=[ 892], 60.00th=[12480],
     | 70.00th=[16512], 80.00th=[20608], 90.00th=[25472], 95.00th=[29568],
     | 99.00th=[37632], 99.50th=[41216], 99.90th=[50432], 99.95th=[54016],
     | 99.99th=[63232]
    bw (KB /s): min= 288, max= 1571, per=3.34%, avg=764.86, stdev=180.69
write: io=2238.7MB, bw=22921KB/s, iops=1432, runt=100012msec
    clat (usec): min=512, max=86384, avg=11054.84, stdev=11861.68
     lat (usec): min=514, max=86387, avg=11057.36, stdev=11862.00
    clat percentiles (usec):
     | 1.00th=[ 588], 5.00th=[ 612], 10.00th=[ 628], 20.00th=[ 644],
     | 30.00th=[ 668], 40.00th=[ 692], 50.00th=[ 820], 60.00th=[14784],
     | 70.00th=[18816], 80.00th=[22656], 90.00th=[27776], 95.00th=[31872],
     | 99.00th=[40192], 99.50th=[44288], 99.90th=[52992], 99.95th=[56064],
     | 99.99th=[64768]
    bw (KB /s): min= 216, max= 1647, per=3.33%, avg=764.28, stdev=114.04
    lat (usec) : 500=0.40%, 750=46.06%, 1000=3.89%
    lat (msec) : 2=0.31%, 4=0.30%, 10=3.33%, 20=21.53%, 50=24.05%
    lat (msec) : 100=0.14%
cpu          : usr=0.06%, sys=0.19%, ctx=500311, majf=0, minf=8
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=143210/w=143271/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=2237.7MB, aggrb=22910KB/s, minb=22910KB/s, maxb=22910KB/s, mint=100012msec, maxt=100012msec
WRITE: io=2238.7MB, aggrb=22920KB/s, minb=22920KB/s, maxb=22920KB/s, mint=100012msec, maxt=100012msec

Disk stats (read/write):
vdb: ios=142982/144179, merge=1065/19, ticks=81503/85786, in_queue=167289, util=87.28%

阿里云ssd测试结果：

[root@iZ2512hmlr7Z datapump]# fio -filename=/oradata/backup/datapump/1_16_889610461.dbf -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
mytest: (g=0): rw=randread, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.57
Starting 30 threads
mytest: Laying out IO file(s) (1 file(s) / 2048MB)
Jobs: 30 (f=30): [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr] [100.0% done] [197.6M/0K /s] [12.4K/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=9636
read : io=18749MB, bw=191874KB/s, iops=11992 , runt=100059msec
    clat (usec): min=38 , max=129759 , avg=2566.50, stdev=2049.41
     lat (usec): min=39 , max=129760 , avg=2566.72, stdev=2049.41
    bw (KB/s) : min= 1108, max=13312, per=3.26%, avg=6246.25, stdev=241.75
cpu          : usr=3.44%, sys=27.93%, ctx=45184590, majf=0, minf=16107
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=1199922/0/0, short=0/0/0
     lat (usec): 50=0.01%, 100=3.02%, 250=10.09%, 500=15.02%, 750=28.63%
     lat (usec): 1000=19.91%
     lat (msec): 2=20.35%, 4=0.44%, 10=0.03%, 20=0.01%, 50=0.01%
     lat (msec): 100=2.39%, 250=0.10%

Run status group 0 (all jobs):
   READ: io=18749MB, aggrb=191874KB/s, minb=196479KB/s, maxb=196479KB/s, mint=100059msec, maxt=100059msec

Disk stats (read/write):
xvdb: ios=1199907/770, merge=8348/255, ticks=2762442/5533, in_queue=2768905, util=99.79%
[root@iZ2512hmlr7Z datapump]# fio -filename=/oradata/backup/datapump/1_16_889610461.dbf -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.57
Starting 30 threads
Jobs: 30 (f=30): [wwwwwwwwwwwwwwwwwwwwwwwwwwwwww] [100.0% done] [0K/98043K /s] [0 /5984 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=9693
write: io=9488.2MB, bw=97151KB/s, iops=6071 , runt=100007msec
    clat (usec): min=98 , max=7475.4K, avg=5410.86, stdev=24709.20
     lat (usec): min=98 , max=7475.4K, avg=5411.05, stdev=24709.20
    bw (KB/s) : min=    2, max=112960, per=2.96%, avg=2877.09, stdev=920.36
cpu          : usr=1.60%, sys=8.54%, ctx=36709541, majf=0, minf=14552
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/607239/0, short=0/0/0
     lat (usec): 100=0.01%, 250=95.61%, 500=2.86%, 750=0.65%, 1000=0.44%
     lat (msec): 2=0.14%, 4=0.11%, 10=0.02%, 20=0.01%, 100=0.01%
     lat (msec): 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.03%
     lat (msec): >=2000=0.12%

Run status group 0 (all jobs):
WRITE: io=9488.2MB, aggrb=97151KB/s, minb=99483KB/s, maxb=99483KB/s, mint=100007msec, maxt=100007msec

Disk stats (read/write):
xvdb: ios=0/611147, merge=0/256, ticks=0/84190, in_queue=84187, util=83.68%
[root@iZ2512hmlr7Z datapump]# fio -filename=/oradata/backup/datapump/1_16_889610461.dbf -direct=1 -iodepth 1 -thread -rw=randrw -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
mytest: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.57
Starting 30 threads
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0% done] [89100K/89850K /s] [5438 /5484 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=9754
read : io=8044.7MB, bw=82374KB/s, iops=5148 , runt=100003msec
    clat (usec): min=41 , max=129820 , avg=2861.02, stdev=635.91
     lat (usec): min=41 , max=129820 , avg=2861.20, stdev=635.91
    bw (KB/s) : min= 152, max= 3523, per=3.34%, avg=2749.74, stdev=59.37
write: io=8033.9MB, bw=82264KB/s, iops=5141 , runt=100003msec
    clat (usec): min=103 , max=128712 , avg=3015.49, stdev=610.40
     lat (usec): min=103 , max=128712 , avg=3015.69, stdev=610.40
    bw (KB/s) : min=   28, max= 4662, per=3.29%, avg=2706.90, stdev=87.27
cpu          : usr=3.16%, sys=13.10%, ctx=54279127, majf=0, minf=14461
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=514854/514168/0, short=0/0/0
     lat (usec): 50=0.84%, 100=22.17%, 250=26.15%, 500=0.98%, 750=0.22%
     lat (usec): 1000=0.15%
     lat (msec): 2=0.06%, 4=6.35%, 10=41.89%, 20=1.05%, 50=0.12%
     lat (msec): 100=0.02%, 250=0.01%

Run status group 0 (all jobs):
   READ: io=8044.7MB, aggrb=82374KB/s, minb=84351KB/s, maxb=84351KB/s, mint=100003msec, maxt=100003msec
WRITE: io=8033.9MB, aggrb=82264KB/s, minb=84238KB/s, maxb=84238KB/s, mint=100003msec, maxt=100003msec

Disk stats (read/write):
xvdb: ios=514169/517584, merge=4136/295, ticks=33913/81307, in_queue=115211, util=85.96%

[root@iZ2512hmlr7Z datapump]# fio -filename=/oradata/backup/datapump/1_16_889610461.dbf -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
mytest: (g=0): rw=read, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.57
Starting 30 threads
Jobs: 30 (f=30): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% done] [193.2M/0K /s] [12.8K/0 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=9868
read : io=18948MB, bw=194024KB/s, iops=12126 , runt=100002msec
    clat (usec): min=29 , max=123104 , avg=2488.68, stdev=2056.41
     lat (usec): min=29 , max=123104 , avg=2488.93, stdev=2056.41
    bw (KB/s) : min= 1830, max=13120, per=3.32%, avg=6434.32, stdev=215.55
cpu          : usr=3.17%, sys=27.31%, ctx=45466578, majf=0, minf=14990
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=1212673/0/0, short=0/0/0
     lat (usec): 50=0.01%, 100=3.48%, 250=10.92%, 500=15.65%, 750=30.83%
     lat (usec): 1000=18.30%
     lat (msec): 2=17.93%, 4=0.37%, 10=0.04%, 50=0.01%, 100=2.40%
     lat (msec): 250=0.07%

Run status group 0 (all jobs):
   READ: io=18948MB, aggrb=194023KB/s, minb=198680KB/s, maxb=198680KB/s, mint=100002msec, maxt=100002msec

Disk stats (read/write):
xvdb: ios=1212333/304, merge=8552/517, ticks=2757254/2182, in_queue=2759917, util=99.89%
[root@iZ2512hmlr7Z datapump]# fio -filename=/oradata/backup/datapump/1_16_889610461.dbf -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=30 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
...
mytest: (g=0): rw=write, bs=16K-16K/16K-16K, ioengine=psync, iodepth=1
fio 1.57
Starting 30 threads
Jobs: 30 (f=30): [WWWWWWWWWWWWWWWWWWWWWWWWWWWWWW] [100.0% done] [0K/108.3M /s] [0 /6767 iops] [eta 00m:00s]
mytest: (groupid=0, jobs=30): err= 0: pid=9927
write: io=9788.4MB, bw=100223KB/s, iops=6263 , runt=100006msec
    clat (usec): min=99 , max=7609.5K, avg=3901.46, stdev=25102.49
     lat (usec): min=99 , max=7609.5K, avg=3901.66, stdev=25102.49
    bw (KB/s) : min=    2, max=117024, per=14.73%, avg=14767.40, stdev=5994.60
cpu          : usr=1.56%, sys=8.07%, ctx=37860721, majf=0, minf=14556
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/626434/0, short=0/0/0
     lat (usec): 100=0.01%, 250=95.91%, 500=2.69%, 750=0.79%, 1000=0.30%
     lat (msec): 2=0.11%, 4=0.05%, 10=0.01%, 20=0.01%, 100=0.01%
     lat (msec): 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
     lat (msec): >=2000=0.12%

Run status group 0 (all jobs):
WRITE: io=9788.4MB, aggrb=100223KB/s, minb=102628KB/s, maxb=102628KB/s, mint=100006msec, maxt=100006msec

详细参数说明：

Previously, I blogged about setting up my benchmarking machine. Nowthat it's up and running, I've started exploring the fio benchmarking tool.

fio - the Flexible IO Tester is an application written byJens Axboe, who may be better known as the maintainer of the Linux kernel'sblock IO subsystem. It resembles the older ffsb tool in a few ways, but doesn't seem to have any relation to it. Aspower tools go, it's capable of generating pretty much arbitrary load. The tradeoff is that it's difficult to learn andthat's exactly what I've been doing.

Here's a section-by-section breakdown of the default output. I'll look at other output options in future posts. The datadisplayed is from a Samsung 840 Pro SSD.

The explanation for each section can be found below the output text.

  read : io=10240MB, bw=63317KB/s, iops=15829, runt=165607msec

The first line is pretty easy to read. fio did a total of 10GB of IO at 63.317MB/s for a total of 15829 IOPS (at thedefault 4k block size), and ran for 2 minutes and 45 seconds.

The first latency metric you'll see is the 'slat' or submission latency. It is pretty much what it sounds like, meaning"how long did it take to submit this IO to the kernel for processing?"

    slat (usec): min=3, max=335, avg= 9.73, stdev= 5.76

I originally thought that submission latency would be useless for tuning, but the numbers below changed my mind. 269usecor 1/4 of a millisecond seems to be noise, but check it out. I haven't tuned anything yet, so I suspect that changingthe scheduler and telling the kernel it's not a rotating device will help.

Here are some more examples from the other devices:

    slat (usec): min=3, max=335, avg= 9.73, stdev= 5.76 (SATA SSD)
    slat (usec): min=5, max=68,  avg=26.21, stdev= 5.97 (SAS 7200)
    slat (usec): min=5, max=63,  avg=25.86, stdev= 6.12 (SATA 7200)
    slat (usec): min=3, max=269, avg= 9.78, stdev= 2.85 (SATA SSD)
    slat (usec): min=6, max=66,  avg=27.74, stdev= 6.12 (MDRAID0/SAS)

    clat (usec): min=1, max=18600, avg=51.29, stdev=16.79

Next up is completion latency. This is the time that passes between submission to the kernel and when the IO iscomplete, not including submission latency. In older versions of fio, this was the best metric for approximatingapplication-level latency.

     lat (usec): min=44, max=18627, avg=61.33, stdev=17.91

From what I can see, the 'lat' metric is fairly new. It's not documented in the man page or docs. Looking at the Ccode, it seems that this metric starts the moment the IO struct is created in fio and is completed right after clat,making this the one that best represents what applications will experience. This is the one that I will graph.

    clat percentiles (usec):
     |  1.00th=[   42],  5.00th=[   45], 10.00th=[   45], 20.00th=[   46],
     | 30.00th=[   47], 40.00th=[   47], 50.00th=[   49], 60.00th=[   51],
     | 70.00th=[   53], 80.00th=[   56], 90.00th=[   60], 95.00th=[   67],
     | 99.00th=[   78], 99.50th=[   81], 99.90th=[   94], 99.95th=[  101],
     | 99.99th=[  112]

Completion latency percentiles are fairly self-explanatory and probably the most useful bit of info in the output. Ilooked at the source code and this is not slat + clat; it is tracked in its own struct.

The buckets are configurable in the config file. In the terse output, this is 20 fields of %f=%d;%f=%d;... which makesparsing more fun than it should be.

For comparison, here's the same section from a 7200 RPM SAS drive running the exact same load.

    clat percentiles (usec):
     |  1.00th=[ 3952],  5.00th=[ 5792], 10.00th=[ 7200], 20.00th=[ 8896],
     | 30.00th=[10304], 40.00th=[11456], 50.00th=[12608], 60.00th=[13760],
     | 70.00th=[15168], 80.00th=[16768], 90.00th=[18816], 95.00th=[20608],
     | 99.00th=[23424], 99.50th=[24192], 99.90th=[26752], 99.95th=[28032],
     | 99.99th=[30080]

    bw (KB  /s): min=52536, max=75504, per=67.14%, avg=63316.81, stdev=4057.09

Bandwidth is pretty self-explanatory except for the per= part. The docs say it's meant for testing a single devicewith multiple workloads, so you can see how much of the IO was consumed by each process. When fio is run againstmultiple devices, as I did for this output, it doesn't provide much meaning but is amusing when SSDs are mixed withspinning rust.

And here's the SAS drive again with 0.36% of the total IO out of 4 devices being tested.

    bw (KB  /s): min=   71, max=  251, per=0.36%, avg=154.84, stdev=18.29

    lat (usec) :   2= 0.01%,   4=0.01%,  10=0.01%,   20=0.01%, 50=51.41%
    lat (usec) : 100=48.53%, 250=0.06%, 500=0.01%, 1000=0.01%
    lat (msec) :   2= 0.01%,   4=0.01%,  10=0.01%,   20=0.01%

The latency distribution section took me a couple passes to understand. This is one series of metrics. Instead of usingthe same units for all three lines, the third line switches to milliseconds to keep the text width under control. Readthe last line as 2000, 4000, 10,000, and 20,000usec and it makes more sense.

As this is a latency distribution, it's saying that 51.41% of requests took less than 50usec, 48.53% took less than100usec and so on.

    lat (msec) : 4=1.07%, 10=27.04%, 20=65.43%, 50=6.46%, 100=0.01%

In case you were thinking of parsing this madness with a quick script, you might want to know that the lat section willomit entries and whole lines if there is no data. For example, the SAS drive I've been referencing didn't manage to doany IO faster than a millisecond, so this is the only line.

  cpu          : usr=5.32%, sys=21.95%, ctx=2829095, majf=0, minf=21

Here's the user/system CPU percentages followed by context switches then major and minorpagefaults. Since the test isconfigured to use direct IO, there should be very few page faults.

  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0%

Fio has an iodepth setting that controls how many IOs it issues to the OS at any given time. This is entirelyapplication-side, meaning it is not the same thing as the device's IO queue. In this case, iodepth was set to 1 so theIO depth was always 1 100% of the time.

     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0%

submit and complete represent the number of submitted IOs at a time by fio and the number completed at a time. In thecase of the thrashing test used to generate this output, the iodepth is at the default value of 1, so 100% of IOs weresubmitted 1 at a time placing the results in the 1-4 bucket. Basically these only matter if iodepth is greater than 1.

These will get much more interesting when I get around to testing the various schedulers.

     issued    : total=r=2621440/w=0/d=0, short=r=0/w=0/d=0

The number of IOs issued. Something is weird here since this was a 50/50 read/write load, so there should have been anequal number of writes. I suspect havingunified_rw_reportingenabled is making fio count all IOs as reads.

If you see short IOs in a direct IO test something has probably gone wrong. The reference I found in theLinux kernelindicates that this happens at EOF and likely end of device.

     latency   : target=0, window=0, percentile=100.00%, depth=1

Fio can be configured with a latency target, which will cause it to adjust throughput until it can consistently hit theconfigured latency target. I haven't messed with this much yet. In time or size-based tests, this line will always lookthe same. All four of these values represent the configuration settingslatency_target,latency_window,latency_percentile,andiodepth.

Run status group 0 (all jobs):

fio supports grouping different tests for aggregation. For example, I can have one config for SSDs and HDDs mixed in thesame file, but set up groups to report the IO separately. I'm not doing this for now, but future configs will need thisfunctionality.

  MIXED: io=12497MB, aggrb=42653KB/s, minb=277KB/s, maxb=41711KB/s, mint=300000msec, maxt=300012msec

And finally, the total throughput and time. io= indicates the amount of IO done in total. It will be variable for timedtests and should match thesizeparameter for sized tests. aggrb is the aggregate bandwidth across all processes / devices. minb/maxb showminimum/maximum observed bandwidth. mint/maxt show the shortest & longest times for tests.Similar to the io= parameter, these should match theruntime parameter fortime-based tests and will vary in size-based tests.

Since I ran this test withunified_rw_reporting enabled,we only see a line for MIXED. If it's disabled there will be separate lines for READ and WRITE.

Simple, right? I'll be spending a lot more time with fio for the next few weeks and will post more examples of configs,output, and graphing code.