Fio for I/O performance testing

Fio is short for Flexible I/O, a versatile IO workload generator. It was original developed by Jens Axboe, the backbone behind and author of the IO stack in the Linux kernel.

• It can be used to collect all sorts of I/O performance information, including complete IO latencies and percentiles. 
• It supports different types of I/O engines (e.g., sync, mmap, libaio, posixaio, network, etc), rate I/O, forked or threaded jobs.
• It provides many options to define different workloads (e.g., random and sequential IO mix, or read/write mix).

Install and Run fio

1. Make sure libaio-dev / libaio-devel / libaio is installed.
2. Use apt-get to install fio on Ubuntu; or download source code of the latest version from GitHub (Link), and build & install it.

3. There are two ways to run fio: 1) passing a config file specifying workload options; 2) passing workload options as command line parameters. The following shows a simple example, where we create a config file random_read.fio.

   1 2 3 4 5 6 7 8

   #Job name between brackets [random_read]   #Random Read test rw=randread   #Transfer 1024MB data size=1024m

4. Run testing using this configure file, which will test the I/O performance for randomly reading 1024MB data under current directory (other unspecified options remains default)

   fio ./random_read.fio

   This is the same as the following, where options are specified in command line.

   fio --name=rand_read --rw=randread --size=1024m

5. The output will be shown as follows. It displays plenty of performance information (e.g., bw=20972KB/s, iops=5242). The interpretation of output can be found here.

   random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.5-19-gdc63 Starting 1 process random-read: Laying out IO file(s) (1 file(s) / 1024MB) Jobs: 1 (f=1): [r] [100.0% done] [23092KB/0KB/0KB /s] [5773/0/0 iops] [eta 00m:00s] random-read: (groupid=0, jobs=1): err= 0: pid=17458: Mon Feb 24 17:31:03 2014    read : io=1024.0MB, bw=20972KB/s, iops=5242, runt= 50000msec      clat (usec): min=104, max=6026, avg=186.75, stdev=68.49       lat (usec): min=104, max=6026, avg=187.07, stdev=68.49      clat percentiles (usec):       |  1.00th=[  125],  5.00th=[  161], 10.00th=[  167], 20.00th=[  169],       | 30.00th=[  177], 40.00th=[  181], 50.00th=[  183], 60.00th=[  187],       | 70.00th=[  191], 80.00th=[  195], 90.00th=[  203], 95.00th=[  207],       | 99.00th=[  258], 99.50th=[  330], 99.90th=[ 1080], 99.95th=[ 1512],       | 99.99th=[ 2480]      bw (KB  /s): min=11712, max=22840, per=99.87%, avg=20943.90, stdev=1858.80      lat (usec) : 250=98.69%, 500=0.95%, 750=0.13%, 1000=0.11%      lat (msec) : 2=0.09%, 4=0.03%, 10=0.01%    cpu          : usr=2.74%, sys=16.74%, ctx=263863, majf=0, minf=22    IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%       submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%       complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%       issued    : total=r=262144/w=0/d=0, short=r=0/w=0/d=0       latency   : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs):     READ: io=1024.0MB, aggrb=20971KB/s, minb=20971KB/s, maxb=20971KB/s, mint=50000msec, maxt=50000msec

6. The following config file includes some more useful options. Much more options can be found in the man page, and fio source code also provides several examples.

   # job name between brackets (except when value is "global" ) [random_read]   #rw= #   read        Sequential reads #   write       Sequential writes #   randwrite   Random writes #   randread    Random reads #   rw          Sequential mixed reads and writes #   randrw      Random mixed reads and writes   rw=randread # ioengine= #    sync       Basic read(2) or write(2) io. lseek(2) is #               used to position the io location. #    psync      Basic pread(2) or pwrite(2) io. #    vsync      Basic readv(2) or writev(2) IO. #    libaio     Linux native asynchronous io. #    posixaio   glibc posix asynchronous io. #    solarisaio Solaris native asynchronous io. #    windowsaio Windows native asynchronous io. ioengine=libaio   #iodepth Number of I/O units to keep in flight against the file. iodepth=32   # direct If value is true, use non-buffered io. This is usually #        O_DIRECT. Note that ZFS on Solaris doesn't support direct io. direct=1   #invalidate Invalidate buffer-cache for the file prior to starting I/O. Default: true. invalidate=1   # bs The block size used for the io units. Defaults to 4k. bs=4k   #directory Prefix filenames with this directory. Used to place files in a location other than './'. directory=. /data   #nrfiles= Number of files to use for this job. Defaults to 1. nrfiles=1   #size Total size of I/O for this job. size=200m   #numjobs Number of clones (processes/threads performing the same workload) of this job. Default: 1. numjobs=4

Preconditioning

Preconditioning is important to get steady performance data, especially for write. When the SSD drive is in a clean state, the write operation is fast (even higher than read performance). However, when garbage collector starts to work due to low available capacity, the write performance will go down, and reach a steady state. We can see an example here (The graph of interest is the "preconditioning curve"). (Refer to SNIA Solid State Storage Performance Test Specification, discussion-1, discussion-2)

Useful links:

• fio GitHub
• fio HOWTO
• fio Man Page
• Inspecting disk IO performance with fio
• The truth about SSD performance benchmarks
• Many SSD Benchmark Reviews Contain Flaws