btrfs vs ext4 vs (zfs on freebsd) iozone-CSDN博客

之前测试了btrfs和ext4的fsync接口性能对比，再用iozone测试一下综合性能。

http://blog.163.com/digoal@126/blog/static/163877040201510301944705/

http://blog.163.com/digoal@126/blog/static/1638770402015103004729831/

测试case:
2G文件，单个请求32K到16MB分布，测试所有的iozone case，全程使用O_DIRECT。
[root@digoal current]# ./iozone -Ra -I -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks

详细的参数请参考iozone -h

测试结果如下

ext4:

[root@digoal ~]# mount /dev/sdb1 /data01 -o defaults,noatime,nodiratime,discard,data=ordered

[root@digoal current]# ./iozone -Ra -I -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks
        Iozone: Performance Test of File I/O
                Version $Revision: 3.434 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa,
                     Alexey Skidanov.

        Run began: Mon Nov 30 01:47:40 2015

        Excel chart generation enabled
        Auto Mode
        O_DIRECT feature enabled
        CPU utilization Resolution = 0.000 seconds.
        CPU utilization Excel chart enabled
        Using Minimum Record Size 32 kB
        Using Maximum Record Size 16384 kB
        Using maximum file size of 2097152 kilobytes.
        Using minimum file size of 2097152 kilobytes.
        Command line used: ./iozone -Ra -I -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152      32   134418   137172   375562   384119    86187   136983    88777    121142     85634   928787  1422844  7479780  8053423
         2097152      64   134167   202202   468653   460789   126486   201188   122861    197159    126381  1383822  1105178  7725882  8586657
         2097152     128    76223   246728   478150   476280   132230   232430   133712    218963    141642  1270394  1430517  7054489  7655208
         2097152     256    90344   278123   470824   463306   201172   269974   202418    255450    205696  1358647  1442362  6553250  5856184
         2097152     512   248710   278122   480782   477937   249097   285558   255350    156027    244695  1386226  1447207  6709274  7163894
         2097152    1024   323825   324080   486221   483573   330679   322758   323153    255837    331737  1375727  1444986  6635777  7530193
         2097152    2048   346527   347812   456888   473026   362050   347068   386283    353686    369822  1335331  1428099  5584289  5337212
         2097152    4096   359385   365783   477321   437269   424284   356512   425930    373190    423774  1335068  1429786  5250308  5224279
         2097152    8192   381589   379722   457510   473173   443560   384028   441171    380177    443229  1601698  1416931  5054927  5087076
         2097152   16384   360667   370847   456841   465093   458894   380607   455677    390324    460216  1357927  1439198  4670778  5244898

iozone test complete.
Excel output is below:

"Writer report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   134418  134167  76223  90344  248710  323825  346527  359385  381589  360667 

"Re-writer report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   137172  202202  246728  278123  278122  324080  347812  365783  379722  370847 

"Reader report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   375562  468653  478150  470824  480782  486221  456888  477321  457510  456841 

"Re-Reader report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   384119  460789  476280  463306  477937  483573  473026  437269  473173  465093 

"Random read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   86187  126486  132230  201172  249097  330679  362050  424284  443560  458894 

"Random write report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   136983  201188  232430  269974  285558  322758  347068  356512  384028  380607 

"Backward read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   88777  122861  133712  202418  255350  323153  386283  425930  441171  455677 

"Record rewrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   121142  197159  218963  255450  156027  255837  353686  373190  380177  390324 

"Stride read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   85634  126381  141642  205696  244695  331737  369822  423774  443229  460216 

"Fwrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   928787  1383822  1270394  1358647  1386226  1375727  1335331  1335068  1601698  1357927 

"Re-Fwrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   1422844  1105178  1430517  1442362  1447207  1444986  1428099  1429786  1416931  1439198 

"Fread report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   7479780  7725882  7054489  6553250  6709274  6635777  5584289  5250308  5054927  4670778 

"Re-Fread report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   8053423  8586657  7655208  5856184  7163894  7530193  5337212  5224279  5087076  5244898 

"Writer CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    30.53  60.79  89.21  92.41  56.59  19.67  19.63  20.76  17.98  16.37

"Re-writer CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    26.89  21.81  17.84  16.17  16.59  19.02  17.47  19.12  18.79  15.68

"Reader CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    67.65  37.39  19.16  14.72  25.13  24.53  28.76  26.52  32.65  34.26

"Re-Reader CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    68.50  38.06  19.45  14.53  21.60  24.35  27.76  30.61  30.78  33.09

"Random read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    17.01  14.16  15.05  13.30  17.93  19.12  20.03  23.06  30.26  33.92

"Random write CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    26.84  23.07  19.57  15.66  16.07  17.82  16.73  18.39  16.56  22.41

"Backward read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    16.87  14.43  14.58  12.16  16.66  17.02  21.45  22.85  30.40  33.42

"Record rewrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    27.38  22.68  20.13  17.25  15.92  19.72  22.78  22.70  21.70  23.66

"Stride read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    18.18  13.91  12.72  12.13  14.51  18.89  20.04  23.34  31.29  35.46

"Fwrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    20.11  22.00  25.52  21.41  20.77  23.36  24.63  24.11  26.59  24.78

"Re-Fwrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    17.98  17.55  18.95  17.68  17.47  20.64  20.77  20.43  20.93  19.63

"Fread CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00

"Re-Fread CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00

btrfs:

[root@digoal ~]# mkfs.btrfs /dev/sdb1 -m single -n 4096 -f
[root@digoal ~]# mount /dev/sdb1 /data01 -o ssd,discard,nodatacow,noatime,nodiratime,compress=no,space_cache,ssd_spread

[root@digoal current]# ./iozone -Ra -I -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks
        Iozone: Performance Test of File I/O
                Version $Revision: 3.434 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa,
                     Alexey Skidanov.

        Run began: Mon Nov 30 02:13:32 2015

        Excel chart generation enabled
        Auto Mode
        O_DIRECT feature enabled
        CPU utilization Resolution = 0.000 seconds.
        CPU utilization Excel chart enabled
        Using Minimum Record Size 32 kB
        Using Maximum Record Size 16384 kB
        Using maximum file size of 2097152 kilobytes.
        Using minimum file size of 2097152 kilobytes.
        Command line used: ./iozone -Ra -I -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152      32   123755   124696   266012   259530    81904   118866    85865    125944     81506  1000642  1223436  7518993  7328625
         2097152      64   181575   193539   467478   470595   115328   191047   114627    177387    118170  1129998  1199197  7728930  8435097
         2097152     128   240240   219705   470595   477346   139620   229685   142870    180770    144397  1210305  1378005  7879644  7726370
         2097152     256   286119   274818   482720   475115   185018   260297   195585    240211    189769  1349486  1422568  6938493  7246275
         2097152     512   297994   285457   473885   475142   258003   283346   246033    152852    233404  1432865  1484320  6834497  7239470
         2097152    1024   333364   334765   476736   475768   311638   322389   305667    254151    306960  1413879  1421603  5366004  6433693
         2097152    2048   301161   346783   463064   448675   362464   337696   361640    352503    351189  1305534  1429446  5181695  5897865
         2097152    4096   359236   359397   456855   458875   410763   356342   406654    366979    407723  1404694  1485010  5273213  5358528
         2097152    8192   370117   377368   447627   435809   421917   377948   424365    374778    427770  1374963  1478975  5108439  5320885
         2097152   16384   394523   391426   443925   444696   426283   396330   451946    392762    437817  1410894  1472782  4814077  5291389

iozone test complete.
Excel output is below:

"Writer report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   123755  181575  240240  286119  297994  333364  301161  359236  370117  394523 

"Re-writer report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   124696  193539  219705  274818  285457  334765  346783  359397  377368  391426 

"Reader report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   266012  467478  470595  482720  473885  476736  463064  456855  447627  443925 

"Re-Reader report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   259530  470595  477346  475115  475142  475768  448675  458875  435809  444696 

"Random read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   81904  115328  139620  185018  258003  311638  362464  410763  421917  426283 

"Random write report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   118866  191047  229685  260297  283346  322389  337696  356342  377948  396330 

"Backward read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   85865  114627  142870  195585  246033  305667  361640  406654  424365  451946 

"Record rewrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   125944  177387  180770  240211  152852  254151  352503  366979  374778  392762 

"Stride read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   81506  118170  144397  189769  233404  306960  351189  407723  427770  437817 

"Fwrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   1000642  1129998  1210305  1349486  1432865  1413879  1305534  1404694  1374963  1410894 

"Re-Fwrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   1223436  1199197  1378005  1422568  1484320  1421603  1429446  1485010  1478975  1472782 

"Fread report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   7518993  7728930  7879644  6938493  6834497  5366004  5181695  5273213  5108439  4814077 

"Re-Fread report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   7328625  8435097  7726370  7246275  7239470  6433693  5897865  5358528  5320885  5291389 

"Writer CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    31.13  32.56  24.07  20.98  19.19  19.54  16.23  22.89  15.82  20.52

"Re-writer CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    36.68  30.88  21.13  21.15  21.10  19.30  19.13  22.00  16.95  22.20

"Reader CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    67.37  63.95  37.04  24.60  29.54  29.30  28.48  30.28  34.70  40.93

"Re-Reader CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    67.94  63.09  37.30  25.44  28.45  27.81  28.87  28.37  32.42  41.46

"Random read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    22.26  19.14  17.18  18.82  18.37  21.10  22.86  25.40  34.15  39.66

"Random write CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    36.74  31.76  24.97  21.75  20.55  18.32  20.79  19.50  17.83  19.63

"Backward read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    22.53  19.80  15.23  18.39  20.34  19.65  24.16  25.13  34.62  42.35

"Record rewrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    36.96  30.94  24.51  20.80  20.21  23.24  22.76  25.37  24.11  28.44

"Stride read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    24.96  18.72  15.47  18.05  22.34  20.85  24.63  25.26  34.36  40.93

"Fwrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    37.11  24.38  18.98  19.67  19.13  22.92  25.49  24.95  23.87  22.91

"Re-Fwrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    26.77  17.46  17.38  17.31  17.74  17.80  20.47  20.07  20.45  19.73

"Fread CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    98.65  99.79  99.96  99.83  99.72  98.37  99.87  99.97  98.98  99.81

"Re-Fread CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   100.00 100.00 100.00 100.00  99.97  99.88 100.00 100.00 100.00 100.00

解释 :

Write: 测试向一个新文件写入的性能。当一个新文件被写入时，不仅仅是那些文件中的数据需要被存储，还包括那些用于定位数据存储在存储介质的具体位置的额外信息。这些额外信息被称作“元数据”。它包括目录信息，所分配的空间和一些与该文件有关但又并非该文件所含数据的其他数据。拜这些额外信息所赐，Write的性能通常会比Re-write的性能低。

Re-write: 测试向一个已存在的文件写入的性能。当一个已存在的文件被写入时，所需工作量较少，因为此时元数据已经存在。Re-write的性能通常比Write的性能高。

Read: 测试读一个已存在的文件的性能。

Re-Read: 测试读一个最近读过的文件的性能。Re-Read性能会高些，因为操作系统通常会缓存最近读过的文件数据。这个缓存可以被用于读以提高性能。

Random Read: 测试读一个文件中的随机偏移量的性能。许多因素都可能影响这种情况下的系统性能，例如：操作系统缓存的大小，磁盘数量，寻道延迟和其他。

Random Write: 测试写一个文件中的随机偏移量的性能。同样，有许多因素可能影响这种情况下的系统性能，例如：操作系统缓存的大小，磁盘数量，寻道延迟和其他。

Random Mix: 测试读写一个文件中的随机偏移量的性能。许多因素可能影响这种情况下的系统性能运作，例如：操作系统缓存的大小，磁盘数量，寻道延迟和其他。这个测试只有在吞吐量测试模式下才能进行。每个线程/进程运行读或写测试。这种分布式读/写测试是基于round robin 模式的。最好使用多于一个线程/进程执行此测试。

Backwards Read: 测试使用倒序读一个文件的性能。这种读文件方法可能看起来很可笑，事实上，有些应用确实这么干。MSC Nastran是一个使用倒序读文件的应用程序的一个例子。它所读的文件都十分大（大小从G级别到T级别）。尽管许多操作系统使用一些特殊实现来优化顺序读文件的速度，很少有操作系统注意到并增强倒序读文件的性能。

Record Rewrite: 测试写与覆盖写一个文件中的特定块的性能。这个块可能会发生一些很有趣的事。如果这个块足够小（比CPU数据缓存小），测出来的性能将会非常高。如果比CPU数据缓存大而比TLB小，测出来的是另一个阶段的性能。如果比此二者都大，但比操作系统缓存小，得到的性能又是一个阶段。若大到超过操作系统缓存，又是另一番结果。

Strided Read: 测试跳跃读一个文件的性能。举例如下：在0偏移量处读4Kbytes，然后间隔200Kbytes,读4Kbytes，再间隔200Kbytes，如此反复。此时的模式是读4Kbytes，间隔200Kbytes并重复这个模式。这又是一个典型的应用行为，文件中使用了数据结构并且访问这个数据结构的特定区域的应用程序常常这样做。

许多操作系统并没注意到这种行为或者针对这种类型的访问做一些优化。同样，这种访问行为也可能导致一些有趣的性能异常。一个例子是在一个数据片化的文件系统里，应用程序的跳跃导致某一个特定的磁盘成为性能瓶颈。

Fwrite: 测试调用库函数fwrite()来写文件的性能。这是一个执行缓存与阻塞写操作的库例程。缓存在用户空间之内。如果一个应用程序想要写很小的传输块，fwrite()函数中的缓存与阻塞I/O功能能通过减少实际操作系统调用并在操作系统调用时增加传输块的大小来增强应用程序的性能。

这个测试是写一个新文件，所以元数据的写入也是要的。

Frewrite:测试调用库函数fwrite()来写文件的性能。这也是一个执行缓存与阻塞写操作的库例程。是缓存在用户空间之内。如果一个应用程序想要写很小的传输块，fwrite()函数中的缓存与阻塞I/O功能可以通过减少实际操作系统调用并在操作系统调用时增加传输块的大小来增强应用程序的性能。

这个测试是写入一个已存在的文件，由于无元数据操作，测试的性能会高些。

Fread:测试调用库函数fread()来读文件的性能。这是一个执行缓存与阻塞读操作的库例程。缓存在用户空间之内。如果一个应用程序想要读很小的传输块，fwrite()函数中的缓存与阻塞I/O功能能通过减少实际操作系统调用并在操作系统调用时增加传输块的大小从而增强应用程序的性能。

几个特殊测试:

Mmap:许多操作系统支持mmap()的使用来映射一个文件到用户地址空间。映射之后,对内存的读写将同步到文件中去。这对一些希望将文件当作内存块来使用的应用程序来说很方便。一个例子是内存中的一块将同时作为一个文件保存在于文件系统中。

性能图表:

btrfs vs ext4 iozone - 德哥@Digoal - PostgreSQL research

最后加一个ZFS在freebsd下的测试数据：

可以明显看出cpu利用率比较低，IO这块和EXT4相比，除了record rewrite，其他都是ext4胜出一大截。

[root@digoal ~]# zpool create -o ashift=13 zp1 /dev/sdb1 log /dev/sdb2 -f

[root@digoal ~]# zfs create -o atime=off -o canmount=on -o compression=off -o dedup=off -o mountpoint=/data01 -o nbmand=off -o primarycache=metadata -o recordsize=128k -o secondarycache=none zp1/data01

[root@digoal ~]# zfs set atime=off zp1
[root@digoal ~]# zfs set canmount=on zp1
[root@digoal ~]# zfs set compression=off zp1
[root@digoal ~]# zfs set dedup=off zp1
[root@digoal ~]# zfs set mountpoint=/data01 zp1
[root@digoal ~]# zfs set nbmand=off zp1
[root@digoal ~]# zfs set primarycache=metadata zp1
[root@digoal ~]# zfs set recordsize=128k zp1
[root@digoal ~]# zfs set secondarycache=none zp1 zp1

root@:~/iozone3_434/src/current # ./iozone -Ra -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks
        Iozone: Performance Test of File I/O
                Version $Revision: 3.434 $
                Compiled for 64 bit mode.
                Build: freebsd 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                     Vangel Bojaxhi, Ben England, Vikentsi Lapa,
                     Alexey Skidanov.

        Run began: Tue Dec  1 10:36:48 2015

        Excel chart generation enabled
        Auto Mode
        CPU utilization Resolution = 0.000 seconds.
        CPU utilization Excel chart enabled
        Using Minimum Record Size 32 kB
        Using Maximum Record Size 16384 kB
        Using maximum file size of 2097152 kilobytes.
        Using minimum file size of 2097152 kilobytes.
        Command line used: ./iozone -Ra -+u -y 32k -q 16m -g 2G -n 2G -f /data01/test -b /data01/test.wks
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         2097152      32   325355   107915   143338   149964    35394    15052    74398   3285408     35786    60713    90725   140505   141348
         2097152      64   176581    95661   253717   249525    72847    30346   105825   6446013     73071    57899    88224   234365   245531
         2097152     128   179099   165956   341052   342761   140638   117313   142163   9670310    144940   113754   175801   307036   315640
         2097152     256   186827   161277   329778   326657   177880   177567   180326   9290488    177305   165554   195725   353961   328730
         2097152     512   195290   200283   334442   337011   234452   196693   244377   9129760    241425   204185   196523   340597   341560
         2097152    1024   198035   211504   374475   372747   295725   203285   293102   8605003    300160   197968   178176   347560   343489
         2097152    2048   194653   197203   357659   366141   289980   189969   313109   6626219    284537   195303   190947   300276   249390
         2097152    4096   164893   193768   344436   345652   338939   191322   331562   5749641    325459   180384   175225   299436   284174
         2097152    8192   175295   170895   375246   361781   335327   192435   312687   5086840    334125   172890   176487   332262   334838
         2097152   16384   196395   188921   377546   373273   341244   167365   339940   5508472    333385   187974   204265   318357   326438

iozone test complete.
Excel output is below:

"Writer report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   325355  176581  179099  186827  195290  198035  194653  164893  175295  196395 

"Re-writer report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   107915  95661  165956  161277  200283  211504  197203  193768  170895  188921 

"Reader report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   143338  253717  341052  329778  334442  374475  357659  344436  375246  377546 

"Re-Reader report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   149964  249525  342761  326657  337011  372747  366141  345652  361781  373273 

"Random read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   35394  72847  140638  177880  234452  295725  289980  338939  335327  341244 

"Random write report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   15052  30346  117313  177567  196693  203285  189969  191322  192435  167365 

"Backward read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   74398  105825  142163  180326  244377  293102  313109  331562  312687  339940 

"Record rewrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   3285408  6446013  9670310  9290488  9129760  8605003  6626219  5749641  5086840  5508472 

"Stride read report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   35786  73071  144940  177305  241425  300160  284537  325459  334125  333385 

"Fwrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   60713  57899  113754  165554  204185  197968  195303  180384  172890  187974 

"Re-Fwrite report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   90725  88224  175801  195725  196523  178176  190947  175225  176487  204265 

"Fread report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   140505  234365  307036  353961  340597  347560  300276  299436  332262  318357 

"Re-Fread report"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"   141348  245531  315640  328730  341560  343489  249390  284174  334838  326438 

"Writer CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    21.04   6.19   4.49   4.77   4.67   4.68   4.71   5.58   4.64   5.68

"Re-writer CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    26.32  26.22   4.63   4.73   4.64   6.56   6.44   6.04   4.86   5.69

"Reader CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    44.32  43.68  37.44  39.82  41.78  47.55  48.21  48.48  45.13  47.72

"Re-Reader CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    44.29  42.93  37.62  41.01  42.61  48.50  47.30  49.17  47.40  46.61

"Random read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    13.30  13.06  14.39  23.61  27.35  31.87  46.36  43.20  46.99  50.72

"Random write CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    17.06  16.84   3.76   4.63   5.91   6.55   4.71   6.24   6.50   5.93

"Backward read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    30.55  24.54  15.15  23.72  28.88  36.62  42.17  44.60  48.48  46.37

"Record rewrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    99.88  99.92 100.00 100.00  99.99 100.00 100.00  98.80  99.32  99.99

"Stride read CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    13.12  12.91  13.04  22.87  26.49  33.25  44.24  45.54  47.13  48.65

"Fwrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    17.20  16.21   3.02   3.65   4.88   5.32   5.41   5.76   5.94   6.12

"Re-Fwrite CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    25.29  24.17   5.23   5.30   4.98   4.41   5.58   5.66   5.70   7.13

"Fread CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    44.44  45.43  38.57  40.53  48.28  52.60  57.92  56.53  51.43  51.20

"Re-Fread CPU utilization report (Zero values should be ignored)"
        "32"  "64"  "128"  "256"  "512"  "1024"  "2048"  "4096"  "8192"  "16384"
"2097152"    45.17  44.55  40.81  44.92  48.16  54.15  62.99  57.99  51.86  53.01

[参考]

http://baike.baidu.com/item/iozone

https://btrfs.wiki.kernel.org/index.php/Mount_options

http://www.iozone.org/docs/IOzone_msword_98.pdf