大数据2.4T 磁盘HDD io 测试

最新推荐文章于 2024-04-25 19:41:00 发布

陶武杰

最新推荐文章于 2024-04-25 19:41:00 发布

阅读量892

点赞数

分类专栏：服务器基础操作 raid 文章标签：运维

本文链接：https://blog.csdn.net/weixin_43423965/article/details/128562890

版权

服务器基础操作同时被 2 个专栏收录

9 篇文章 0 订阅

订阅专栏

raid

3 篇文章 0 订阅

订阅专栏

大数据2.4T 磁盘HDD io 测试

一. FIO介绍

FIO是测试IOPS的非常好的工具，用来对磁盘进行压力测试和验证。磁盘IO是检查磁盘性能的重要指标，可以按照负载情况分成照顺序读写，随机读写两大类。FIO是一个可以产生很多线程或进程并执行用户指定的特定类型I/O操作的工具，FIO的典型用途是编写和模拟的I/O负载匹配的作业文件。也就是说FIO 是一个多线程io生成工具，可以生成多种IO模式，用来测试磁盘设备的性能（也包含文件系统：如针对网络文件系统 NFS 的IO测试）。fio在github上的坐标：https://github.com/axboe/fio 。

1.1. FIO 工具常用参数：

参数说明：

filename=/dev/sdb1 测试文件名称，通常选择需要测试的盘的data目录。
direct=1 是否使用directIO，测试过程绕过OS自带的buffer，使测试磁盘的结果更真实。Linux读写的时候，内核维护了缓存，数据先写到缓存，后面再后台写到SSD。读的时候也优先读缓存里的数据。这样速度可以加快，但是一旦掉电缓存里的数据就没了。所以有一种模式叫做DirectIO，跳过缓存，直接读写SSD。 
rw=randwrite 测试随机写的I/O
rw=randrw 测试随机写和读的I/O
bs=16k 单次io的块文件大小为16k
bsrange=512-2048 同上，提定数据块的大小范围
size=5G 每个线程读写的数据量是5GB。
numjobs=1 每个job（任务）开1个线程，这里用了几，后面每个用-name指定的任务就开几个线程测试。所以最终线程数=任务数（几个name=jobx）* numjobs。 
name=job1：一个任务的名字，重复了也没关系。如果fio -name=job1 -name=job2，建立了两个任务，共享-name=job1之前的参数。-name之后的就是job2任务独有的参数。 
thread  使用pthread_create创建线程，另一种是fork创建进程。进程的开销比线程要大，一般都采用thread测试。 
runtime=1000 测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止。
ioengine=libaio 指定io引擎使用libaio方式。libaio：Linux本地异步I/O。请注意，Linux可能只支持具有非缓冲I/O的排队行为（设置为“direct=1”或“buffered=0”）；rbd:通过librbd直接访问CEPH Rados 
iodepth=16 队列的深度为16.在异步模式下，CPU不能一直无限的发命令到SSD。比如SSD执行读写如果发生了卡顿，那有可能系统会一直不停的发命令，几千个，甚至几万个，这样一方面SSD扛不住，另一方面这么多命令会很占内存，系统也要挂掉了。这样，就带来一个参数叫做队列深度。
Block Devices（RBD），无需使用内核RBD驱动程序（rbd.ko）。该参数包含很多ioengine，如：libhdfs/rdma等
rwmixwrite=30 在混合读写的模式下，写占30%
group_reporting 关于显示结果的，汇总每个进程的信息。
此外
lockmem=1g 只使用1g内存进行测试。
zero_buffers 用0初始化系统buffer。
nrfiles=8 每个进程生成文件的数量。
磁盘读写常用测试点：

1. Read=100% Ramdon=100% rw=randread (100%随机读)
2. Read=100% Sequence=100% rw=read （100%顺序读）
3. Write=100% Sequence=100% rw=write （100%顺序写）
4. Write=100% Ramdon=100% rw=randwrite （100%随机写）
5. Read=70% Sequence=100% rw=rw, rwmixread=70, rwmixwrite=30
   （70%顺序读，30%顺序写）
6. Read=70% Ramdon=100% rw=randrw, rwmixread=70, rwmixwrite=30
   (70%随机读，30%随机写)

1.2. fio 安装:

yum -y install fio

二. 大数据磁盘压力测试：

大数据新旧磁盘大小均为2.4T HDD 磁盘，其中/data1 -> /data7 为旧磁盘,/data8 -> /data15 为新磁盘.

在这里插入图片描述

针对新旧磁盘/data7,/data15 使用fio命令分别进行随机/顺序io进行io测试.

命令案例:

fio -filename=/data7/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在/data7/创建测试文件,direct直接读取,去除bufer/cache, io深度1,模式为xx,bs块未4k,size大小10G,运行时间100s,jobs数量80

2.1 随机读写测试

测试命令:

fio -filename=/data7/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在这里插入图片描述

2.2 随机读对比

测试命令:

fio -filename=/data7/randr -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在这里插入图片描述

2.3 随机写:

测试命令:

fio -filename=/data7/randw -direct=1 -iodepth 1 -thread -rw=randwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在这里插入图片描述

2.4 顺序读写:

测试命令:

fio -filename=/data7/rw -direct=1 -iodepth 1 -thread -rw=readwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在这里插入图片描述

2.5 顺序读:

测试命令:

fio -filename=/data7/r -direct=1 -iodepth 1 -thread -rw=read -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在这里插入图片描述

2.6 顺序写:

测试命令:

fio -filename=/data7/w -direct=1 -iodepth 1 -thread -rw=write -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest

在这里插入图片描述

三. 综合性能对比

磁盘	测试类型	测试块大小	iops读	iops写	读速率	写速率	磁盘io使用百分比
/data7 [旧盘]	随机读写	4k	753	86	3087kB/s	354kB/s	100%
/data15 [新盘]	随机读写	4k	762	87	3123kB/s	258kB/s	100%
/data7 [旧盘]	随机读	4k	918		3762kB/s		99%
/data15 [新盘]	随机读	4k	880		3608kB/s		100%
/data7 [旧盘]	随机写	4k		1310		5366kB/s	100%
/data15 [新盘]	随机写	4k		1337		5478kB/s	100%
/data7 [旧盘]	顺序读写	4k	15.6k	1737	63.9MB/s	7117kB/s	100%
/data15 [新盘]	顺序读写	4k	15.0k	1780	65.4MB/s	7295kB/s	100%
/data7 [旧盘]	顺序读	4k	14.1K		57.7MB/s		100%
/data15 [新盘]	顺序读	4k	15.2K		62.1MB/s	-	100%
/data7 [旧盘]	顺序写	4k		88.1k		361MB/s	83.35%
/data15 [新盘]	顺序写	4k		111k		456MB/s	100%

在这里插入图片描述

io性能分随机读写与顺序读写:

随机读写性能测试中,新旧磁盘差距不大,新硬盘略低于老硬盘,随机读写一直是磁盘io性能瓶颈问题.

顺序读写中,新盘读写性能略高于旧盘.

整体来言,新旧磁盘性能在合理访问内.

PS: 本次测试磁盘均在正常使用,测试结果可能受制于磁盘可用空间,当前磁盘io使用量等.可能出现偏差.

四. 附 io 测试详情：

[root@bj-wlj-datanode09-172-38-1-210 ~]# # old disk randrw
[root@bj-wlj-datanode09-172-38-1-210 ~]# df
文件系统                                           1K-块       已用       可用 已用% 挂载点
devtmpfs                                       164794756          0  164794756    0% /dev
tmpfs                                          164806764          4  164806760    1% /dev/shm
tmpfs                                          164806764      27276  164779488    1% /run
tmpfs                                          164806764          0  164806764    0% /sys/fs/cgroup
/dev/mapper/centos-root                         52403200   17991632   34411568   35% /
/dev/sda1                                        1038336     153596     884740   15% /boot
/dev/mapper/vg--ssd--g1--data1-lv--ssd--data1 1560821124 1202231684  358589440   78% /data1
/dev/mapper/vg--hdd--data4-lv--hdd--data4     2342418948 1829166600  513252348   79% /data4
/dev/mapper/vg--hdd--data6-lv--hdd--data6     2342418948 1817236892  525182056   78% /data6
/dev/mapper/vg--hdd--data5-lv--hdd--data5     2342418948 1819285400  523133548   78% /data5
/dev/mapper/vg--hdd--data2-lv--hdd--data2     2342418948 1830784580  511634368   79% /data2
/dev/mapper/vg--hdd--data7-lv--hdd--data7     2342418948 1825019024  517399924   78% /data7
/dev/mapper/centos-lv_data                     419225600      45632  419179968    1% /data
cm_processes                                   164806764      65632  164741132    1% /run/cloudera-scm-agent/process
tmpfs                                           32961356          0   32961356    0% /run/user/1001
/dev/sdk1                                     2146434048  165583628 1980850420    8% /data9
/dev/sdl1                                     2146434048  163279364 1983154684    8% /data10
/dev/sdm1                                     2146434048  163748512 1982685536    8% /data11
/dev/sdn1                                     2146434048  164037888 1982396160    8% /data12
/dev/sdo1                                     2146434048  169086524 1977347524    8% /data13
/dev/sdp1                                     2146434048  166700816 1979733232    8% /data14
/dev/sdq1                                     2146434048  165240372 1981193676    8% /data15
/dev/sdj1                                     2146434048  165038420 1981395628    8% /data8
/dev/mapper/vg--hdd--data3-lv--hdd--data3     2342418948 1289470652 1052948296   56% /data3
tmpfs                                           32961356          0   32961356    0% /run/user/0
tmpfs                                           32961356          0   32961356    0% /run/user/1000

[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data7/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [m(80)][100.0%][r=3003KiB/s,w=384KiB/s][r=750,w=96 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=62570: Mon Aug  2 14:21:55 2021
   read: IOPS=753, BW=3015KiB/s (3087kB/s)(295MiB/100177msec)
    clat (usec): min=41, max=1050.1k, avg=105997.86, stdev=61728.13
     lat (usec): min=41, max=1050.1k, avg=105998.41, stdev=61728.13
    clat percentiles (usec):
     |  1.00th=[    53],  5.00th=[    74], 10.00th=[ 24249], 20.00th=[ 71828],
     | 30.00th=[ 86508], 40.00th=[ 94897], 50.00th=[102237], 60.00th=[110625],
     | 70.00th=[122160], 80.00th=[139461], 90.00th=[170918], 95.00th=[208667],
     | 99.00th=[312476], 99.50th=[362808], 99.90th=[492831], 99.95th=[526386],
     | 99.99th=[666895]
   bw (  KiB/s): min=    7, max=  103, per=1.25%, avg=37.69, stdev=10.97, samples=15985
   iops        : min=    1, max=   25, avg= 9.38, stdev= 2.75, samples=15985
  write: IOPS=86, BW=346KiB/s (354kB/s)(33.8MiB/100177msec)
    clat (usec): min=45, max=82908, avg=94.10, stdev=890.32
     lat (usec): min=46, max=82909, avg=94.86, stdev=890.32
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   61], 10.00th=[   65], 20.00th=[   70],
     | 30.00th=[   75], 40.00th=[   82], 50.00th=[   90], 60.00th=[   93],
     | 70.00th=[   95], 80.00th=[   96], 90.00th=[   97], 95.00th=[  102],
     | 99.00th=[  124], 99.50th=[  130], 99.90th=[  237], 99.95th=[  338],
     | 99.99th=[83362]
   bw (  KiB/s): min=    7, max=   64, per=3.25%, avg=11.21, stdev= 5.81, samples=6152
   iops        : min=    1, max=   16, avg= 2.76, stdev= 1.47, samples=6152
  lat (usec)   : 50=0.44%, 100=17.87%, 250=0.72%, 500=0.02%
  lat (msec)   : 10=0.01%, 20=0.03%, 50=3.39%, 100=30.52%, 250=44.66%
  lat (msec)   : 500=2.27%, 750=0.07%, 1000=0.01%
  cpu          : usr=0.01%, sys=0.05%, ctx=84427, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=75505,8657,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3015KiB/s (3087kB/s), 3015KiB/s-3015KiB/s (3087kB/s-3087kB/s), io=295MiB (309MB), run=100177-100177msec
  WRITE: bw=346KiB/s (354kB/s), 346KiB/s-346KiB/s (354kB/s-354kB/s), io=33.8MiB (35.5MB), run=100177-100177msec

Disk stats (read/write):
    dm-8: ios=75913/8689, merge=0/0, ticks=8512254/497, in_queue=8519654, util=100.00%, aggrios=75914/8688, aggrmerge=0/1, aggrticks=8520769/474, aggrin_queue=8521480, aggrutil=100.00%
  sdi: ios=75914/8688, merge=0/1, ticks=8520769/474, in_queue=8521480, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# rm /data7/randrw -f
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data7/randread -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [r(80)][100.0%][r=3267KiB/s,w=0KiB/s][r=816,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=74528: Mon Aug  2 14:28:27 2021
   read: IOPS=918, BW=3674KiB/s (3762kB/s)(359MiB/100151msec)
    clat (usec): min=35, max=893253, avg=87003.38, stdev=68263.87
     lat (usec): min=35, max=893254, avg=87003.92, stdev=68263.87
    clat percentiles (usec):
     |  1.00th=[    49],  5.00th=[    56], 10.00th=[    74], 20.00th=[ 22414],
     | 30.00th=[ 46400], 40.00th=[ 68682], 50.00th=[ 90702], 60.00th=[104334],
     | 70.00th=[117965], 80.00th=[131597], 90.00th=[158335], 95.00th=[196084],
     | 99.00th=[308282], 99.50th=[362808], 99.90th=[497026], 99.95th=[557843],
     | 99.99th=[683672]
   bw (  KiB/s): min=    7, max=  144, per=1.25%, avg=45.96, stdev=17.42, samples=15973
   iops        : min=    1, max=   36, avg=11.45, stdev= 4.36, samples=15973
  lat (usec)   : 50=1.73%, 100=17.08%, 250=0.21%
  lat (msec)   : 20=0.30%, 50=12.62%, 100=25.34%, 250=40.52%, 500=2.09%
  lat (msec)   : 750=0.09%, 1000=0.01%
  cpu          : usr=0.01%, sys=0.05%, ctx=92423, majf=0, minf=88
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=91995,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3674KiB/s (3762kB/s), 3674KiB/s-3674KiB/s (3762kB/s-3762kB/s), io=359MiB (377MB), run=100151-100151msec

Disk stats (read/write):
    dm-8: ios=92374/141, merge=0/0, ticks=8552731/10, in_queue=8554691, util=100.00%, aggrios=92374/140, aggrmerge=0/1, aggrticks=8555062/9, aggrin_queue=8555007, aggrutil=99.99%
  sdi: ios=92374/140, merge=0/1, ticks=8555062/9, in_queue=8555007, util=99.99%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data7/randwrite -direct=1 -iodepth 1 -thread -rw=randwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [w(80)][100.0%][r=0KiB/s,w=4728KiB/s][r=0,w=1182 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=76415: Mon Aug  2 14:31:38 2021
  write: IOPS=1310, BW=5241KiB/s (5366kB/s)(512MiB/100071msec)
    clat (usec): min=109, max=301713, avg=61032.88, stdev=26819.04
     lat (usec): min=110, max=301714, avg=61033.61, stdev=26819.14
    clat percentiles (usec):
     |  1.00th=[  1582],  5.00th=[  2180], 10.00th=[ 19268], 20.00th=[ 53740],
     | 30.00th=[ 61604], 40.00th=[ 64750], 50.00th=[ 66847], 60.00th=[ 69731],
     | 70.00th=[ 71828], 80.00th=[ 74974], 90.00th=[ 81265], 95.00th=[ 90702],
     | 99.00th=[122160], 99.50th=[137364], 99.90th=[258999], 99.95th=[295699],
     | 99.99th=[299893]
   bw (  KiB/s): min=   31, max= 1112, per=1.25%, avg=65.46, stdev=74.17, samples=16000
   iops        : min=    7, max=  278, avg=16.32, stdev=18.54, samples=16000
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=3.76%, 4=3.46%, 10=0.19%, 20=5.52%, 50=6.22%
  lat (msec)   : 100=77.69%, 250=3.00%, 500=0.14%
  cpu          : usr=0.01%, sys=0.23%, ctx=157716, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,131108,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5241KiB/s (5366kB/s), 5241KiB/s-5241KiB/s (5366kB/s-5366kB/s), io=512MiB (537MB), run=100071-100071msec

Disk stats (read/write):
    dm-8: ios=377/131385, merge=0/0, ticks=439846/7999170, in_queue=8489393, util=100.00%, aggrios=360/131603, aggrmerge=0/1, aggrticks=439846/8016310, aggrin_queue=8490889, aggrutil=100.00%
  sdi: ios=360/131603, merge=0/1, ticks=439846/8016310, in_queue=8490889, util=100.00%

[root@bj-wlj-datanode09-172-38-1-210 ~]# # new disk randrw
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [m(80)][100.0%][r=2970KiB/s,w=360KiB/s][r=742,w=90 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=122164: Mon Aug  2 14:52:06 2021
   read: IOPS=762, BW=3050KiB/s (3123kB/s)(298MiB/100156msec)
    clat (usec): min=30, max=1635.8k, avg=104768.40, stdev=61372.97
     lat (usec): min=30, max=1635.8k, avg=104768.86, stdev=61372.97
    clat percentiles (usec):
     |  1.00th=[    44],  5.00th=[    63], 10.00th=[ 36963], 20.00th=[ 72877],
     | 30.00th=[ 85459], 40.00th=[ 91751], 50.00th=[ 98042], 60.00th=[105382],
     | 70.00th=[115868], 80.00th=[132645], 90.00th=[168821], 95.00th=[210764],
     | 99.00th=[325059], 99.50th=[375391], 99.90th=[517997], 99.95th=[566232],
     | 99.99th=[708838]
   bw (  KiB/s): min=    7, max=  112, per=1.25%, avg=38.12, stdev=10.84, samples=15985
   iops        : min=    1, max=   28, avg= 9.48, stdev= 2.72, samples=15985
  write: IOPS=87, BW=349KiB/s (358kB/s)(34.2MiB/100156msec)
    clat (usec): min=31, max=138809, avg=224.11, stdev=3760.81
     lat (usec): min=31, max=138810, avg=224.74, stdev=3760.80
    clat percentiles (usec):
     |  1.00th=[    41],  5.00th=[    49], 10.00th=[    55], 20.00th=[    60],
     | 30.00th=[    64], 40.00th=[    68], 50.00th=[    72], 60.00th=[    79],
     | 70.00th=[    86], 80.00th=[    89], 90.00th=[    91], 95.00th=[    95],
     | 99.00th=[   184], 99.50th=[   873], 99.90th=[ 87557], 99.95th=[105382],
     | 99.99th=[139461]
   bw (  KiB/s): min=    7, max=   64, per=3.23%, avg=11.26, stdev= 5.86, samples=6184
   iops        : min=    1, max=   16, avg= 2.77, stdev= 1.48, samples=6184
  lat (usec)   : 50=3.14%, 100=13.70%, 250=0.32%, 500=0.07%, 750=0.02%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.02%, 10=0.02%, 20=0.07%, 50=3.45%
  lat (msec)   : 100=36.69%, 250=39.97%, 500=2.40%, 750=0.11%, 1000=0.01%
  cpu          : usr=0.01%, sys=0.03%, ctx=85291, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=76363,8751,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3050KiB/s (3123kB/s), 3050KiB/s-3050KiB/s (3123kB/s-3123kB/s), io=298MiB (313MB), run=100156-100156msec
  WRITE: bw=349KiB/s (358kB/s), 349KiB/s-349KiB/s (358kB/s-358kB/s), io=34.2MiB (35.8MB), run=100156-100156msec

Disk stats (read/write):
  sdq: ios=76294/8782, merge=0/5, ticks=7990454/1655, in_queue=7998157, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# 
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/randr -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [r(80)][100.0%][r=3707KiB/s,w=0KiB/s][r=926,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=183343: Mon Aug  2 15:47:39 2021
   read: IOPS=880, BW=3523KiB/s (3608kB/s)(345MiB/100167msec)
    clat (usec): min=37, max=1492.8k, avg=90732.82, stdev=69070.11
     lat (usec): min=37, max=1492.8k, avg=90733.41, stdev=69070.12
    clat percentiles (usec):
     |  1.00th=[    49],  5.00th=[    68], 10.00th=[    72], 20.00th=[ 31589],
     | 30.00th=[ 53216], 40.00th=[ 80217], 50.00th=[ 92799], 60.00th=[100140],
     | 70.00th=[109577], 80.00th=[125305], 90.00th=[160433], 95.00th=[204473],
     | 99.00th=[333448], 99.50th=[396362], 99.90th=[566232], 99.95th=[633340],
     | 99.99th=[935330]
   bw (  KiB/s): min=    7, max=  144, per=1.25%, avg=44.12, stdev=15.85, samples=15963
   iops        : min=    1, max=   36, avg=10.99, stdev= 3.97, samples=15963
  lat (usec)   : 50=1.37%, 100=10.50%, 250=0.09%
  lat (msec)   : 10=0.01%, 20=0.62%, 50=16.15%, 100=31.31%, 250=37.26%
  lat (msec)   : 500=2.50%, 750=0.17%, 1000=0.02%
  cpu          : usr=0.01%, sys=0.04%, ctx=88581, majf=0, minf=88
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=88223,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=3523KiB/s (3608kB/s), 3523KiB/s-3523KiB/s (3608kB/s-3608kB/s), io=345MiB (361MB), run=100167-100167msec

Disk stats (read/write):
  sdq: ios=88197/9, merge=0/1, ticks=7998250/0, in_queue=8002523, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/randw -direct=1 -iodepth 1 -thread -rw=randwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [w(80)][100.0%][r=0KiB/s,w=4584KiB/s][r=0,w=1146 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=184599: Mon Aug  2 15:49:38 2021
  write: IOPS=1337, BW=5349KiB/s (5478kB/s)(523MiB/100074msec)
    clat (usec): min=67, max=354262, avg=59793.28, stdev=26293.74
     lat (usec): min=68, max=354263, avg=59794.00, stdev=26293.83
    clat percentiles (usec):
     |  1.00th=[  1631],  5.00th=[  2737], 10.00th=[ 16909], 20.00th=[ 52691],
     | 30.00th=[ 60556], 40.00th=[ 63177], 50.00th=[ 65799], 60.00th=[ 67634],
     | 70.00th=[ 70779], 80.00th=[ 72877], 90.00th=[ 79168], 95.00th=[ 86508],
     | 99.00th=[115868], 99.50th=[128451], 99.90th=[304088], 99.95th=[341836],
     | 99.99th=[350225]
   bw (  KiB/s): min=   23, max= 1024, per=1.25%, avg=66.82, stdev=63.63, samples=16000
   iops        : min=    5, max=  256, avg=16.66, stdev=15.91, samples=16000
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
  lat (msec)   : 2=2.89%, 4=2.62%, 10=0.11%, 20=10.14%, 50=3.37%
  lat (msec)   : 100=78.62%, 250=2.08%, 500=0.15%
  cpu          : usr=0.02%, sys=0.19%, ctx=154625, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,133833,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=5349KiB/s (5478kB/s), 5349KiB/s-5349KiB/s (5478kB/s-5478kB/s), io=523MiB (548MB), run=100074-100074msec

Disk stats (read/write):
  sdq: ios=0/134043, merge=0/3, ticks=0/8003445, in_queue=8005210, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/rw -direct=1 -iodepth 1 -thread -rw=readwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [M(80)][100.0%][r=68.7MiB/s,w=7944KiB/s][r=17.6k,w=1986 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=190580: Mon Aug  2 15:53:11 2021
   read: IOPS=15.0k, BW=62.4MiB/s (65.4MB/s)(6242MiB/100005msec)
    clat (usec): min=40, max=214187, avg=4998.39, stdev=5819.43
     lat (usec): min=40, max=214187, avg=4998.56, stdev=5819.45
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[   25], 99.50th=[   44], 99.90th=[   92], 99.95th=[  114],
     | 99.99th=[  155]
   bw (  KiB/s): min=  232, max= 1272, per=1.25%, avg=798.84, stdev=126.15, samples=15997
   iops        : min=   58, max=  318, avg=199.69, stdev=31.54, samples=15997
  write: IOPS=1780, BW=7124KiB/s (7295kB/s)(696MiB/100005msec)
    clat (usec): min=30, max=41108, avg=59.35, stdev=178.68
     lat (usec): min=30, max=41109, avg=59.62, stdev=178.70
    clat percentiles (usec):
     |  1.00th=[   35],  5.00th=[   36], 10.00th=[   36], 20.00th=[   38],
     | 30.00th=[   41], 40.00th=[   45], 50.00th=[   52], 60.00th=[   58],
     | 70.00th=[   63], 80.00th=[   69], 90.00th=[   81], 95.00th=[   93],
     | 99.00th=[  139], 99.50th=[  277], 99.90th=[  742], 99.95th=[ 1004],
     | 99.99th=[ 1942]
   bw (  KiB/s): min=    7, max=  240, per=1.25%, avg=89.04, stdev=31.41, samples=15991
   iops        : min=    1, max=   60, avg=22.23, stdev= 7.85, samples=15991
  lat (usec)   : 50=4.80%, 100=4.88%, 250=0.30%, 500=0.01%, 750=0.34%
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=0.24%, 4=6.07%, 10=81.36%, 20=0.80%, 50=0.78%
  lat (msec)   : 100=0.29%, 250=0.07%
  cpu          : usr=0.07%, sys=0.37%, ctx=1782827, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1597933,178104,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=6242MiB (6545MB), run=100005-100005msec
  WRITE: bw=7124KiB/s (7295kB/s), 7124KiB/s-7124KiB/s (7295kB/s-7295kB/s), io=696MiB (730MB), run=100005-100005msec

Disk stats (read/write):
  sdq: ios=1597290/178061, merge=0/1, ticks=7961492/7531, in_queue=7973957, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/r -direct=1 -iodepth 1 -thread -rw=read -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 75 (f=75): [R(3),E(1),R(5),E(1),R(2),E(1),R(4),E(1),R(59),E(1),R(2)][100.0%][r=62.8MiB/s,w=0KiB/s][r=16.1k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=192953: Mon Aug  2 15:55:51 2021
   read: IOPS=15.2k, BW=59.3MiB/s (62.1MB/s)(5927MiB/100005msec)
    clat (usec): min=86, max=342933, avg=5270.92, stdev=8000.98
     lat (usec): min=87, max=342933, avg=5271.09, stdev=8001.00
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    5], 10.00th=[    5], 20.00th=[    5],
     | 30.00th=[    5], 40.00th=[    5], 50.00th=[    5], 60.00th=[    5],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    5], 95.00th=[    6],
     | 99.00th=[   27], 99.50th=[   54], 99.90th=[  138], 99.95th=[  159],
     | 99.99th=[  201]
   bw (  KiB/s): min=   15, max= 1048, per=1.25%, avg=758.49, stdev=223.25, samples=15969
   iops        : min=    3, max=  262, avg=189.59, stdev=55.82, samples=15969
  lat (usec)   : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.25%, 4=2.69%, 10=95.13%, 20=0.66%, 50=0.71%
  lat (msec)   : 100=0.31%, 250=0.24%, 500=0.01%
  cpu          : usr=0.06%, sys=0.28%, ctx=1518051, majf=0, minf=140
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1517401,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=59.3MiB/s (62.1MB/s), 59.3MiB/s-59.3MiB/s (62.1MB/s-62.1MB/s), io=5927MiB (6215MB), run=100005-100005msec

Disk stats (read/write):
  sdq: ios=1513410/14, merge=0/1, ticks=7961404/0, in_queue=7967479, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/w -direct=1 -iodepth 1 -thread -rw=write -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [W(80)][100.0%][r=0KiB/s,w=437MiB/s][r=0,w=112k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=194453: Mon Aug  2 15:58:08 2021
  write: IOPS=111k, BW=434MiB/s (456MB/s)(42.4GiB/100003msec)
    clat (usec): min=50, max=3562, avg=718.51, stdev=25.02
     lat (usec): min=50, max=3562, avg=718.65, stdev=25.02
    clat percentiles (usec):
     |  1.00th=[  676],  5.00th=[  693], 10.00th=[  701], 20.00th=[  709],
     | 30.00th=[  709], 40.00th=[  709], 50.00th=[  717], 60.00th=[  717],
     | 70.00th=[  725], 80.00th=[  734], 90.00th=[  742], 95.00th=[  750],
     | 99.00th=[  783], 99.50th=[  791], 99.90th=[  898], 99.95th=[ 1123],
     | 99.99th=[ 1385]
   bw (  KiB/s): min= 5272, max= 5624, per=1.25%, avg=5560.12, stdev=25.56, samples=15966
   iops        : min= 1318, max= 1406, avg=1390.03, stdev= 6.39, samples=15966
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=94.06%, 1000=5.87%
  lat (msec)   : 2=0.07%, 4=0.01%
  cpu          : usr=0.27%, sys=7.46%, ctx=24318385, majf=0, minf=4092
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,11121198,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=434MiB/s (456MB/s), 434MiB/s-434MiB/s (456MB/s-456MB/s), io=42.4GiB (45.6GB), run=100003-100003msec

Disk stats (read/write):
  sdq: ios=0/11105144, merge=0/6, ticks=0/7430033, in_queue=7451498, util=100.00%