大数据2.4T 磁盘HDD io 测试
一. FIO介绍
FIO是测试IOPS的非常好的工具,用来对磁盘进行压力测试和验证。磁盘IO是检查磁盘性能的重要指标,可以按照负载情况分成照顺序读写,随机读写两大类。FIO是一个可以产生很多线程或进程并执行用户指定的特定类型I/O操作的工具,FIO的典型用途是编写和模拟的I/O负载匹配的作业文件。也就是说FIO 是一个多线程io生成工具,可以生成多种IO模式,用来测试磁盘设备的性能(也包含文件系统:如针对网络文件系统 NFS 的IO测试)。fio在github上的坐标:https://github.com/axboe/fio 。
1.1. FIO 工具常用参数:
参数说明:
filename=/dev/sdb1 测试文件名称,通常选择需要测试的盘的data目录。
direct=1 是否使用directIO,测试过程绕过OS自带的buffer,使测试磁盘的结果更真实。Linux读写的时候,内核维护了缓存,数据先写到缓存,后面再后台写到SSD。读的时候也优先读缓存里的数据。这样速度可以加快,但是一旦掉电缓存里的数据就没了。所以有一种模式叫做DirectIO,跳过缓存,直接读写SSD。
rw=randwrite 测试随机写的I/O
rw=randrw 测试随机写和读的I/O
bs=16k 单次io的块文件大小为16k
bsrange=512-2048 同上,提定数据块的大小范围
size=5G 每个线程读写的数据量是5GB。
numjobs=1 每个job(任务)开1个线程,这里用了几,后面每个用-name指定的任务就开几个线程测试。所以最终线程数=任务数(几个name=jobx)* numjobs。
name=job1:一个任务的名字,重复了也没关系。如果fio -name=job1 -name=job2,建立了两个任务,共享-name=job1之前的参数。-name之后的就是job2任务独有的参数。
thread 使用pthread_create创建线程,另一种是fork创建进程。进程的开销比线程要大,一般都采用thread测试。
runtime=1000 测试时间为1000秒,如果不写则一直将5g文件分4k每次写完为止。
ioengine=libaio 指定io引擎使用libaio方式。libaio:Linux本地异步I/O。请注意,Linux可能只支持具有非缓冲I/O的排队行为(设置为“direct=1”或“buffered=0”);rbd:通过librbd直接访问CEPH Rados
iodepth=16 队列的深度为16.在异步模式下,CPU不能一直无限的发命令到SSD。比如SSD执行读写如果发生了卡顿,那有可能系统会一直不停的发命令,几千个,甚至几万个,这样一方面SSD扛不住,另一方面这么多命令会很占内存,系统也要挂掉了。这样,就带来一个参数叫做队列深度。
Block Devices(RBD),无需使用内核RBD驱动程序(rbd.ko)。该参数包含很多ioengine,如:libhdfs/rdma等
rwmixwrite=30 在混合读写的模式下,写占30%
group_reporting 关于显示结果的,汇总每个进程的信息。
此外
lockmem=1g 只使用1g内存进行测试。
zero_buffers 用0初始化系统buffer。
nrfiles=8 每个进程生成文件的数量。
磁盘读写常用测试点:
1. Read=100% Ramdon=100% rw=randread (100%随机读)
2. Read=100% Sequence=100% rw=read (100%顺序读)
3. Write=100% Sequence=100% rw=write (100%顺序写)
4. Write=100% Ramdon=100% rw=randwrite (100%随机写)
5. Read=70% Sequence=100% rw=rw, rwmixread=70, rwmixwrite=30
(70%顺序读,30%顺序写)
6. Read=70% Ramdon=100% rw=randrw, rwmixread=70, rwmixwrite=30
(70%随机读,30%随机写)
1.2. fio 安装:
yum -y install fio
二. 大数据磁盘压力测试:
大数据新旧磁盘大小均为2.4T HDD 磁盘,其中/data1 -> /data7 为旧磁盘,/data8 -> /data15 为新磁盘.
针对新旧磁盘/data7,/data15 使用fio命令分别进行随机/顺序io进行io测试.
命令案例:
- fio -filename=/data7/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
在/data7/创建测试文件,direct直接读取,去除bufer/cache, io深度1,模式为xx,bs块未4k,size大小10G,运行时间100s,jobs数量80
2.1 随机读写测试
测试命令:
fio -filename=/data7/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
2.2 随机读对比
测试命令:
fio -filename=/data7/randr -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
2.3 随机写:
测试命令:
fio -filename=/data7/randw -direct=1 -iodepth 1 -thread -rw=randwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
2.4 顺序读写:
测试命令:
fio -filename=/data7/rw -direct=1 -iodepth 1 -thread -rw=readwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
2.5 顺序读:
测试命令:
fio -filename=/data7/r -direct=1 -iodepth 1 -thread -rw=read -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
2.6 顺序写:
测试命令:
fio -filename=/data7/w -direct=1 -iodepth 1 -thread -rw=write -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
三. 综合性能对比
磁盘 | 测试类型 | 测试块大小 | iops读 | iops写 | 读速率 | 写速率 | 磁盘io使用百分比 |
---|---|---|---|---|---|---|---|
/data7 [旧盘] | 随机读写 | 4k | 753 | 86 | 3087kB/s | 354kB/s | 100% |
/data15 [新盘] | 随机读写 | 4k | 762 | 87 | 3123kB/s | 258kB/s | 100% |
/data7 [旧盘] | 随机读 | 4k | 918 | 3762kB/s | 99% | ||
/data15 [新盘] | 随机读 | 4k | 880 | 3608kB/s | 100% | ||
/data7 [旧盘] | 随机写 | 4k | 1310 | 5366kB/s | 100% | ||
/data15 [新盘] | 随机写 | 4k | 1337 | 5478kB/s | 100% | ||
/data7 [旧盘] | 顺序读写 | 4k | 15.6k | 1737 | 63.9MB/s | 7117kB/s | 100% |
/data15 [新盘] | 顺序读写 | 4k | 15.0k | 1780 | 65.4MB/s | 7295kB/s | 100% |
/data7 [旧盘] | 顺序读 | 4k | 14.1K | 57.7MB/s | 100% | ||
/data15 [新盘] | 顺序读 | 4k | 15.2K | 62.1MB/s | - | 100% | |
/data7 [旧盘] | 顺序写 | 4k | 88.1k | 361MB/s | 83.35% | ||
/data15 [新盘] | 顺序写 | 4k | 111k | 456MB/s | 100% |
io性能分随机读写与顺序读写:
随机读写性能测试中,新旧磁盘差距不大,新硬盘略低于老硬盘,随机读写一直是磁盘io性能瓶颈问题.
顺序读写中,新盘读写性能略高于旧盘.
整体来言,新旧磁盘性能在合理访问内.
PS: 本次测试磁盘均在正常使用,测试结果可能受制于磁盘可用空间,当前磁盘io使用量等.可能出现偏差.
四. 附 io 测试详情:
[root@bj-wlj-datanode09-172-38-1-210 ~]# # old disk randrw
[root@bj-wlj-datanode09-172-38-1-210 ~]# df
文件系统 1K-块 已用 可用 已用% 挂载点
devtmpfs 164794756 0 164794756 0% /dev
tmpfs 164806764 4 164806760 1% /dev/shm
tmpfs 164806764 27276 164779488 1% /run
tmpfs 164806764 0 164806764 0% /sys/fs/cgroup
/dev/mapper/centos-root 52403200 17991632 34411568 35% /
/dev/sda1 1038336 153596 884740 15% /boot
/dev/mapper/vg--ssd--g1--data1-lv--ssd--data1 1560821124 1202231684 358589440 78% /data1
/dev/mapper/vg--hdd--data4-lv--hdd--data4 2342418948 1829166600 513252348 79% /data4
/dev/mapper/vg--hdd--data6-lv--hdd--data6 2342418948 1817236892 525182056 78% /data6
/dev/mapper/vg--hdd--data5-lv--hdd--data5 2342418948 1819285400 523133548 78% /data5
/dev/mapper/vg--hdd--data2-lv--hdd--data2 2342418948 1830784580 511634368 79% /data2
/dev/mapper/vg--hdd--data7-lv--hdd--data7 2342418948 1825019024 517399924 78% /data7
/dev/mapper/centos-lv_data 419225600 45632 419179968 1% /data
cm_processes 164806764 65632 164741132 1% /run/cloudera-scm-agent/process
tmpfs 32961356 0 32961356 0% /run/user/1001
/dev/sdk1 2146434048 165583628 1980850420 8% /data9
/dev/sdl1 2146434048 163279364 1983154684 8% /data10
/dev/sdm1 2146434048 163748512 1982685536 8% /data11
/dev/sdn1 2146434048 164037888 1982396160 8% /data12
/dev/sdo1 2146434048 169086524 1977347524 8% /data13
/dev/sdp1 2146434048 166700816 1979733232 8% /data14
/dev/sdq1 2146434048 165240372 1981193676 8% /data15
/dev/sdj1 2146434048 165038420 1981395628 8% /data8
/dev/mapper/vg--hdd--data3-lv--hdd--data3 2342418948 1289470652 1052948296 56% /data3
tmpfs 32961356 0 32961356 0% /run/user/0
tmpfs 32961356 0 32961356 0% /run/user/1000
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data7/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [m(80)][100.0%][r=3003KiB/s,w=384KiB/s][r=750,w=96 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=62570: Mon Aug 2 14:21:55 2021
read: IOPS=753, BW=3015KiB/s (3087kB/s)(295MiB/100177msec)
clat (usec): min=41, max=1050.1k, avg=105997.86, stdev=61728.13
lat (usec): min=41, max=1050.1k, avg=105998.41, stdev=61728.13
clat percentiles (usec):
| 1.00th=[ 53], 5.00th=[ 74], 10.00th=[ 24249], 20.00th=[ 71828],
| 30.00th=[ 86508], 40.00th=[ 94897], 50.00th=[102237], 60.00th=[110625],
| 70.00th=[122160], 80.00th=[139461], 90.00th=[170918], 95.00th=[208667],
| 99.00th=[312476], 99.50th=[362808], 99.90th=[492831], 99.95th=[526386],
| 99.99th=[666895]
bw ( KiB/s): min= 7, max= 103, per=1.25%, avg=37.69, stdev=10.97, samples=15985
iops : min= 1, max= 25, avg= 9.38, stdev= 2.75, samples=15985
write: IOPS=86, BW=346KiB/s (354kB/s)(33.8MiB/100177msec)
clat (usec): min=45, max=82908, avg=94.10, stdev=890.32
lat (usec): min=46, max=82909, avg=94.86, stdev=890.32
clat percentiles (usec):
| 1.00th=[ 56], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 70],
| 30.00th=[ 75], 40.00th=[ 82], 50.00th=[ 90], 60.00th=[ 93],
| 70.00th=[ 95], 80.00th=[ 96], 90.00th=[ 97], 95.00th=[ 102],
| 99.00th=[ 124], 99.50th=[ 130], 99.90th=[ 237], 99.95th=[ 338],
| 99.99th=[83362]
bw ( KiB/s): min= 7, max= 64, per=3.25%, avg=11.21, stdev= 5.81, samples=6152
iops : min= 1, max= 16, avg= 2.76, stdev= 1.47, samples=6152
lat (usec) : 50=0.44%, 100=17.87%, 250=0.72%, 500=0.02%
lat (msec) : 10=0.01%, 20=0.03%, 50=3.39%, 100=30.52%, 250=44.66%
lat (msec) : 500=2.27%, 750=0.07%, 1000=0.01%
cpu : usr=0.01%, sys=0.05%, ctx=84427, majf=0, minf=10
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=75505,8657,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3015KiB/s (3087kB/s), 3015KiB/s-3015KiB/s (3087kB/s-3087kB/s), io=295MiB (309MB), run=100177-100177msec
WRITE: bw=346KiB/s (354kB/s), 346KiB/s-346KiB/s (354kB/s-354kB/s), io=33.8MiB (35.5MB), run=100177-100177msec
Disk stats (read/write):
dm-8: ios=75913/8689, merge=0/0, ticks=8512254/497, in_queue=8519654, util=100.00%, aggrios=75914/8688, aggrmerge=0/1, aggrticks=8520769/474, aggrin_queue=8521480, aggrutil=100.00%
sdi: ios=75914/8688, merge=0/1, ticks=8520769/474, in_queue=8521480, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# rm /data7/randrw -f
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data7/randread -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [r(80)][100.0%][r=3267KiB/s,w=0KiB/s][r=816,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=74528: Mon Aug 2 14:28:27 2021
read: IOPS=918, BW=3674KiB/s (3762kB/s)(359MiB/100151msec)
clat (usec): min=35, max=893253, avg=87003.38, stdev=68263.87
lat (usec): min=35, max=893254, avg=87003.92, stdev=68263.87
clat percentiles (usec):
| 1.00th=[ 49], 5.00th=[ 56], 10.00th=[ 74], 20.00th=[ 22414],
| 30.00th=[ 46400], 40.00th=[ 68682], 50.00th=[ 90702], 60.00th=[104334],
| 70.00th=[117965], 80.00th=[131597], 90.00th=[158335], 95.00th=[196084],
| 99.00th=[308282], 99.50th=[362808], 99.90th=[497026], 99.95th=[557843],
| 99.99th=[683672]
bw ( KiB/s): min= 7, max= 144, per=1.25%, avg=45.96, stdev=17.42, samples=15973
iops : min= 1, max= 36, avg=11.45, stdev= 4.36, samples=15973
lat (usec) : 50=1.73%, 100=17.08%, 250=0.21%
lat (msec) : 20=0.30%, 50=12.62%, 100=25.34%, 250=40.52%, 500=2.09%
lat (msec) : 750=0.09%, 1000=0.01%
cpu : usr=0.01%, sys=0.05%, ctx=92423, majf=0, minf=88
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=91995,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3674KiB/s (3762kB/s), 3674KiB/s-3674KiB/s (3762kB/s-3762kB/s), io=359MiB (377MB), run=100151-100151msec
Disk stats (read/write):
dm-8: ios=92374/141, merge=0/0, ticks=8552731/10, in_queue=8554691, util=100.00%, aggrios=92374/140, aggrmerge=0/1, aggrticks=8555062/9, aggrin_queue=8555007, aggrutil=99.99%
sdi: ios=92374/140, merge=0/1, ticks=8555062/9, in_queue=8555007, util=99.99%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data7/randwrite -direct=1 -iodepth 1 -thread -rw=randwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [w(80)][100.0%][r=0KiB/s,w=4728KiB/s][r=0,w=1182 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=76415: Mon Aug 2 14:31:38 2021
write: IOPS=1310, BW=5241KiB/s (5366kB/s)(512MiB/100071msec)
clat (usec): min=109, max=301713, avg=61032.88, stdev=26819.04
lat (usec): min=110, max=301714, avg=61033.61, stdev=26819.14
clat percentiles (usec):
| 1.00th=[ 1582], 5.00th=[ 2180], 10.00th=[ 19268], 20.00th=[ 53740],
| 30.00th=[ 61604], 40.00th=[ 64750], 50.00th=[ 66847], 60.00th=[ 69731],
| 70.00th=[ 71828], 80.00th=[ 74974], 90.00th=[ 81265], 95.00th=[ 90702],
| 99.00th=[122160], 99.50th=[137364], 99.90th=[258999], 99.95th=[295699],
| 99.99th=[299893]
bw ( KiB/s): min= 31, max= 1112, per=1.25%, avg=65.46, stdev=74.17, samples=16000
iops : min= 7, max= 278, avg=16.32, stdev=18.54, samples=16000
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=3.76%, 4=3.46%, 10=0.19%, 20=5.52%, 50=6.22%
lat (msec) : 100=77.69%, 250=3.00%, 500=0.14%
cpu : usr=0.01%, sys=0.23%, ctx=157716, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,131108,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=5241KiB/s (5366kB/s), 5241KiB/s-5241KiB/s (5366kB/s-5366kB/s), io=512MiB (537MB), run=100071-100071msec
Disk stats (read/write):
dm-8: ios=377/131385, merge=0/0, ticks=439846/7999170, in_queue=8489393, util=100.00%, aggrios=360/131603, aggrmerge=0/1, aggrticks=439846/8016310, aggrin_queue=8490889, aggrutil=100.00%
sdi: ios=360/131603, merge=0/1, ticks=439846/8016310, in_queue=8490889, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# # new disk randrw
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/randrw -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [m(80)][100.0%][r=2970KiB/s,w=360KiB/s][r=742,w=90 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=122164: Mon Aug 2 14:52:06 2021
read: IOPS=762, BW=3050KiB/s (3123kB/s)(298MiB/100156msec)
clat (usec): min=30, max=1635.8k, avg=104768.40, stdev=61372.97
lat (usec): min=30, max=1635.8k, avg=104768.86, stdev=61372.97
clat percentiles (usec):
| 1.00th=[ 44], 5.00th=[ 63], 10.00th=[ 36963], 20.00th=[ 72877],
| 30.00th=[ 85459], 40.00th=[ 91751], 50.00th=[ 98042], 60.00th=[105382],
| 70.00th=[115868], 80.00th=[132645], 90.00th=[168821], 95.00th=[210764],
| 99.00th=[325059], 99.50th=[375391], 99.90th=[517997], 99.95th=[566232],
| 99.99th=[708838]
bw ( KiB/s): min= 7, max= 112, per=1.25%, avg=38.12, stdev=10.84, samples=15985
iops : min= 1, max= 28, avg= 9.48, stdev= 2.72, samples=15985
write: IOPS=87, BW=349KiB/s (358kB/s)(34.2MiB/100156msec)
clat (usec): min=31, max=138809, avg=224.11, stdev=3760.81
lat (usec): min=31, max=138810, avg=224.74, stdev=3760.80
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 49], 10.00th=[ 55], 20.00th=[ 60],
| 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 79],
| 70.00th=[ 86], 80.00th=[ 89], 90.00th=[ 91], 95.00th=[ 95],
| 99.00th=[ 184], 99.50th=[ 873], 99.90th=[ 87557], 99.95th=[105382],
| 99.99th=[139461]
bw ( KiB/s): min= 7, max= 64, per=3.23%, avg=11.26, stdev= 5.86, samples=6184
iops : min= 1, max= 16, avg= 2.77, stdev= 1.48, samples=6184
lat (usec) : 50=3.14%, 100=13.70%, 250=0.32%, 500=0.07%, 750=0.02%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.02%, 10=0.02%, 20=0.07%, 50=3.45%
lat (msec) : 100=36.69%, 250=39.97%, 500=2.40%, 750=0.11%, 1000=0.01%
cpu : usr=0.01%, sys=0.03%, ctx=85291, majf=0, minf=8
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=76363,8751,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3050KiB/s (3123kB/s), 3050KiB/s-3050KiB/s (3123kB/s-3123kB/s), io=298MiB (313MB), run=100156-100156msec
WRITE: bw=349KiB/s (358kB/s), 349KiB/s-349KiB/s (358kB/s-358kB/s), io=34.2MiB (35.8MB), run=100156-100156msec
Disk stats (read/write):
sdq: ios=76294/8782, merge=0/5, ticks=7990454/1655, in_queue=7998157, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]#
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/randr -direct=1 -iodepth 1 -thread -rw=randread -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [r(80)][100.0%][r=3707KiB/s,w=0KiB/s][r=926,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=183343: Mon Aug 2 15:47:39 2021
read: IOPS=880, BW=3523KiB/s (3608kB/s)(345MiB/100167msec)
clat (usec): min=37, max=1492.8k, avg=90732.82, stdev=69070.11
lat (usec): min=37, max=1492.8k, avg=90733.41, stdev=69070.12
clat percentiles (usec):
| 1.00th=[ 49], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 31589],
| 30.00th=[ 53216], 40.00th=[ 80217], 50.00th=[ 92799], 60.00th=[100140],
| 70.00th=[109577], 80.00th=[125305], 90.00th=[160433], 95.00th=[204473],
| 99.00th=[333448], 99.50th=[396362], 99.90th=[566232], 99.95th=[633340],
| 99.99th=[935330]
bw ( KiB/s): min= 7, max= 144, per=1.25%, avg=44.12, stdev=15.85, samples=15963
iops : min= 1, max= 36, avg=10.99, stdev= 3.97, samples=15963
lat (usec) : 50=1.37%, 100=10.50%, 250=0.09%
lat (msec) : 10=0.01%, 20=0.62%, 50=16.15%, 100=31.31%, 250=37.26%
lat (msec) : 500=2.50%, 750=0.17%, 1000=0.02%
cpu : usr=0.01%, sys=0.04%, ctx=88581, majf=0, minf=88
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=88223,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=3523KiB/s (3608kB/s), 3523KiB/s-3523KiB/s (3608kB/s-3608kB/s), io=345MiB (361MB), run=100167-100167msec
Disk stats (read/write):
sdq: ios=88197/9, merge=0/1, ticks=7998250/0, in_queue=8002523, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/randw -direct=1 -iodepth 1 -thread -rw=randwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [w(80)][100.0%][r=0KiB/s,w=4584KiB/s][r=0,w=1146 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=184599: Mon Aug 2 15:49:38 2021
write: IOPS=1337, BW=5349KiB/s (5478kB/s)(523MiB/100074msec)
clat (usec): min=67, max=354262, avg=59793.28, stdev=26293.74
lat (usec): min=68, max=354263, avg=59794.00, stdev=26293.83
clat percentiles (usec):
| 1.00th=[ 1631], 5.00th=[ 2737], 10.00th=[ 16909], 20.00th=[ 52691],
| 30.00th=[ 60556], 40.00th=[ 63177], 50.00th=[ 65799], 60.00th=[ 67634],
| 70.00th=[ 70779], 80.00th=[ 72877], 90.00th=[ 79168], 95.00th=[ 86508],
| 99.00th=[115868], 99.50th=[128451], 99.90th=[304088], 99.95th=[341836],
| 99.99th=[350225]
bw ( KiB/s): min= 23, max= 1024, per=1.25%, avg=66.82, stdev=63.63, samples=16000
iops : min= 5, max= 256, avg=16.66, stdev=15.91, samples=16000
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
lat (msec) : 2=2.89%, 4=2.62%, 10=0.11%, 20=10.14%, 50=3.37%
lat (msec) : 100=78.62%, 250=2.08%, 500=0.15%
cpu : usr=0.02%, sys=0.19%, ctx=154625, majf=0, minf=7
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,133833,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=5349KiB/s (5478kB/s), 5349KiB/s-5349KiB/s (5478kB/s-5478kB/s), io=523MiB (548MB), run=100074-100074msec
Disk stats (read/write):
sdq: ios=0/134043, merge=0/3, ticks=0/8003445, in_queue=8005210, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/rw -direct=1 -iodepth 1 -thread -rw=readwrite -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [M(80)][100.0%][r=68.7MiB/s,w=7944KiB/s][r=17.6k,w=1986 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=190580: Mon Aug 2 15:53:11 2021
read: IOPS=15.0k, BW=62.4MiB/s (65.4MB/s)(6242MiB/100005msec)
clat (usec): min=40, max=214187, avg=4998.39, stdev=5819.43
lat (usec): min=40, max=214187, avg=4998.56, stdev=5819.45
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5],
| 99.00th=[ 25], 99.50th=[ 44], 99.90th=[ 92], 99.95th=[ 114],
| 99.99th=[ 155]
bw ( KiB/s): min= 232, max= 1272, per=1.25%, avg=798.84, stdev=126.15, samples=15997
iops : min= 58, max= 318, avg=199.69, stdev=31.54, samples=15997
write: IOPS=1780, BW=7124KiB/s (7295kB/s)(696MiB/100005msec)
clat (usec): min=30, max=41108, avg=59.35, stdev=178.68
lat (usec): min=30, max=41109, avg=59.62, stdev=178.70
clat percentiles (usec):
| 1.00th=[ 35], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 38],
| 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 52], 60.00th=[ 58],
| 70.00th=[ 63], 80.00th=[ 69], 90.00th=[ 81], 95.00th=[ 93],
| 99.00th=[ 139], 99.50th=[ 277], 99.90th=[ 742], 99.95th=[ 1004],
| 99.99th=[ 1942]
bw ( KiB/s): min= 7, max= 240, per=1.25%, avg=89.04, stdev=31.41, samples=15991
iops : min= 1, max= 60, avg=22.23, stdev= 7.85, samples=15991
lat (usec) : 50=4.80%, 100=4.88%, 250=0.30%, 500=0.01%, 750=0.34%
lat (usec) : 1000=0.05%
lat (msec) : 2=0.24%, 4=6.07%, 10=81.36%, 20=0.80%, 50=0.78%
lat (msec) : 100=0.29%, 250=0.07%
cpu : usr=0.07%, sys=0.37%, ctx=1782827, majf=0, minf=35
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1597933,178104,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=6242MiB (6545MB), run=100005-100005msec
WRITE: bw=7124KiB/s (7295kB/s), 7124KiB/s-7124KiB/s (7295kB/s-7295kB/s), io=696MiB (730MB), run=100005-100005msec
Disk stats (read/write):
sdq: ios=1597290/178061, merge=0/1, ticks=7961492/7531, in_queue=7973957, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/r -direct=1 -iodepth 1 -thread -rw=read -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 75 (f=75): [R(3),E(1),R(5),E(1),R(2),E(1),R(4),E(1),R(59),E(1),R(2)][100.0%][r=62.8MiB/s,w=0KiB/s][r=16.1k,w=0 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=192953: Mon Aug 2 15:55:51 2021
read: IOPS=15.2k, BW=59.3MiB/s (62.1MB/s)(5927MiB/100005msec)
clat (usec): min=86, max=342933, avg=5270.92, stdev=8000.98
lat (usec): min=87, max=342933, avg=5271.09, stdev=8001.00
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 5],
| 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 5],
| 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 6],
| 99.00th=[ 27], 99.50th=[ 54], 99.90th=[ 138], 99.95th=[ 159],
| 99.99th=[ 201]
bw ( KiB/s): min= 15, max= 1048, per=1.25%, avg=758.49, stdev=223.25, samples=15969
iops : min= 3, max= 262, avg=189.59, stdev=55.82, samples=15969
lat (usec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.25%, 4=2.69%, 10=95.13%, 20=0.66%, 50=0.71%
lat (msec) : 100=0.31%, 250=0.24%, 500=0.01%
cpu : usr=0.06%, sys=0.28%, ctx=1518051, majf=0, minf=140
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1517401,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=59.3MiB/s (62.1MB/s), 59.3MiB/s-59.3MiB/s (62.1MB/s-62.1MB/s), io=5927MiB (6215MB), run=100005-100005msec
Disk stats (read/write):
sdq: ios=1513410/14, merge=0/1, ticks=7961404/0, in_queue=7967479, util=100.00%
[root@bj-wlj-datanode09-172-38-1-210 ~]# fio -filename=/data15/w -direct=1 -iodepth 1 -thread -rw=write -rwmixread=90 -ioengine=psync -bs=4k -size=10G -numjobs=80 -runtime=100 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 80 threads
mytest: Laying out IO file (1 file / 10240MiB)
Jobs: 80 (f=80): [W(80)][100.0%][r=0KiB/s,w=437MiB/s][r=0,w=112k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=80): err= 0: pid=194453: Mon Aug 2 15:58:08 2021
write: IOPS=111k, BW=434MiB/s (456MB/s)(42.4GiB/100003msec)
clat (usec): min=50, max=3562, avg=718.51, stdev=25.02
lat (usec): min=50, max=3562, avg=718.65, stdev=25.02
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 693], 10.00th=[ 701], 20.00th=[ 709],
| 30.00th=[ 709], 40.00th=[ 709], 50.00th=[ 717], 60.00th=[ 717],
| 70.00th=[ 725], 80.00th=[ 734], 90.00th=[ 742], 95.00th=[ 750],
| 99.00th=[ 783], 99.50th=[ 791], 99.90th=[ 898], 99.95th=[ 1123],
| 99.99th=[ 1385]
bw ( KiB/s): min= 5272, max= 5624, per=1.25%, avg=5560.12, stdev=25.56, samples=15966
iops : min= 1318, max= 1406, avg=1390.03, stdev= 6.39, samples=15966
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=94.06%, 1000=5.87%
lat (msec) : 2=0.07%, 4=0.01%
cpu : usr=0.27%, sys=7.46%, ctx=24318385, majf=0, minf=4092
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,11121198,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=434MiB/s (456MB/s), 434MiB/s-434MiB/s (456MB/s-456MB/s), io=42.4GiB (45.6GB), run=100003-100003msec
Disk stats (read/write):
sdq: ios=0/11105144, merge=0/6, ticks=0/7430033, in_queue=7451498, util=100.00%