一、 集群环境
1.1 部署环境说明
mon组件部署: ceph-xxx-osd00, ceph-xxx-osd01, ceph-xxx-osd02
osd组件部署: ceph-xxx-osd00, ceph-xxx-osd01, ceph-xxx-osd02,ceph-xxx-osd03, ceph-xxx-osd04
磁盘: SATA
ceph版本: ceph 12.2.2 bluestore
二、 性能测试
2.1 rados bench测试
2.1.1 写测试
//默认block size是4M,30个线程并发,测试时间200s
测试结果:30线程并发写,带宽:1119.68 MB/s 平均IOPS:279 平均延迟:0.107s
rados bench -p test_pool 200 write -t 30 --no-cleanup --cluster=test
Total time run: 200.075896
Total writes made: 56005
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1119.68
Stddev Bandwidth: 13.3296
Max bandwidth (MB/sec): 1148
Min bandwidth (MB/sec): 1048
Average IOPS: 279
Stddev IOPS: 3
Max IOPS: 287
Min IOPS: 262
Average Latency(s): 0.107161
Stddev Latency(s): 0.0469331
Max latency(s): 0.50955
Min latency(s): 0.0251761
2.1.2 顺序读
//默认block size是4M,30个线程并发,测试时间200s
测试结果:30线程并发,带宽:1121.07 MB/s 平均IOPS:280 平均延迟0.106s
rados bench -p test_pool 200 seq -t 30 --no-cleanup --cluster=test
Total time run: 199.827279
Total reads made: 56005
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1121.07
Average IOPS: 280
Stddev IOPS: 2
Max IOPS: 288
Min IOPS: 260
Average Latency(s): 0.106283
Max latency(s): 0.780835
Min latency(s): 0.0198169
2.1.3 随机读
//默认block size是4M,30个线程并发,测试时间200s
测试结果:30线程并发,带宽:1109.64 MB/s 平均IOPS:279 平均延迟:0.106s
rados bench -p test_pool 200 rand -t 30 --no-cleanup --cluster=test
Total time run: 200.074550
Total reads made: 56003
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 1119.64
Average IOPS: 279
Stddev IOPS: 3
Max IOPS: 286
Min IOPS: 245
Average Latency(s): 0.106435
Max latency(s): 0.732702
Min latency(s): 0.0216828
2.2 rbd性能测试
2.2.1 顺序读写
//默认block size是4k,30个线程并发
测试结果:30线程并发,带宽:297 MB/s 平均IOPS:72681.76
rbd bench-write test_image --io-threads 30 --pool=test_pool --io-pattern seq --io-total 17199730000 --cluster=test
elapsed: 57 ops: 4199153 ops/sec: 72681.76 bytes/sec: 297704487.46
//block size是4M,30个线程并发
测试结果:30线程并发,带宽:844MB/s 平均IOPS:206.16
rbd bench-write test_image --io-threads 30 --pool=test_pool --io-pattern seq --io-total 17199730000 --io-size 4096000 --cluster=test
elapsed: 20 ops: 4200 ops/sec: 206.16 bytes/sec: 844451109.43
2.2.2 随机读写
//默认block size是4k,30个线程并发
测试结果:30线程并发,带宽:36 MB/s 平均IOPS:8925.40
rbd bench-write test_image --io-threads 30 --pool=test_pool --io-pattern rand --io-total 17199730000 --cluster=test
elapsed: 470 ops: 4199153 ops/sec: 8925.40 bytes/sec: 36558420.10
//block size是4M,30个线程并发
测试结果:30线程并发,带宽:860 MB/s 平均IOPS: 210.18
rbd bench-write test_image --io-threads 30 --pool=test_pool --io-pattern rand --io-total 17199730000 --io-size 4096000 --cluster=test
elapsed: 19 ops: 4200 ops/sec: 210.18 bytes/sec: 860894998.27
2.3 fio+libaio测试
2.3.1 顺序读(block size 4M)
//block size是4M, 30个线程并发,持续时间200s
测试结果:30线程并发,带宽:2365.5 MB/s 平均IOPS: 591 耗时:50.58ms
fio -filename=/mnt/test/xxx -direct=1 -iodepth 1 -thread -rw=read -ioengine=libaio -bs=4M -size=1G -numjobs=30 -runtime=200 -group_reporting -name=read-libaio
read-libaio: (g=0): rw=read, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=1
...
fio-2.2.8
Starting 30 threads
read-libaio: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 30 (f=30): [R(30)] [100.0% done] [2580MB/0KB/0KB /s] [645/0/0 iops] [eta 00m:00s]
read-libaio: (groupid=0, jobs=30): err= 0: pid=32265: Tue Jan 16 14:42:45 2018
read : io=30720MB, bw=2365.5MB/s, iops=591, runt= 12987msec
slat (usec): min=381, max=111307, avg=39421.29, stdev=12150.39
clat (usec): min=806, max=99391, avg=11156.56, stdev=8561.94
lat (msec): min=2, max=127, avg=50.58, stdev=11.41
clat percentiles (usec):
| 1.00th=[ 4128], 5.00th=[ 6496], 10.00th=[ 7072], 20.00th=[ 7072],
| 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7136], 60.00th=[ 7200],
| 70.00th=[ 9152], 80.00th=[14528], 90.00th=[21888], 95.00th=[29568],
| 99.00th=[47360], 99.50th=[49920], 99.90th=[54528], 99.95th=[94720],
| 99.99th=[99840]
bw (KB /s): min=67764, max=105233, per=3.33%, avg=80770.99, stdev=4432.59
lat (usec) : 1000=0.04%
lat (msec) : 2=0.25%, 4=0.55%, 10=69.62%, 20=17.27%, 50=11.76%
lat (msec) : 100=0.52%
cpu : usr=0.01%, sys=0.79%, ctx=23227, majf=0, minf=15436
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=7680/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: io=30720MB, aggrb=2365.5MB/s, minb=2365.5MB/s, maxb=2365.5MB/s, mint=12987msec, maxt=12987msec
Disk stats (read/write):
nbd0: ios=243854/2, merge=0/1, ticks=1801606/27, in_queue=1802231, util=99.28%
2.3.1 随机读(block size 4M)
//block size是4M, 30个线程并发,持续时间200s
测试结果:30线程并发,带宽:1172.9 MB/s 平均IOPS: 293 耗时:101.27ms
fio -filename=/mnt/test/xxx -direct=1 -iodepth 1 -thread -rw=randread -ioengine=libaio -bs=4M -size=1G -numjobs=30 -runtime=200 -group_reporting -name=read-libaio
read-libaio: (g=0): rw=randread, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=1
...
fio-2.2.8
Starting 30 threads
read-libaio: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 19 (f=18): [_(1),r(3),_(1),r(3),_(1),r(2),_(1),r(1),_(1),r(1),_(1),r(1),_(2),r(5),_(1),r(3),_(2)] [96.4% done] [1312MB/0KB/0KB /s] [328/0/0 iops] [eta 00m:01s]
read-libaio: (groupid=0, jobs=30): err= 0: pid=34255: Tue Jan 16 14:50:14 2018
read : io=30720MB, bw=1172.9MB/s, iops=293, runt= 26194msec
slat (usec): min=320, max=215905, avg=76016.00, stdev=31251.51
clat (usec): min=774, max=249806, avg=25251.42, stdev=16090.29
lat (msec): min=1, max=341, avg=101.27, stdev=35.33
clat percentiles (usec):
| 1.00th=[ 1544], 5.00th=[ 8256], 10.00th=[12992], 20.00th=[17024],
| 30.00th=[19584], 40.00th=[21632], 50.00th=[23680], 60.00th=[25728],
| 70.00th=[28032], 80.00th=[31360], 90.00th=[36608], 95.00th=[43264],
| 99.00th=[66048], 99.50th=[102912], 99.90th=[228352], 99.95th=[238592],
| 99.99th=[248832]
bw (KB /s): min=19200, max=81593, per=3.34%, avg=40089.91, stdev=6232.06
lat (usec) : 1000=0.03%
lat (msec) : 2=2.71%, 4=0.86%, 10=2.83%, 20=25.34%, 50=65.53%
lat (msec) : 100=2.20%, 250=0.51%
cpu : usr=0.01%, sys=0.43%, ctx=23282, majf=0, minf=15444
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=7680/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : targ