ceph分布式存储-块存储BlueStore性能测试

一、 集群环境

1.1 部署环境说明

mon组件部署： ceph-xxx-osd00, ceph-xxx-osd01, ceph-xxx-osd02
osd组件部署： ceph-xxx-osd00, ceph-xxx-osd01, ceph-xxx-osd02，ceph-xxx-osd03, ceph-xxx-osd04
磁盘： SATA
ceph版本： ceph 12.2.2 bluestore

二、 性能测试

2.1 rados bench测试

2.1.1 写测试

//默认block size是4M，30个线程并发，测试时间200s
测试结果：30线程并发写，带宽：1119.68 MB/s 平均IOPS：279 平均延迟：0.107s

rados bench -p test_pool 200 write -t 30 --no-cleanup  --cluster=test
Total time run:         200.075896
Total writes made:      56005
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     1119.68
Stddev Bandwidth:       13.3296
Max bandwidth (MB/sec): 1148
Min bandwidth (MB/sec): 1048
Average IOPS:           279
Stddev IOPS:            3
Max IOPS:               287
Min IOPS:               262
Average Latency(s):     0.107161
Stddev Latency(s):      0.0469331
Max latency(s):         0.50955
Min latency(s):         0.0251761

2.1.2 顺序读

//默认block size是4M，30个线程并发，测试时间200s
测试结果：30线程并发，带宽：1121.07 MB/s 平均IOPS：280 平均延迟0.106s

rados bench -p test_pool 200 seq -t 30 --no-cleanup  --cluster=test
Total time run:       199.827279
Total reads made:     56005
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1121.07
Average IOPS:         280
Stddev IOPS:          2
Max IOPS:             288
Min IOPS:             260
Average Latency(s):   0.106283
Max latency(s):       0.780835
Min latency(s):       0.0198169

2.1.3 随机读

//默认block size是4M，30个线程并发，测试时间200s
测试结果：30线程并发，带宽：1109.64 MB/s 平均IOPS：279 平均延迟：0.106s

rados bench -p test_pool 200 rand -t 30 --no-cleanup --cluster=test
Total time run:       200.074550
Total reads made:     56003
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   1119.64
Average IOPS:         279
Stddev IOPS:          3
Max IOPS:             286
Min IOPS:             245
Average Latency(s):   0.106435
Max latency(s):       0.732702
Min latency(s):       0.0216828

2.2 rbd性能测试

2.2.1 顺序读写

//默认block size是4k，30个线程并发
测试结果：30线程并发，带宽：297 MB/s 平均IOPS：72681.76

rbd bench-write test_image  --io-threads 30  --pool=test_pool   --io-pattern seq --io-total 17199730000 --cluster=test
elapsed:    57  ops:  4199153  ops/sec: 72681.76  bytes/sec: 297704487.46

//block size是4M，30个线程并发
测试结果：30线程并发，带宽：844MB/s 平均IOPS：206.16

rbd bench-write test_image  --io-threads 30  --pool=test_pool   --io-pattern seq --io-total 17199730000 --io-size 4096000 --cluster=test 
elapsed:    20  ops:     4200  ops/sec:   206.16  bytes/sec: 844451109.43

2.2.2 随机读写

//默认block size是4k，30个线程并发
测试结果：30线程并发，带宽：36 MB/s 平均IOPS：8925.40

rbd bench-write test_image  --io-threads 30  --pool=test_pool   --io-pattern rand --io-total 17199730000  --cluster=test
elapsed:   470  ops:  4199153  ops/sec:  8925.40  bytes/sec: 36558420.10

//block size是4M，30个线程并发
测试结果：30线程并发，带宽：860 MB/s 平均IOPS： 210.18

rbd bench-write test_image  --io-threads 30  --pool=test_pool   --io-pattern rand --io-total 17199730000 --io-size 4096000 --cluster=test
elapsed:    19  ops:     4200  ops/sec:   210.18  bytes/sec: 860894998.27

2.3 fio+libaio测试

2.3.1 顺序读(block size 4M)

//block size是4M, 30个线程并发，持续时间200s
测试结果：30线程并发，带宽：2365.5 MB/s 平均IOPS： 591 耗时：50.58ms

fio -filename=/mnt/test/xxx -direct=1 -iodepth 1 -thread -rw=read -ioengine=libaio -bs=4M -size=1G -numjobs=30 -runtime=200 -group_reporting -name=read-libaio
 
read-libaio: (g=0): rw=read, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=1
...
fio-2.2.8
Starting 30 threads
read-libaio: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 30 (f=30): [R(30)] [100.0% done] [2580MB/0KB/0KB /s] [645/0/0 iops] [eta 00m:00s]
read-libaio: (groupid=0, jobs=30): err= 0: pid=32265: Tue Jan 16 14:42:45 2018
  read : io=30720MB, bw=2365.5MB/s, iops=591, runt= 12987msec
    slat (usec): min=381, max=111307, avg=39421.29, stdev=12150.39
    clat (usec): min=806, max=99391, avg=11156.56, stdev=8561.94
     lat (msec): min=2, max=127, avg=50.58, stdev=11.41
    clat percentiles (usec):
     |  1.00th=[ 4128],  5.00th=[ 6496], 10.00th=[ 7072], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 9152], 80.00th=[14528], 90.00th=[21888], 95.00th=[29568],
     | 99.00th=[47360], 99.50th=[49920], 99.90th=[54528], 99.95th=[94720],
     | 99.99th=[99840]
    bw (KB  /s): min=67764, max=105233, per=3.33%, avg=80770.99, stdev=4432.59
    lat (usec) : 1000=0.04%
    lat (msec) : 2=0.25%, 4=0.55%, 10=69.62%, 20=17.27%, 50=11.76%
    lat (msec) : 100=0.52%
  cpu          : usr=0.01%, sys=0.79%, ctx=23227, majf=0, minf=15436
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7680/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
   READ: io=30720MB, aggrb=2365.5MB/s, minb=2365.5MB/s, maxb=2365.5MB/s, mint=12987msec, maxt=12987msec
Disk stats (read/write):
  nbd0: ios=243854/2, merge=0/1, ticks=1801606/27, in_queue=1802231, util=99.28%

2.3.1 随机读(block size 4M)

//block size是4M, 30个线程并发，持续时间200s
测试结果：30线程并发，带宽：1172.9 MB/s 平均IOPS： 293 耗时：101.27ms

fio -filename=/mnt/test/xxx -direct=1 -iodepth 1 -thread -rw=randread -ioengine=libaio -bs=4M -size=1G -numjobs=30 -runtime=200 -group_reporting -name=read-libaio
 
read-libaio: (g=0): rw=randread, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=1
...
fio-2.2.8
Starting 30 threads
read-libaio: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 19 (f=18): [_(1),r(3),_(1),r(3),_(1),r(2),_(1),r(1),_(1),r(1),_(1),r(1),_(2),r(5),_(1),r(3),_(2)] [96.4% done] [1312MB/0KB/0KB /s] [328/0/0 iops] [eta 00m:01s]
read-libaio: (groupid=0, jobs=30): err= 0: pid=34255: Tue Jan 16 14:50:14 2018
  read : io=30720MB, bw=1172.9MB/s, iops=293, runt= 26194msec
    slat (usec): min=320, max=215905, avg=76016.00, stdev=31251.51
    clat (usec): min=774, max=249806, avg=25251.42, stdev=16090.29
     lat (msec): min=1, max=341, avg=101.27, stdev=35.33
    clat percentiles (usec):
     |  1.00th=[ 1544],  5.00th=[ 8256], 10.00th=[12992], 20.00th=[17024],
     | 30.00th=[19584], 40.00th=[21632], 50.00th=[23680], 60.00th=[25728],
     | 70.00th=[28032], 80.00th=[31360], 90.00th=[36608], 95.00th=[43264],
     | 99.00th=[66048], 99.50th=[102912], 99.90th=[228352], 99.95th=[238592],
     | 99.99th=[248832]
    bw (KB  /s): min=19200, max=81593, per=3.34%, avg=40089.91, stdev=6232.06
    lat (usec) : 1000=0.03%
    lat (msec) : 2=2.71%, 4=0.86%, 10=2.83%, 20=25.34%, 50=65.53%
    lat (msec) : 100=2.20%, 250=0.51%
  cpu          : usr=0.01%, sys=0.43%, ctx=23282, majf=0, minf=15444
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=7680/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : targ