ceph 性能优化

写 4M 数据

write: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=rbd, iodepth=32
fio-3.7
Starting 1 thread
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=244MiB/s][r=0,w=61 IOPS][eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=92589: Sat Sep 12 22:29:12 2020
  Description  : ["test with block size"]
  write: IOPS=52, BW=212MiB/s (222MB/s)(20.8GiB/100525msec)
    slat (usec): min=1246, max=5938, avg=2001.48, stdev=393.38
    clat (msec): min=99, max=1156, avg=602.70, stdev=81.81
     lat (msec): min=101, max=1159, avg=604.70, stdev=81.80
    clat percentiles (msec):
     |  1.00th=[  435],  5.00th=[  493], 10.00th=[  523], 20.00th=[  550],
     | 30.00th=[  567], 40.00th=[  584], 50.00th=[  592], 60.00th=[  609],
     | 70.00th=[  617], 80.00th=[  651], 90.00th=[  701], 95.00th=[  743],
     | 99.00th=[  869], 99.50th=[  936], 99.90th=[ 1070], 99.95th=[ 1099],
     | 99.99th=[ 1150]
   bw (  KiB/s): min= 8192, max=294912, per=99.42%, avg=215398.93, stdev=28200.11, samples=201
   iops        : min=    2, max=   72, avg=52.51, stdev= 6.89, samples=201
  lat (msec)   : 100=0.02%, 250=0.17%, 500=5.91%, 750=89.13%, 1000=4.51%
  cpu          : usr=10.38%, sys=0.46%, ctx=4043, majf=0, minf=88160
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5317,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=212MiB/s (222MB/s), 212MiB/s-212MiB/s (222MB/s-222MB/s), io=20.8GiB (22.3GB), run=100525-100525msec

Disk stats (read/write):
  sdg: ios=1/838, merge=0/583, ticks=6/736, in_queue=578, util=0.40%
 

新写 4K 数据

Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=13.1MiB/s][r=0,w=3360 IOPS][eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=92666: Sat Sep 12 22:34:13 2020
  Description  : ["test with block size"]
  write: IOPS=2882, BW=11.3MiB/s (11.8MB/s)(1126MiB/100011msec)
    slat (nsec): min=1398, max=1234.4k, avg=11009.79, stdev=9508.77
    clat (usec): min=741, max=275450, avg=11089.24, stdev=10265.50
     lat (usec): min=765, max=275466, avg=11100.25, stdev=10265.60
    clat percentiles (usec):
     |  1.00th=[  1565],  5.00th=[  2933], 10.00th=[  3949], 20.00th=[  5407],
     | 30.00th=[  6325], 40.00th=[  7111], 50.00th=[  7898], 60.00th=[  8979],
     | 70.00th=[ 10421], 80.00th=[ 14222], 90.00th=[ 23200], 95.00th=[ 29492],
     | 99.00th=[ 51643], 99.50th=[ 60556], 99.90th=[ 85459], 99.95th=[105382],
     | 99.99th=[254804]
   bw (  KiB/s): min= 3528, max=15632, per=99.99%, avg=11527.17, stdev=2887.35, samples=200
   iops        : min=  882, max= 3908, avg=2881.77, stdev=721.83, samples=200
  lat (usec)   : 750=0.01%, 1000=0.04%
  lat (msec)   : 2=2.10%, 4=8.11%, 10=57.24%, 20=19.29%, 50=12.06%
  lat (msec)   : 100=1.09%, 250=0.05%, 500=0.01%
  cpu          : usr=4.58%, sys=2.75%, ctx=148334, majf=0, minf=20541
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,288238,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=11.3MiB/s (11.8MB/s), 11.3MiB/s-11.3MiB/s (11.8MB/s-11.8MB/s), io=1126MiB (1181MB), run=100011-100011msec

Disk stats (read/write):
  sdg: ios=0/760, merge=0/553, ticks=0/117, in_queue=0, util=0.32%
------------------------------------------------------------------------------------------------------------------------------

修改写4K

write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.7
Starting 1 thread
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=27.5MiB/s][r=0,w=7039 IOPS][eta 00m:00s]
write: (groupid=0, jobs=1): err= 0: pid=92743: Sat Sep 12 22:36:43 2020
  Description  : ["test with block size"]
  write: IOPS=6963, BW=27.2MiB/s (28.5MB/s)(2720MiB/100003msec)
    slat (nsec): min=1375, max=1233.8k, avg=9729.99, stdev=9027.33
    clat (usec): min=460, max=256722, avg=4583.43, stdev=4906.07
     lat (usec): min=470, max=256737, avg=4593.16, stdev=4906.31
    clat percentiles (usec):
     |  1.00th=[  1004],  5.00th=[  1762], 10.00th=[  2376], 20.00th=[  2900],
     | 30.00th=[  3228], 40.00th=[  3490], 50.00th=[  3752], 60.00th=[  4047],
     | 70.00th=[  4359], 80.00th=[  4883], 90.00th=[  5997], 95.00th=[  8455],
     | 99.00th=[ 26084], 99.50th=[ 30278], 99.90th=[ 50070], 99.95th=[ 58459],
     | 99.99th=[127402]
   bw (  KiB/s): min= 3184, max=36752, per=99.99%, avg=27850.31, stdev=7509.90, samples=200
   iops        : min=  796, max= 9188, avg=6962.59, stdev=1877.48, samples=200
  lat (usec)   : 500=0.01%, 750=0.18%, 1000=0.80%
  lat (msec)   : 2=5.56%, 4=52.41%, 10=37.01%, 20=1.74%, 50=2.20%
  lat (msec)   : 100=0.07%, 250=0.03%, 500=0.01%
  cpu          : usr=9.48%, sys=5.02%, ctx=321603, majf=0, minf=61932
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,696346,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=27.2MiB/s (28.5MB/s), 27.2MiB/s-27.2MiB/s (28.5MB/s-28.5MB/s), io=2720MiB (2852MB), run=100003-100003msec

Disk stats (read/write):
  sdg: ios=0/784, merge=0/598, ticks=0/118, in_queue=0, util=0.31%

---------------------------------------------------------------------------------------------------------------------------------------------
读4M

Jobs: 1 (f=1): [R(1)][14.8%][r=32.0MiB/s,w=0KiB/s][r=8,w=0 IOPS][eta 09m:41s] 
READ: (groupid=0, jobs=1): err= 0: pid=92903: Sat Sep 12 22:40:50 2020
  Description  : ["test with block"]
   read: IOPS=37, BW=151MiB/s (159MB/s)(14.9GiB/101025msec)
    slat (nsec): min=1509, max=158928, avg=14261.88, stdev=4433.00
    clat (msec): min=129, max=7034, avg=845.99, stdev=969.07
     lat (msec): min=129, max=7034, avg=846.01, stdev=969.07
    clat percentiles (msec):
     |  1.00th=[  186],  5.00th=[  209], 10.00th=[  224], 20.00th=[  257],
     | 30.00th=[  292], 40.00th=[  338], 50.00th=[  393], 60.00th=[  506],
     | 70.00th=[  776], 80.00th=[ 1318], 90.00th=[ 2232], 95.00th=[ 3037],
     | 99.00th=[ 4396], 99.50th=[ 5269], 99.90th=[ 6678], 99.95th=[ 6812],
     | 99.99th=[ 7013]
   bw (  KiB/s): min= 8192, max=245760, per=99.70%, avg=154415.95, stdev=28231.48, samples=201
   iops        : min=    2, max=   60, avg=37.67, stdev= 6.89, samples=201
  lat (msec)   : 250=18.19%, 500=41.34%, 750=9.63%, 1000=6.10%
  cpu          : usr=0.22%, sys=0.15%, ctx=3825, majf=0, minf=82
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.4%, 32=99.2%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=3820,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=151MiB/s (159MB/s), 151MiB/s-151MiB/s (159MB/s-159MB/s), io=14.9GiB (16.0GB), run=101025-101025msec

Disk stats (read/write):
  sdg: ios=1/835, merge=0/602, ticks=0/1128, in_queue=972, util=0.39%

 

随机读4K

READ: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32
fio-3.7
Starting 1 thread
Jobs: 1 (f=1): [r(1)][100.0%][r=14.2MiB/s,w=0KiB/s][r=3645,w=0 IOPS][eta 00m:00s]
READ: (groupid=0, jobs=1): err= 0: pid=93128: Sat Sep 12 22:42:53 2020
  Description  : ["test with block"]
   read: IOPS=10.5k, BW=41.1MiB/s (43.1MB/s)(4111MiB/100017msec)
    slat (nsec): min=340, max=1023.0k, avg=3349.69, stdev=4069.03
    clat (usec): min=145, max=120012, avg=3036.57, stdev=7534.29
     lat (usec): min=147, max=120017, avg=3039.92, stdev=7535.22
    clat percentiles (usec):
     |  1.00th=[  388],  5.00th=[  502], 10.00th=[  611], 20.00th=[  758],
     | 30.00th=[  865], 40.00th=[  947], 50.00th=[ 1020], 60.00th=[ 1090],
     | 70.00th=[ 1205], 80.00th=[ 1434], 90.00th=[ 5211], 95.00th=[16712],
     | 99.00th=[41157], 99.50th=[50070], 99.90th=[68682], 99.95th=[76022],
     | 99.99th=[92799]
   bw (  KiB/s): min=11040, max=135584, per=100.00%, avg=42089.18, stdev=46583.14, samples=200
   iops        : min= 2760, max=33896, avg=10522.27, stdev=11645.77, samples=200
  lat (usec)   : 250=0.01%, 500=4.92%, 750=14.21%, 1000=28.66%
  lat (msec)   : 2=39.38%, 4=2.16%, 10=3.27%, 20=3.36%, 50=3.53%
  lat (msec)   : 100=0.51%, 250=0.01%
  cpu          : usr=8.11%, sys=6.76%, ctx=809383, majf=0, minf=5099
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1052371,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=4111MiB (4311MB), run=100017-100017msec

Disk stats (read/write):
  sdg: ios=1/841, merge=0/614, ticks=1/949, in_queue=801, util=0.39%
 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值