ceph（luminous）-基于pgupmap单个pool数据均衡方案

原理

• 由于ceph分片的特性，使得数据分配均衡就可以认为是pool中osd的pg数目分布大致均衡.
• pgupmap作为ceph12新加入的特性，在osdmap中记录了某个pg的单独映射，客户端计算完crushmap后需要遍历upmap，修改pg和osd的对应关系才能找到正确对应的osd，12版本 新加入的balancer的一个方案就是基于upmap进行均衡的，不过balancer是调节全部的pool，无法对于单个pool进行调节，其实balancer可以满足了很多情况下的均衡问题，不过还存在一些需要单独调节单个pool得场景.（比如添加节点后为了防止迁移单独对新存储池进行调节）
  ps：reweight-by-pg可以对于单个存储池进行调节，但是如果多个pool公用一组osd，由于reweight改变了，其他的pool映射也会改变.

pgupmap举例

形如以下：
pg_upmap_items 4.1 [17,15,8,7] // 对于pg4.1，把应该映射到17的osd改为15，应该映射到8的osd改为7.
pg_upmap_items 4.6 [12,15,8,7]
pg_upmap_items 4.8 [11,6,17,15]
pg_upmap_items 4.a [8,6]
pg_upmap_items 4.b [11,6,16,15]
可以通过ceph osd dump 查看

调节步骤

之前没有发现这个工具，还自己写了个脚本去调节，后来发现ceph已经集成了这样得工具了，真是太方便了,是利用osdmaptool工具进行调节.

1. 利用ceph osd getmap 获取最新得osdmap

[root@node01 autoreweight]# ceph osd getmap > osdmap.bin //把osdmap导入到文件
got osdmap epoch 7908

2. osdmaptool --upmap-pool [poolname] [osdmapfile] --upmap [outfilename]

[root@node01 autoreweight]# osdmaptool --upmap-pool volumes osdmap.bin --upmap upmap.txt
osdmaptool: osdmap file 'osdmap.bin'
writing upmap command output to: upmap.txt
checking for upmap cleanups
upmap, max-count 100, max deviation 0.01
 limiting to pools volumes (21)
[root@node01 autoreweight]# cat upmap.txt 
ceph osd pg-upmap-items 21.1 1 5
ceph osd pg-upmap-items 21.2 11 6 12 14 2 5
ceph osd pg-upmap-items 21.3 8 6 1 5
ceph osd pg-upmap-items 21.4 11 6
ceph osd pg-upmap-items 21.5 17 14 11 6 4 5
ceph osd pg-upmap-items 21.8 8 6
ceph osd pg-upmap-items 21.a 17 14
ceph osd pg-upmap-items 21.d 16 14 11 6
ceph osd pg-upmap-items 21.e 11 7
ceph osd pg-upmap-items 21.f 8 6 1 0
ceph osd pg-upmap-items 21.10 11 9
ceph osd pg-upmap-items 21.11 2 5 17 14
ceph osd pg-upmap-items 21.17 3 5 17 14
ceph osd pg-upmap-items 21.1a 8 7
ceph osd pg-upmap-items 21.1b 16 13
ceph osd pg-upmap-items 21.1d 17 14
ceph osd pg-upmap-items 21.1e 11 6
ceph osd pg-upmap-items 21.20 17 14 10 6
ceph osd pg-upmap-items 21.36 17 14
ceph osd pg-upmap-items 21.3d 8 7 17 14
ceph osd pg-upmap-items 21.43 17 14

3. source 一下导出得upmap方案

21
0:83
1:89
2:88
3:87
4:87
5:78
6:76
7:82
8:91
9:83
10:87
11:93
12:87
13:84
14:73
15:85
16:88
17:95
//以上是执行source upmap.txt之前的pg再osd上得分布.
[root@node01 autoreweight]# source upmap.txt 
set 21.1 pg_upmap_items mapping to [1->5]
set 21.2 pg_upmap_items mapping to [11->6,12->14,2->5]
set 21.3 pg_upmap_items mapping to [8->6,1->5]
set 21.4 pg_upmap_items mapping to [11->6]
set 21.5 pg_upmap_items mapping to [17->14,11->6,4->5]
set 21.8 pg_upmap_items mapping to [8->6]
set 21.a pg_upmap_items mapping to [17->14]
set 21.d pg_upmap_items mapping to [16->14,11->6]
set 21.e pg_upmap_items mapping to [11->7]
set 21.f pg_upmap_items mapping to [8->6,1->0]
set 21.10 pg_upmap_items mapping to [11->9]
set 21.11 pg_upmap_items mapping to [2->5,17->14]
set 21.17 pg_upmap_items mapping to [3->5,17->14]
set 21.1a pg_upmap_items mapping to [8->7]
set 21.1b pg_upmap_items mapping to [16->13]
set 21.1d pg_upmap_items mapping to [17->14]
set 21.1e pg_upmap_items mapping to [11->6]
set 21.20 pg_upmap_items mapping to [17->14,10->6]
set 21.36 pg_upmap_items mapping to [17->14]
set 21.3d pg_upmap_items mapping to [8->7,17->14]
set 21.43 pg_upmap_items mapping to [17->14]
//以下是调节之后得，osd之前pg数目相差无几了.
21
0:84
1:86
2:86
3:86
4:86
5:84
6:85
7:85
8:86
9:84
10:86
11:86
12:86
13:85
14:84
15:85
16:86
17:86