ceph中的PG和PGP

在最初使用ceph的时候就有一个疑问，ceph存储时将对象映射到PG(Placement Groups)中，然后以PG为单位，遵循CRUSH的规则分布到OSD中，那么PGP又是什么？因此决定自己探索一下。

ceph对象的映射关系架构图

环境介绍

[root@ceph01 my-cluster]# ceph osd tree
ID WEIGHT  TYPE NAME       UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 0.57088 root default                                      
-2 0.19029     host ceph02                                   
 0 0.19029         osd.0        up  1.00000          1.00000 
-3 0.19029     host ceph03                                   
 1 0.19029         osd.1        up  1.00000          1.00000 
-4 0.19029     host ceph01                                   
 2 0.19029         osd.2        up  1.00000          1.00000 

操作验证

1.创建test_pool

[root@ceph01 my-cluster]# ceph osd pool create test_pool 6 6
pool 'test_pool' created

2.查看创建之后的PG分布情况

使用命令

[root@ceph01 my-cluster]# ceph pg ls-by-pool test_pool | awk '{print $1,$2,$15}'
pg_stat objects up_primary
10.0 0 [2,0]
10.1 0 [0,2]
10.2 0 [1,2]
10.3 0 [0,2]
10.4 0 [1,2]
10.5 0 [1,2]

可以看出test_pool的pg都是以10开头的。然后也可以用其他命令查询PG的分布情况，如下：

[root@ceph01 my-cluster]#  ceph pg dump pgs|grep ^10|awk '{print $1,$2,$15}'
dumped pgs in format plain
10.2 0 [1,2]
10.3 0 [0,2]
10.0 0 [2,0]
10.1 0 [0,2]
10.4 0 [1,2]
10.5 0 [1,2]

3.写测试数据到test_pool

[root@ceph01 my-cluster]# rados -p test_pool bench 20 write --no-cleanup

4.再次查询PG分布情况

[root@ceph01 my-cluster]# ceph pg ls-by-pool test_pool | awk '{print $1,$2,$15}'
pg_stat objects up_primary
10.0 87 [2,0]
10.1 71 [0,2]
10.2 157 [1,2]
10.3 182 [0,2]
10.4 76 [1,2]
10.5 77 [1,2]

可以看见，第二列的对象数目增加了，但是PG的分布并无变化。

5.增大pg_num测试

[root@ceph01 my-cluster]# ceph osd pool set test_pool pg_num 12
set pool 10 pg_num to 12
[root@ceph01 my-cluster]# ceph pg ls-by-pool test_pool | awk '{print $1,$2,$15}'
pg_stat objects up_primary
10.0 43 [2,0]
10.1 35 [0,2]
10.2 52 [1,2]
10.3 60 [0,2]
10.4 76 [1,2]
10.5 77 [1,2]
10.6 53 [1,2]
10.7 61 [0,2]
10.8 44 [2,0]
10.9 36 [0,2]
10.a 52 [1,2]
10.b 61 [0,2]

整理之后，数据我门根据对象数，和副本的分布，我们可以明显的发现分裂的痕迹

10.0 -> 10.0 + 10.8
10.1 -> 10.1 + 10.9
10.2 -> 10.2 + 10.6 + 10.a
10.3 -> 10.3 + 10.7 + 10.b
10.4 -> 10.4
10.5 -> 10.5

也就是说，增大pg_num，原来大的pg会平均分裂成几个小的PG，而原来的PG之间是不会互相迁移数据的。

6.增大pgp_num测试

[root@ceph01 my-cluster]# ceph osd pool set test_pool pgp_num 12
set pool 10 pgp_num to 12
[root@ceph01 my-cluster]# ceph pg ls-by-pool test_pool | awk '{print $1,$2,$15}'
pg_stat objects up_primary
10.0 43 [2,0]
10.1 35 [0,2]
10.2 52 [1,2]
10.3 60 [0,2]
10.4 76 [1,2]
10.5 77 [1,2]
10.6 53 [0,2]
10.7 61 [2,1]
10.8 44 [1,0]
10.9 36 [2,0]
10.a 52 [0,2]
10.b 61 [0,1]

我们乍一看好像没什么变化，但是仔细一看就会发现不同，我们来对比一下

修改前           修改后
10.0 43 [2,0]   10.0 43 [2,0]
10.1 35 [0,2]   10.1 35 [0,2]
10.2 52 [1,2]   10.2 52 [1,2]
10.3 60 [0,2]   10.3 60 [0,2]
10.4 76 [1,2]   10.4 76 [1,2]
10.5 77 [1,2]   10.5 77 [1,2]
10.6 53 [1,2]*   10.6 53 [0,2]*
10.7 61 [0,2]*   10.7 61 [2,1]*
10.8 44 [2,0]*   10.8 44 [1,0]*
10.9 36 [0,2]*   10.9 36 [2,0]*
10.a 52 [1,2]*   10.a 52 [0,2]*
10.b 61 [0,2]*   10.b 61 [0,1]*

我们可以看见，10.6至10.b的分布都已经发生了变化。

结论

从测试结果中，我们可以得出以下结论:

• pg_num的增加会使原来PG中的对象均匀地分布到新建的PG中，原来的副本分布方式不变
• pgp_num的增加会使PG的分布方式发生变化，但是PG内的对象并不会变动
• pgp决定pg分布时的组合方式的变化

另外，附加一段网上的解释供大家参考

PG = Placement Group
PGP = Placement Group for Placement purpose
pg_num = number of placement groups mapped to an OSD
When pgnum is increased for any pool, every PG of this pool splits into half, but they all remain mapped to their parent OSD.Until this time, Ceph does not start rebalancing. Now, when you increase the pgpnum value for the same pool, PGs start to migrate from the parent to some other OSD, and cluster rebalancing starts. This is how PGP plays an important role.
By Karan Singh

本文参考了zphj1987的博客：http://www.zphj1987.com/2016/10/19/Ceph中PG和PGP的区别/

---