1. Ceph rbd 与 rgw的寻址(rbd块/对象存储文件的下载)
1.1. 索引的存储
ceph的索引都存储在omap中
- rbd – 每个rbd池有一个
rbd_directory文件 - rgw – 每一个bucket有一个或者多个index文件
1.2. rbd 的寻址
-
接收rbd客户端请求
请求中带着
- rbd 认证信息
- rbd 池名称
- rbd 名称
-
如何通过请求信息找到osd上对应的4m对象文件
-
每个rbd池都有一个
rbd_directory文件
该文件中/文件的元数据中(format 1/format 2)存储这个池所有的 image id和image name的双向映射例如
rados -p test001 listomapvals rbd_directory id_603d6b8b4567 value (8 bytes) : 00000000 04 00 00 00 74 65 73 74 |....test| 00000008 id_60486b8b4567 value (8 bytes) : 00000000 04 00 00 00 61 61 61 61 |....aaaa| 00000008 name_aaaa value (16 bytes) : 00000000 0c 00 00 00 36 30 34 38 36 62 38 62 34 35 36 37 |....60486b8b4567| 00000010 name_test value (16 bytes) : 00000000 0c 00 00 00 36 30 33 64 36 62 38 62 34 35 36 37 |....603d6b8b4567| 00000010可以通过rbd name /rbd id快速找到对应的 rbd id/rbd name
-
rbd 的对象名都为 rbd_head.{id}.{序列号}
所以知道id之后就可以找到这个卷对应的所有对象文件 -
关于rbd其他信息，例如快照等，通过获取rbd_header.{id}的omap信息可以得到
- create_timestamp – 创建的时间戳
- features – rbd features
- object_prefix – rbd prefix(rbd_data.{id})
- order – 2**order = rbd block
- parent – 父卷id 和 snapid，(父卷 -> snap -> clone(子卷))
- size – rbd size
- snap_seq – 快照信息
create_timestamp value (8 bytes) : 00000000 bb 39 68 60 72 aa 49 1b |.9h`r.I.| 00000008 features value (8 bytes) : 00000000 3d 00 00 00 00 00 00 00 |=.......| 00000008 object_prefix value (25 bytes) : 00000000 15 00 00 00 72 62 64 5f 64 61 74 61 2e 36 30 34 |....rbd_data.604| 00000010 38 36 62 38 62 34 35 36 37 |86b8b4567| 00000019 order value (1 bytes) : 00000000 16 |.| 00000001 parent value (48 bytes) : 00000000 01 01 2a 00 00 00 08 00 00 00 00 00 00 00 0e 00 |..*.............| 00000010 00 00 66 62 64 66 61 63 35 63 65 66 61 31 66 38 |..fbdfac5cefa1f8| 00000020 de 00 00 00 00 00 00 00 00 00 00 00 19 00 00 00 |................| 00000030 size value (8 bytes) : 00000000 00 00 00 80 02 00 00 00 |........| 00000008 snap_seq snapshot_0000000000000173 value (113 bytes) : 00000000 06 01 6b 00 00 00 73 01 00 00 00 00 00 00 04 00 |..k...s.........| 00000010 00 00 74 65 73 74 00 00 00 00 19 00 00 00 3d 00 |..test........=.| 00000020 00 00 00 00 00 00 01 01 2a 00 00 00 08 00 00 00 |........*.......| 00000030 00 00 00 00 0e 00 00 00 66 62 64 66 61 63 35 63 |........fbdfac5c| 00000040 65 66 61 31 66 38 de 00 00 00 00 00 00 00 00 00 |efa1f8..........| 00000050 00 00 19 00 00 00 00 00 00 00 00 00 00 00 00 01 |................| 00000060 01 04 00 00 00 00 00 00 00 c4 7e 69 60 bc 83 2c |..........~i`..,| 00000070 0b |.| 00000071
-
-
找到之后返回给客户端组装，就能得到一个完整的rbd
1.3. 小笔记
顺便记一下 rados listompvals 等omap操作命令的流程
- mon使用 crush 算法算出对象所在的osd
- 然后去osd上查omap，查完给客户端返回结果
1.4. rgw的寻址
-
接收客户端请求
请求中带有
- 认证信息
- bucket名
- 对象名
-
如何通过请求信息获取到该rgw对象对应的所有4m块的小文件对象
- 通过bucket可以获取到index对象id
- 通过crush算出index所在osd
- 通过查询omap信息获取所有的对象信息
- 返回rgw
- rgw组装返回
1.5. 数据恢复思路
1.5.1. 场景
机房掉电后，数据损坏，omap损坏，无法启动服务
集群无法读写，也读不到rbd信息
1.5.2. 思路
新建集群，将元数据导入，再将数据导入
- 新建一个和当前集群规模(osd个数)一样的集群， 保证crush之后的结果相同
- 创建池 池的信息可以通过
meta目录 获取osdmap crushmap也可以通过osdmap获取
# crush获取
osdmaptool osdmap.43__0_641716DC__none --export-crush /tmp/crushmap.bin
crushtool -d /tmp/crushmap.bin -o /tmp/crushmap.txt
# omap
osdmaptool --print osdmap.43__0_641716DC__none
#----
epoch 43
fsid 8685ec71-96a6-413a-9e4d-ff47071dc4f5
created 2020-12-22 16:57:37.845173
modified 2021-04-04 15:51:46.729929
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 6
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
pool 1 '.rgw.root' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 7 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 2 'test001' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 40 flags hashpspool stripe_width 0
removed_snaps [1~3]
pool 3 'default.rgw.control' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 21 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.meta' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 23 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.log' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 25 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.buckets.index' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 28 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 7 'default.rgw.buckets.data' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 31 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 8 'default.rgw.buckets.non-ec' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 34 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
max_osd 1
osd.0 up in weight 1 up_from 42 up_thru 42 down_at 41 last_clean_interval [37,40) 172.16.10.10:6801/12622 172.16.10.10:6802/12622 172.16.10.10:6803/12622 172.16.10.10:6804/12622 exists,up 59ebf8d5-e7f7-4c46-8e05-bac5140eee89
- 获取池的rbd_directory元数据
- 获取rbd_header元数据
- 通过map获取rbd_directory rbd_header 所在osd，将元数据put到omap
- 重建元数据之后 可以omap替换到旧的集群，也可以将旧集群的数据拷贝到新集群，启动osd
看似可行，但pg与osdmap的关系，omap版本问题看着不太好解决，有场景或者有需求的时候再来看
2万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
