Ceph rbd的寻址(rbd块文件的下载)

gfengwong

已于 2022-09-22 19:44:11 修改

阅读量1.1k

点赞数

分类专栏： ceph 文章标签：云存储 ceph

于 2021-07-31 15:51:11 首次发布

本文链接：https://blog.csdn.net/Moolight_shadow/article/details/119276840

版权

ceph 专栏收录该内容

16 篇文章 1 订阅

订阅专栏

本文详细介绍了Ceph RBD块存储和RGW对象存储的寻址机制，包括rbd的池和对象的索引存储、rgw的bucket和对象寻址方式。同时，提出了数据恢复的思路，特别是在集群因数据损坏无法启动时，如何通过新建集群和导入元数据来恢复服务。内容涵盖了crush算法、omap操作以及池和对象的元数据信息。

摘要由CSDN通过智能技术生成

1. Ceph rbd 与 rgw的寻址(rbd块/对象存储文件的下载)

1. Ceph rbd 与 rgw的寻址(rbd块/对象存储文件的下载)

1.1. 索引的存储

ceph的索引都存储在omap中

rbd – 每个rbd池有一个 rbd_directory 文件
rgw – 每一个bucket有一个或者多个index文件

1.2. rbd 的寻址

接收rbd客户端请求

请求中带着
- rbd 认证信息
- rbd 池名称
- rbd 名称

如何通过请求信息找到osd上对应的4m对象文件

每个rbd池都有一个 rbd_directory 文件
该文件中/文件的元数据中(format 1/format 2)存储这个池所有的 image id和image name的双向映射

例如

rados -p test001  listomapvals rbd_directory
id_603d6b8b4567
value (8 bytes) :
00000000  04 00 00 00 74 65 73 74                           |....test|
00000008

id_60486b8b4567
value (8 bytes) :
00000000  04 00 00 00 61 61 61 61                           |....aaaa|
00000008

name_aaaa
value (16 bytes) :
00000000  0c 00 00 00 36 30 34 38  36 62 38 62 34 35 36 37  |....60486b8b4567|
00000010

name_test
value (16 bytes) :
00000000  0c 00 00 00 36 30 33 64  36 62 38 62 34 35 36 37  |....603d6b8b4567|
00000010

可以通过rbd name /rbd id快速找到对应的 rbd id/rbd name

rbd 的对象名都为 rbd_head.{id}.{序列号}
所以知道id之后就可以找到这个卷对应的所有对象文件

关于rbd其他信息，例如快照等，通过获取rbd_header.{id}的omap信息可以得到

create_timestamp – 创建的时间戳
features – rbd features
object_prefix – rbd prefix(rbd_data.{id})
order – 2**order = rbd block
parent – 父卷id 和 snapid，(父卷 -> snap -> clone(子卷))
size – rbd size
snap_seq – 快照信息

create_timestamp
value (8 bytes) :
00000000  bb 39 68 60 72 aa 49 1b                           |.9h`r.I.|
00000008

features
value (8 bytes) :
00000000  3d 00 00 00 00 00 00 00                           |=.......|
00000008

object_prefix
value (25 bytes) :
00000000  15 00 00 00 72 62 64 5f  64 61 74 61 2e 36 30 34  |....rbd_data.604|
00000010  38 36 62 38 62 34 35 36  37                       |86b8b4567|
00000019

order
value (1 bytes) :
00000000  16                                                |.|
00000001

parent
value (48 bytes) :
00000000  01 01 2a 00 00 00 08 00  00 00 00 00 00 00 0e 00  |..*.............|
00000010  00 00 66 62 64 66 61 63  35 63 65 66 61 31 66 38  |..fbdfac5cefa1f8|
00000020  de 00 00 00 00 00 00 00  00 00 00 00 19 00 00 00  |................|
00000030

size
value (8 bytes) :
00000000  00 00 00 80 02 00 00 00                           |........|
00000008

snap_seq
snapshot_0000000000000173
value (113 bytes) :
00000000  06 01 6b 00 00 00 73 01  00 00 00 00 00 00 04 00  |..k...s.........|
00000010  00 00 74 65 73 74 00 00  00 00 19 00 00 00 3d 00  |..test........=.|
00000020  00 00 00 00 00 00 01 01  2a 00 00 00 08 00 00 00  |........*.......|
00000030  00 00 00 00 0e 00 00 00  66 62 64 66 61 63 35 63  |........fbdfac5c|
00000040  65 66 61 31 66 38 de 00  00 00 00 00 00 00 00 00  |efa1f8..........|
00000050  00 00 19 00 00 00 00 00  00 00 00 00 00 00 00 01  |................|
00000060  01 04 00 00 00 00 00 00  00 c4 7e 69 60 bc 83 2c  |..........~i`..,|
00000070  0b                                                |.|
00000071

找到之后返回给客户端组装，就能得到一个完整的rbd

1.3. 小笔记

顺便记一下 rados listompvals 等omap操作命令的流程

mon使用 crush 算法算出对象所在的osd
然后去osd上查omap，查完给客户端返回结果

1.4. rgw的寻址

接收客户端请求

请求中带有
- 认证信息
- bucket名
- 对象名
如何通过请求信息获取到该rgw对象对应的所有4m块的小文件对象
- 通过bucket可以获取到index对象id
- 通过crush算出index所在osd
- 通过查询omap信息获取所有的对象信息
- 返回rgw
- rgw组装返回

1.5. 数据恢复思路

1.5.1. 场景

机房掉电后，数据损坏，omap损坏，无法启动服务
集群无法读写，也读不到rbd信息

1.5.2. 思路

新建集群，将元数据导入，再将数据导入

新建一个和当前集群规模(osd个数)一样的集群，保证crush之后的结果相同
创建池池的信息可以通过 meta 目录获取osdmap crushmap也可以通过osdmap获取

# crush获取
osdmaptool osdmap.43__0_641716DC__none --export-crush /tmp/crushmap.bin
crushtool -d /tmp/crushmap.bin -o /tmp/crushmap.txt

# omap
osdmaptool --print osdmap.43__0_641716DC__none
#----
epoch 43
fsid 8685ec71-96a6-413a-9e4d-ff47071dc4f5
created 2020-12-22 16:57:37.845173
modified 2021-04-04 15:51:46.729929
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 6
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous

pool 1 '.rgw.root' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 7 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 2 'test001' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 40 flags hashpspool stripe_width 0
	removed_snaps [1~3]
pool 3 'default.rgw.control' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 21 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.meta' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 23 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.log' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 25 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.buckets.index' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 28 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 7 'default.rgw.buckets.data' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 31 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw
pool 8 'default.rgw.buckets.non-ec' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 34 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw

max_osd 1
osd.0 up   in  weight 1 up_from 42 up_thru 42 down_at 41 last_clean_interval [37,40) 172.16.10.10:6801/12622 172.16.10.10:6802/12622 172.16.10.10:6803/12622 172.16.10.10:6804/12622 exists,up 59ebf8d5-e7f7-4c46-8e05-bac5140eee89