目录
RBD在Ceph底层的存储方式,解释了RBD的实际占用容量和RBD大小的关系,用几个文件的例子演示了文件在RBD(更恰当的是xfs)中的存储位置,最后组装了一个RBD,给出了一些FAQ。
RBD是什么
-
RBD就是Ceph里的块设备,一个4T的块设备的功能和一个4T的SATA类似,挂载的RBD就可以当磁盘用。
-
resizable
:这个块可大可小。 -
data striped
:这个块在ceph里面是被切割成若干小块来保存,不然1PB的块怎么存的下。 -
thin-provisioned
:精简置备,1TB的集群是能创建无数1PB的块的。说白了就是,块的大小和在ceph中实际占用的大小是没有关系的,甚至,刚创建出来的块是不占空间的,今后用多大空间,才会在ceph中占用多大空间。打个比方就是,你有一个32G的U盘,存了一个2G的电影,那么RBD大小就类似于32G,而2G就相当于在ceph中占用的空间。
RBD,和下面说的块
,是一回事。
创建一个1G的块xzcpool
[root@localhost xzc]# ceph -s cluster f739ca83-a2e6-4a17-9a9f-d02a5563c701 health HEALTH_OK monmap e1: 1 mons at {localhost=[::1]:6789/0} election epoch 2, quorum 0 localhost osdmap e5: 1 osds: 1 up, 1 in pgmap v9: 64 pgs, 1 pools, 0 bytes data, 0 objects 2209 MB used, 15184 MB / 17394 MB avail 64 active+clean [root@localhost xzc]# ceph osd pool create xzcpool 10 pool 'xzcpool' created [root@localhost xzc]# rbd create xzcpool/xzcimg --size 1024 [root@localhost xzc]# rbd info xzcpool/xzcimg rbd image 'xzcimg': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rb.0.1014.74b0dc51 format: 1 [root@localhost xzc]# ceph -s cluster f739ca83-a2e6-4a17-9a9f-d02a5563c701 health HEALTH_OK monmap e1: 1 mons at {localhost=[::1]:6789/0} election epoch 2, quorum 0 localhost osdmap e9: 1 osds: 1 up, 1 in pgmap v23: 74 pgs, 2 pools, 134 bytes data, 2 objects 2213 MB used, 15180 MB / 17394 MB avail 74 active+clean [root@localhost xzc]# rbd -p xzcpool ls xzcimg [root@localhost xzc]# rados -p xzcpool ls rbd_directory xzcimg.rbd
[root@localhost xzc]# rbd info xzcpool/xzcimg rbd image 'xzcimg': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rb.0.1014.74b0dc51 format: 1
-
size
: 就是这个块的大小,即1024MB=1G,1024MB/256 = 4M
,共分成了256个对象(object),每个对象4M,这个会在下面详细介绍。 -
order 22
, 22是个编号,4M是22, 8M是23,也就是2^22 bytes = 4MB, 2^23 bytes = 8MB。 -
block_name_prefix
: 这个是块的最重要的属性了,这是每个块在ceph中的唯一前缀编号,有了这个前缀,把服务器上的OSD都拔下来带回家,就能复活所有的VM了。 -
format
: 格式有两种,1和2,下文会讲。
观察建foo
前后的ceph -s
输出,会发现多了两个文件,查看下:
[root@localhost xzc]# ceph -s cluster f739ca83-a2e6-4a17-9a9f-d02a5563c701 health HEALTH_OK monmap e1: 1 mons at {localhost=[::1]:6789/0} election epoch 2, quorum 0 localhost osdmap e9: 1 osds: 1 up, 1 in pgmap v23: 74 pgs, 2 pools, 134 bytes data, 2 objects 2213 MB used, 15180 MB / 17394 MB avail 74 active+clean [root@localhost xzc]# rbd -p xzcpool ls xzcimg [root@localhost xzc]# rados -p xzcpool ls rbd_directory xzcimg.rbd
再查看这两个文件里面的内容:
[root@localhost xzc]# mkdir ./test [root@localhost xzc]# cd ./test/ [root@localhost test]# rados -p xzcpool xzcimg.rbd xzcimg.rpd unrecognized command xzcimg.rbd; -h or --help for usage [root@localhost test]# rados -p xzcpool get xzcimg.rbd xzcimg.rpd [root@localhost test]# rados -p xzcpool get rbd_directory rbd_directory [root@localhost test]# hexdump -C hexdump xzcimg.rpd hexdump: hexdump: No such file or directory 00000000 3c 3c 3c 20 52 61 64 6f 73 20 42 6c 6f 63 6b 20 |<<< Rados Block | 00000010 44 65 76 69 63 65 20 49 6d 61 67 65 20 3e 3e 3e |Device Image >>>| 00000020 0a 00 00 00 00 00 00 00 72 62 2e 30 2e 31 30 31 |........rb.0.101| 00000030 34 2e 37 34 62 30 64 63 35 31 00 00 00 00 00 00 |4.74b0dc51......| 00000040 52 42 44 00 30 30 31 2e 30 30 35 00 16 00 00 00 |RBD.001.005.....| 00000050 00 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 |...@............| 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000070 [root@localhost test]# hexdump -C hexdump rbd_directory hexdump: hexdump: No such file or directory 00000000 00 00 00 00 01 00 00 00 06 00 00 00 78 7a 63 69 |............xzci| 00000010 6d 67 00 00 00 00 |mg....| 00000016
这时候我们再创建一个RBD叫bar
,再次对比查看:
[root@localhost test]# rbd create xzcpool/bar --size 1024 [root@localhost test]# rados -p xzcpool ls rbd_directory bar.rbd xzcimg.rbd [root@localhost test]# rados -p xzcpool get rbd_directory rbd_directory [root@localhost test]# hexdump -C hexdump rbd_directory hexdump: hexdump: No such file or directory 00000000 00 00 00 00 02 00 00 00 03 00 00 00 62 61 72 00 |............bar.| 00000010 00 00 00 06 00 00 00 78 7a 63 69 6d 67 00 00 00 |.......xzcimg...| 00000020 00 |.| 00000021
多出了个bar.rbd
文件,很容易联想到这个文件的内容是和xzcimg.rbd
内容类似的,唯一不同的是保存了各自的block_name_prefix
。然后,rbd_directory
里面多出了bar
这个块名字。
rbd_directory
可以得出以下的推论:
每个块(RBD)刚创建(format 1)时都会生成一个rbdName.rbd
这样的文件(ceph里的对象),里面保存了这个块的prefix
。 同时,rbd_directory
会增加刚刚的创建的rbdName
,顾名思义这个文件就是这个pool里面的所有RBD的索引。
为了验证结论,删掉刚刚的bar
只留下foo
:
[root@localhost test]# rbd rm xzcpool/bar Removing image: 100% complete...done. [root@localhost test]# rados -p xzcpool get rbd_directory rbd_directory [root@localhost test]# hexdump -C hexdump rbd_directory hexdump: hexdump: No such file or directory 00000000 00 00 00 00 01 00 00 00 06 00 00 00 78 7a 63 69 |............xzci| 00000010 6d 67 00 00 00 00 |mg....| 00000016
RBD使用
建好了块,我们就开始使用这个块了:
先打个快照吧
[root@localhost test]# rbd snap create xzcpool/xzcimg@001
[root@localhost test]# rbd map xzcpool/xzcimg /dev/rbd0 [root@localhost test]# mkfs.xfs /dev/rbd0 Discarding blocks...Done. meta-data=/dev/rbd0 isize=512 agcount=8, agsize=32768 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=262144, imaxpct=25 = sunit=1024 swidth=1024 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@localhost test]# mkdir ./xzcimg [root@localhost test]# ll -h total 8.0K -rw-r--r--. 1 root root 22 Oct 12 11:40 rbd_directory drwxr-xr-x. 2 root root 6 Oct 12 11:49 xzcimg -rw-r--r--. 1 root root 112 Oct 12 11:35 xzcimg.rpd [root@localhost test]# mount /dev/rbd0 ./xzcimg [root@localhost test]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 898M 0 898M 0% /dev tmpfs