RGW的实现基于Rados的pool、对象、xattrs和omap。RGW将数据分为3种:metadata,
bucket index和数据。
- metadata
包括user,bucket和bucket.instance。其中user是用户信息。bucket是bucket名字和
bucket instance的映射。bucket.instance中是每个bucket instance的信息。
- bucket index
bucket index在rados对象的omap中保存了k/v映射。其中key是用户put的object的名字,
value保存了该对象的一些基本元数据信息。每个omap中都有一个header,其中存放了bucket
的统计信息(比如,存放的对象个数,占用的空间大小等)。//???
- data
用户put的对象的数据内容存放在一个或者多个rados对象中。
1. rgw bucket怎么表示?
rgw bucket 和 Amazon S3 一致。和Swift的 container 对应。1.1 bucket 的元数据信息
----------------------- [root@yhg-2 centos]# radosgw-admin metadata get bucket:xiamenbucket --cluster yhgtest
{
"key": "bucket:xiamenbucket",
"ver": {
"tag": "_lGZYdGzrVGYu2PYmKUcqs6S",
"ver": 1
},
"mtime": 1457681931,
"data": {
"bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"owner": "osgw-william1",
"creation_time": 1457681931,
"linked": "true",
"has_bucket_info": "false"
}
}
.. Note:: bucket_id/marker的组成:
zone_name.$instance_id.$bucket_id'。
详细内容见rgw/rgw_rados.cc:644
1.2 bucket的表示
--------------------------- bucket index 对象
作用:
在该对象的omap中保存了属于本bucket的所有key的列表。
名称:
.dir.yhgtest-yhgtest.4463.1 (.dir.$bucket_id)
内容:
有两个key,kehekey和xiamenkey。1.1.1/2.2.3 属于hehekey,3.3.2/4.4.3属于xiamenkey。
每次put新数据都会创建两个新的�0_0000000000AA.AA.2/�0_0000000000BB.BB.3,其中
BB=AA+1。
[root@yhg-2 cmds]# rados -p .rgw.buckets.index listomapkeys .dir.yhgtest-yhgtest.4463.1 --cluster yhgtest
hehekey
xiamenkey
�0_00000000001.1.2
�0_00000000002.2.3
�0_00000000003.3.2
�0_00000000004.4.3
�0_00000000005.5.2
�0_00000000006.6.3
�0_00000000007.7.2
�0_00000000008.8.3
[root@yhg-2 cmds]# rados -p .rgw.buckets.index listomapvals .dir.yhgtest-yhgtest.4463.1 --cluster yhgtest
hehekey
value: (221 bytes) :
0000 : 08 03 d7 00 00 00 07 00 00 00 68 65 68 65 6b 65 : ..........heheke
0010 : 79 03 00 00 00 00 00 00 00 01 04 03 83 00 00 00 : y...............
0020 : 01 00 00 50 00 00 00 00 00 ef ce e7 56 00 00 00 : ...P........V...
0030 : 00 20 00 00 00 62 61 37 38 64 32 36 37 62 62 33 : . ...ba78d267bb3
0040 : 34 39 63 62 34 66 65 38 32 32 61 38 66 36 62 36 : 49cb4fe822a8f6b6
0050 : 64 61 64 39 37 0d 00 00 00 6f 73 67 77 2d 77 69 : dad97....osgw-wi
0060 : 6c 6c 69 61 6d 31 15 00 00 00 5a 6f 6e 65 20 75 : lliam1....Zone u
0070 : 73 65 72 20 66 6f 72 20 79 68 67 74 65 73 74 18 : ser for yhgtest.
0080 : 00 00 00 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f : ...application/o
0090 : 63 74 65 74 2d 73 74 72 65 61 6d 00 00 50 00 00 : ctet-stream..P..
00a0 : 00 00 00 00 00 00 00 00 00 00 00 01 01 02 00 00 : ................
00b0 : 00 01 03 08 17 00 00 00 79 68 67 74 65 73 74 2d : ........yhgtest-
00c0 : 79 68 67 74 65 73 74 2e 34 36 33 33 2e 32 35 00 : yhgtest.4633.25.
00d0 : 00 00 00 00 00 00 00 00 00 00 00 00 00 : .............
...
[root@yhg-2 cmds]# radosgw-admin bucket stats xiamenbucket --cluster yhgtest
...
"xiamenbucket",
{
"bucket": "xiamenbucket",
"pool": ".hdd_pool",
"index_pool": ".rgw.buckets.index",
"id": "yhgtest-yhgtest.4463.1",
"marker": "yhgtest-yhgtest.4463.1",
"owner": "osgw-william1",
"ver": "0#9",
"master_ver": "0#0",
"mtime": "2016-03-11 07:38:51.000000",
"max_marker": "0#00000000008.8.3", // 最新的更新
"usage": {
"rgw.main": {
"size_kb": 10240,
"size_kb_actual": 10240,
"num_objects": 2
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
},
另外,还可以直接和对象的osd守护进程交互,获取.dir_XXX对象上的omap的信息。
# ceph daemon --cluster yhg /var/run/ceph/yhg-osd.1.asok getomap $POOL $OBJ
所在pool:
bucket.index_pool (.rgw.buckets.index)
- bucket meta 对象
作用:
其内容为bucket的描述信息,在其xattr中还有user.rgw.acl信息。
名称:
.bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 (.bucket.meta.$bucket_name:$bucket_id)
内容:
[root@yhg-2 cmds]# rados -p .rgw get .bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 /tmp/m --cluster yhgtest
[root@yhg-2 cmds]# ceph-dencoder type RGWBucketInfo import /tmp/m decode dump_json
{
"bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"creation_time": 1457681931,
"owner": "osgw-william1",
"flags": 0,
"region": "yhgtest",
"placement_rule": "default-placement",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"num_shards": 0,
"bi_shard_hash_type": 0
}
[root@yhg-2 cmds]# rados -p .rgw listxattr .bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 --cluster yhgtest
ceph.objclass.version
user.rgw.acl
[root@yhg-2 cmds]# rados -p .rgw getxattr .bucket.meta.xiamenbucket:yhgtest-yhgtest.4463.1 user.rgw.acl --cluster yhgtest > /tmp/m
[root@yhg-2 cmds]# ceph-dencoder type RGWAccessControlPolicy import /tmp/m decode dump_json
{
"acl": {
"acl_user_map": [
{
"user": "osgw-william1",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "osgw-william1",
"grant": {
"type": {
"type": 0
},
"id": "osgw-william1",
"email": "",
"permission": {
"flags": 15
},
"name": "Zone user for yhgtest",
"group": 0
}
}
]
},
"owner": {
"id": "osgw-william1",
"display_name": "Zone user for yhgtest"
}
}
所在pool:
zone.domain_root (.rgw)
实现函数:
// rgw/rgw_rados.cc
7463 int ret = put_bucket_instance_info(info, exclusive, mtime, pattrs)
- bucket 对象
作用:
???
名称:
xiamenbucket (用户指定的bucket名字)
内容:
和bucket meta对象内容类似。
[root@yhg-2 cmds]# rados -p .rgw listxattr xiamenbucket --cluster yhgtest
ceph.objclass.version
所在pool:
zone.domain_root (.rgw)
2. rgw object怎么表示?
对应至少一个 Rados 对象。每个 rgw object 分为两部分,一个对象逻辑头(object logical head/olh)和一个
可选 tail。olh 最大 512KB,tail 按照固定大小切片。切片的大小可以配置,默认4MB。
olh 的对象内容为上传文件的前512KB。如果文件小于该值,则没有tail。
数据会分为header和4M的其他片段。
- header
[root@yhg-2 cmds]# rados -p .hdd_pool listxattr yhgtest-yhgtest.4463.1_hehekey --cluster yhgtest
user.rgw.acl
user.rgw.content_type
user.rgw.etag
user.rgw.idtag
user.rgw.manifest
.. Note:: s3对象的 用户自定义的metadata也存放在这里
查看header对象所在的pg:
[root@yhg-2 cmds]# ceph osd map .hdd_pool yhgtest-yhgtest.4463.1_hehekey --cluster yhgtest
osdmap e72 pool '.hdd_pool' (1) object 'yhgtest-yhgtest.4463.1_hehekey' -> pg 1.beb53f29 (1.29) -> up ([0], p0) acting ([0], p0)
- manifest
rgw object的 manifest 指示了tail部分的分片信息,用于读取对象时查找分片。
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.manifest --cluster yhgtest > /tmp/m
[root@yhg-2 cmds]# ceph-dencoder type RGWObjManifest import /tmp/m decode dump_json
{
"objs": [],
"obj_size": 5242880,
"explicit_objs": "false",
"head_obj": {
"bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"key": "",
"ns": "",
"object": "hehekey",
"instance": ""
},
"head_size": 524288,
"max_head_size": 524288,
"prefix": ".eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4_",
"tail_bucket": {
"name": "xiamenbucket",
"pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index",
"marker": "yhgtest-yhgtest.4463.1",
"bucket_id": "yhgtest-yhgtest.4463.1"
},
"rules": [
{
"key": 0,
"val": {
"start_part_num": 0,
"start_ofs": 524288,
"part_size": 0,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
]
}
manifest的prefix字段指示了rgw对象的当前版本的分片的名称信息。
[root@yhg-2 cmds]# rados -p .hdd_pool ls --cluster yhgtest | grep eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4
yhgtest-yhgtest.4463.1__shadow_.eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4_2
yhgtest-yhgtest.4463.1__shadow_.eaw8unIlkJ3Lk4eV2XBMO5hB4Du4UY4_1
- idtag
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.idtag --cluster yhgtest
yhgtest-yhgtest.4633.25
- etag
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.etag --cluster yhgtest
ba78d267bb349cb4fe822a8f6b6dad97
- content type
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.content_type --cluster yhgtest
application/octet-stream
- acl
[root@yhg-2 cmds]# rados -p .hdd_pool getxattr yhgtest-yhgtest.4463.1_hehekey user.rgw.acl --cluster yhgtest > /tmp/m
[root@yhg-2 cmds]# ceph-dencoder type RGWAccessControlPolicy import /tmp/m decode dump_json
{
"acl": {
"acl_user_map": [
{
"user": "osgw-william1",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "osgw-william1",
"grant": {
"type": {
"type": 0
},
"id": "osgw-william1",
"email": "",
"permission": {
"flags": 15
},
"name": "Zone user for yhgtest",
"group": 0
}
}
]
},
"owner": {
"id": "osgw-william1",
"display_name": "Zone user for yhgtest"
}
}
3. 怎样列举某个用户的buckets
在zone.user_uid_pool (pool '.users.uid') 中,为每个用户创建了一个$user_name.buckets对象(比如,osgw-william1.buckets)。在该对象的omap中存放了该用户的buckets。在更新
bucket内容,更新获取bucket的统计信息时会访问这些信息。
内容:
[root@yhg-2 cmds]# rados -p .users.uid listomapkeys osgw-william1.buckets --cluster yhgtest
xiamenbucket
xiamenbucket2
xiamenbucket3
xiamenbucket4
::
[root@yhg-2 cmds]# rados -p .users.uid listomapvals osgw-william1.buckets --cluster yhgtest
xiamenbucket
value: (170 bytes) :
0000 : 06 05 a4 00 00 00 00 00 00 00 00 00 a0 00 00 00 : ................
0010 : 00 00 0b 76 e2 56 02 00 00 00 00 00 00 00 07 03 : ...v.V..........
0020 : 7d 00 00 00 0c 00 00 00 78 69 61 6d 65 6e 62 75 : }.......xiamenbu
0030 : 63 6b 65 74 09 00 00 00 2e 68 64 64 5f 70 6f 6f : cket.....hdd_poo
0040 : 6c 16 00 00 00 79 68 67 74 65 73 74 2d 79 68 67 : l....yhgtest-yhg
0050 : 74 65 73 74 2e 34 34 36 33 2e 31 16 00 00 00 79 : test.4463.1....y
0060 : 68 67 74 65 73 74 2d 79 68 67 74 65 73 74 2e 34 : hgtest-yhgtest.4
0070 : 34 36 33 2e 31 12 00 00 00 2e 72 67 77 2e 62 75 : 463.1.....rgw.bu
0080 : 63 6b 65 74 73 2e 69 6e 64 65 78 12 00 00 00 2e : ckets.index.....
0090 : 72 67 77 2e 62 75 63 6b 65 74 73 2e 65 78 74 72 : rgw.buckets.extr
00a0 : 61 00 00 a0 00 00 00 00 00 01 : a.........
xiamenbucket2
value: (171 bytes) :
...
---------------------------
其他
---------------------------[1] zone map
[root@yhg-2 cmds]# radosgw-admin zone get --cluster yhgtest
{
"domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"system_key": {
"access_key": "66D5C0ZBSXYX6HGHH2D8",
"secret_key": "BgDVnjKmeM5bMAXQXGPg1CyxzIQKzmlCCpKrfm4Q"
},
"placement_pools": [
{
"key": ".hdd_pool",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra"
}
},
{
"key": "default-placement",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".hdd_pool",
"data_extra_pool": ".rgw.buckets.extra"
}
}
]
}