Ceph RGW：分段上传和整体上传元数据详解

背景

Ceph的对象存储兼容s3协议，支持对象的整体上传和分段上传功能，详细可参考网上资料。本文主要是基于具体实践，观察整体上传和分段上传的差异。

本文结论

• 关键配置
  • rgw_max_chunk_size：整体上传时使用，为对象文件拆分后第一个rados对象大小。
  • rgw_obj_stripe_size：整体上传和分段上传均使用，为对象文件拆分后每个stripe的大小。
• 整体上传rados对象分布

rados对象	size	角色
{bucketid} + ‘_’ + {objectname}	rgw_max_chunk_size	header对象，存放元数据也存放部分数据
{bucketid} + ‘_’ + ‘_shadow_’ + {objectprefix} + ‘_’ + {id}	rgw_obj_stripe_size	存放数据

• 分段上传rados对象分布

rados对象	size	角色
{bucketid} + ‘_’ + {objectname}	0	header对象，存放元数据
{bucketid} + ‘_’ + ‘multipart’ + {objectprefix} + ‘_’ + {分段id}	rgw_obj_stripe_size	每个分段第一个对象
{bucketid} + ‘_’ + ‘_shadow_’ + {objectprefix} + ‘_’ + {分段id} + {id}	rgw_obj_stripe_size	每个分段其他对象

对象上传实践

环境准备

查询存储池配置rgw_max_chunk_size=2M和rgw_obj_stripe_size=8M

查询配置，其中rgw_max_chunk_size=2M，rgw_obj_stripe_size=8M。

[root@poctest_100 /data/objtest]# ceph daemon /var/run/ceph/*.asok config show | grep "rgw_obj_stripe_size\|rgw_max_chunk_size"
    "rgw_max_chunk_size": "2097152",
    "rgw_obj_stripe_size": "8388608",

创建一个bucket，名字为testbucket0

[root@poctest_100 /data/objtest]# s3cmd mb s3://testbucket0
Bucket 's3://testbucket0/' created

分段上传分析

创建一个100M的文件file_100M.bin，上传至testbucket0中，分段size为30M

[root@poctest_100 /data/objtest]# s3cmd put file_100M.bin s3://testbucket0 --multipart-chunk-size-mb=30
upload: 'file_100M.bin' -> 's3://testbucket0/file_100M.bin'  [part 1 of 4, 30MB] [1 of 1]
 31457280 of 31457280   100% in    0s    43.98 MB/s  done
upload: 'file_100M.bin' -> 's3://testbucket0/file_100M.bin'  [part 2 of 4, 30MB] [1 of 1]
 31457280 of 31457280   100% in    0s    41.81 MB/s  done
upload: 'file_100M.bin' -> 's3://testbucket0/file_100M.bin'  [part 3 of 4, 30MB] [1 of 1]
 31457280 of 31457280   100% in    0s    40.51 MB/s  done
upload: 'file_100M.bin' -> 's3://testbucket0/file_100M.bin'  [part 4 of 4, 10MB] [1 of 1]
 10485760 of 10485760   100% in    0s    30.69 MB/s  done

获取testbucket0的id

[root@poctest_100 /data/objtest]# radosgw-admin bucket stats | grep testbucket0 -A 20
        "bucket": "testbucket0",
        "num_shards": 128,
        "tenant": "",
        "zonegroup": "63a80505-8c43-43e8-b1c3-b7a600fabdfe",
        "placement_rule": "objpool0",
        "explicit_placement": {
   
            "data_pool": "",
            "data_tail_pool": "",
            "data_extra_pool": "",
            "index_pool": ""
        },
        "index_type": "Normal",
        "id": "e8392725-d4f8-4d37-940f-eb948c81c3a3.686977.1",  // testbucket0的id
        "marker": "e8392725-d4f8-4d37-940f-eb948c81c3a3.686977.1",
        "index_type": "Normal",
        "owner": "admin",

查询manifest对象

bucketid = e8392725-d4f8-4d37-940f-eb948c81c3a3.686977.1
manifest对象名：{bucketid} + _ + {objectname} = e8392725-d4f8-4d37-940f-eb948c81c3a3.686977.1_file_100M.bin

[root@poctest_100 /data/objtest]# rados -p 00000000-default.rgw.buckets.data listxattr e8392725-d4f8-4d37-940f-eb948c81c3a3.686977.1_file_100M.bin user.rgw.manifest
user.rgw.acl
user.rgw.content_type
user.rgw.etag
user.rgw.idtag
user.rgw.manifest
user.rgw.pg_ver
user.rgw.source_zone
user.rgw.tail_tag
user.rgw.x-amz-date
user.rgw.x-amz-meta-s3cmd-attrs

[root@poctest_100 /data/objtest]# rados -p 00000000-default.rgw.buckets.data getxattr e8392725-d4f8-4d37-940f-eb948c81c3a3.686977.1_file_100M.bin user.rgw.manifest > manifest.file_100M.bin.txt
[root@poctest_100 /data/objtest]# ceph-dencoder import 'manifest.file_100M.bin.txt' type RGWObjManifest decode dump_json
{
   
    "objs": [],
    "obj_size": 104857600,
    "explicit_objs": "false",
    "head_size": 0,
    "max_head_size":