rgw object read and write

2019独角兽企业重金招聘Python工程师标准>>>

一、Get Object。

1、读取Object的主要处理流程。

RGWGetObj::execute()

|__创建RGWGetObj_CB类实例，其中handle_data()函数为回调函数，该函数会调用RGWGetObj::get_data_cb()函数，而该函数最终会调用send_reponse_data()

|__创建RGWRados::Object类实例

|__创建RGWRados::Object::Read类实例

|__get_params() 从HTTP请求字符串中读取range_str/if_mod/if_unmod/if_match/if_nomatch信息

|__init_common() 解析range_str/if_mod/if_unmod内容

|__RGWRados::Object::Read::prepare()

    |__RGWRados::get_state()

|__RGWRados::get_obj_state()

    |__RGWRados::get_obj_state_impl()

|__RGWObjectCtx::get_state()

|__RGWRados::raw_obj_stat()

|__检查attrset中是否包含RGW_ATTR_SHADOW_OBJ属性并将属性值写入到RGWObjState::shadow_obj中

|__检查attrset中是否包含RGW_ATTR_MANIFEST属性并将属性值写入到RGWObjState::manifest中，同时更新has_manifest=true以及size值

|__检查attrset中是否包含RGW_ATTR_PG_VER属性并将属性值写入到RGWObjState::pg_ver中

|__检查attrset中是否包含RGW_ATTR_SOURCE_ZONE属性并将属性值写入到RGWObjState::zone_short_id中

|__检查attrset中是否包含RGW_ATTR_OLH_ID_TAG属性并将属性值写入到RGWObjState::olh_tag中

    |__RGWRados::get_obj_ioctx()

|__get_obj_bucket_and_oid_loc()

|__open_bucket_data_ctx()

    |__检查属性值中关于mod_ptr/unmod_ptr/if_match/if_unmatch是否有效

|__RGWGetObj::handle_user_manifest()

    |__解析prefix字符串且以”/”为分隔符，解析出bucket_name和obj_prefix

    |__得到rgw_bucket/RGWBucketInfo/RGWAccessControlPolicy实例

    |__iterate_user_manifest_parts() 找到object的total_len以及object已经完成的字节数

    |__iterate_user_manifest_parts()

|__get_obj_user_manifest_iterate_cb()

    |__RGWGetOp::read_user_manifest_part()

|__创建RGWRados::Object类实例

|__创建RGWRados::Object::Read类实例

|__RGWRados::Object::Read::prepare()

|__rgw_policy_from_attrsets() 从attrs属性列表中找到RGW_ATTR_ACL属性列表值

|__verify_object_permission() 校验bucket ACL和object ACL的合法性

|__RGWRados::Object::Read::read()

|__send_response_data()

|__RGWGetObj::handle_slo_manifest()

    |__iterate_slo_parts()

|__get_obj_user_manifest_iterate_cb()

    |__RGWGetObj::read_user_manifest_part()

|__创建RGWRados::Object类实例

|__创建RGWRados::Object::Read类实例

|__RGWRados::Object::Read::prepare()

|__rgw_policy_from_attrsets() 从attrs属性列表中找到RGW_ATTR_ACL属性列表值

|__verify_object_permission() 校验bucket ACL和object ACL的合法性

|__RGWRados::Object::Read::read()

|__send_response_data()

|__RGWRados::Object::Read::iterate()

    |__构建struct get_obj_data实例

    |__RGWRados::iterate_obj()

|__RGWRados::get_obj_state()

|___get_obj_iterate_cb()

    |__RGWRados::get_obj_iterate_cb()

|__创建ObjectReadOperation类实例

|__ObjectReadOperation::read()

|__librados::IoCtx::aio_operate() 执行读取对象内容操作

|__RGWRados::flush_read_list()

    |__RGWGetObj_CB::handle_data()

     |__RGWGetObj::get_data_obj()

    |__RGWGetObj_ObjStore_S3::send_response_data()

|__send_response_data()

二、Put Object。

1、上传Object的主要处理流程(非分片)。

RGWPutObj::execute()

|__RGWPutObj_ObjStore::get_params() 从HTTP请求中读取HTTP_CONTENT_MD5值

|__get_system_versioning_params() 从HTTP请求中读取versioned-epoch和version-id值

|__ceph_unarmor() 计算supplied_md5_bin数组的值

|__RGWRados::check_quota() 获取user和bucket的quota信息

|__RGWPutObj::select_processor() 得到RGWPutObjProcessor_Atomic类实例

|__RGWPutObjProcessor_Atomic::prepare()

    |__RGWPutObjProcessor_Atomic::prepare_init()

|__RGWRados::get_max_chunk_size() 得到max_chunk_size值

    |__RGWObjManifest::set_trivial_rule()

    |__RGWObjManifest::generator::create_begin()

|__RGWObjManifest::get_rule()

|__RGWObjManifest::get_head_size()

|__RGWObjManifest::get_implicit_location()

对于第一次上传且所上传的数据小于max_head_size，则不分片；

对于第一次上传但所上传的数据大于max_head_size，则oid要加上cur_stripe信息且rgw_obj的namespace是”shadow”；

对于非第一次上传但上传整个分片数据，则oid要加上.cur_part_id信息且rgw_obj的namespace是”multipart”；

对于非第一次上传且上传非整个分片数据，则oid要加上.cur_part_id_cur_stripe信息且rgw_obj的namespace是”shadow”；

|__RGWObjManifest::update_iterators()

|__RGWPutObj_ObjStore_S3::get_data()

    |__RGWPutObj_ObjStore::get_data()

|__STREAM_IO()->read()

|__put_data_and_throttle()

    |__RGWPutObjProcessor_Atomic::handle_data()

|__RGWPutObjProcessor_Atomic::prepare_next_part()

|__RGWPutObjProcessor_Atomic::write_data()

    |__RGWPutObjProcessor_Aio::handle_obj_data()

|__RGWRados::aio_put_obj_data()

    |__RGWPutObjProcessor_Atomic::throttle_data()

|__RGWPutObjProcessor_Aio::throttle_data()

|__RGW_Auth_S3::authorize_aws4_auth_complete()

|__RGWRados::check_quota()

|__RGWPutObjProcessor_Atomic::complete_hash()

|__设置RGW_ATTR_ACL/RGW_ATTR_ETAG属性

|__rgw_get_requst_metadata() 从HTTP请求中读取RGW_ATTR_PREFIX属性

|__encode_delete_at_attr() 设置RGW_ATTR_DELETE_AT属性

|__RGWPutObjProcessor::complete()

    |__RGWPutObjProcessor_Atomic::do_complete()

|__RGWPutObjProcessor_Atomic::complete_writing_data()

    |__RGWPutObjProcessor_Atomic::write_data()

    |__RGWPutObjProcessor_Aio::throttle_data()

    |__RGWPutObjProcessor_Atomic::complete_parts()

    |__RGWPutObjProcessor_Aio::drain_pending()

|__RGWRados::Object::Write::write_meta() 写object的metadata，最后更新quota使用量cache

|__dispose_processor()

2、上传Object的主要处理流程(分片)。

RGWPutObj::execute()

|__RGWPutObj_ObjStore::get_params() 从HTTP请求中读取HTTP_CONTENT_MD5值

|__get_system_versioning_params() 从HTTP请求中读取versioned-epoch和version-id值

|__ceph_unarmor() 计算supplied_md5_bin数组的值

|__RGWRados::check_quota() 获取user和bucket的quota信息

|__RGWPutObj::select_processor() 得到RGWPutObjProcessor_Atomic类实例

|__RGWPutObjProcessor_Multipart::prepare()

    |__RGWPutObjProcessor_Atomic::prepare_init()

|__RGWRados::get_max_chunk_size() 得到max_chunk_size值

    |__RGWMPObj::init() 初始化multipart object

    |__RGWObjManifest::set_multipart_part_rule()

    |__RGWObjManifest::generator::create_begin()

|__RGWObjManifest::get_rule()

|__RGWObjManifest::get_head_size()

|__RGWObjManifest::get_implicit_location()

对于第一次上传且所上传的数据小于max_head_size，则不分片；

对于第一次上传但所上传的数据大于max_head_size，则oid要加上cur_stripe信息且rgw_obj的namespace是”shadow”；

对于非第一次上传但上传整个分片数据，则oid要加上.cur_part_id信息且rgw_obj的namespace是”multipart”；

对于非第一次上传且上传非整个分片数据，则oid要加上.cur_part_id_cur_stripe信息且rgw_obj的namespace是”shadow”；

|__RGWPutObj_ObjStore_S3::get_data()

    |__RGWPutObj_ObjStore::get_data()

|__STREAM_IO()->read()

|__put_data_and_throttle()

    |__RGWPutObjProcessor_Atomic::handle_data()

|__RGWPutObjProcessor_Atomic::prepare_next_part()

|__RGWPutObjProcessor_Atomic::write_data()

    |__RGWPutObjProcessor_Aio::handle_obj_data()

|__RGWRados::aio_put_obj_data()

    |__RGWPutObjProcessor_Atomic::throttle_data()

|__RGWPutObjProcessor_Aio::throttle_data()

|__RGW_Auth_S3::authorize_aws4_auth_complete()

|__RGWRados::check_quota()

|__RGWPutObjProcessor_Atomic::complete_hash()

|__设置RGW_ATTR_ACL/RGW_ATTR_ETAG属性

|__rgw_get_requst_metadata() 从HTTP请求中读取RGW_ATTR_PREFIX属性

|__encode_delete_at_attr() 设置RGW_ATTR_DELETE_AT属性

|__RGWPutObjProcessor::complete()

    |__RGWPutObjProcessor_Multipart::do_complete()

|__RGWPutObjProcessor_Multipart::complete_writing_data()

    |__RGWPutObjProcessor_Multipart::write_data()

    |__RGWPutObjProcessor_Aio::throttle_data()

    |__RGWPutObjProcessor_Multipart::complete_parts()

    |__RGWPutObjProcessor_Aio::drain_pending()

|__RGWRados::Object::Write::write_meta() 写object的metadata，最后更新quota使用量cache

|__创建RGWUploadPartInfo对象且设置初始化该对象

|__初始化multipart的rgw_obj对象且设置namespace以及设置in_extra_data=true

|__RGWRados::omap_set() 设置对象的omap信息且omap的key＝part.part_num

|__dispose_processor()

转载于:https://my.oschina.net/linuxhunter/blog/657499