Ceph多站点部署和配置

已于 2023-11-10 09:15:41 修改
阅读量273 收藏
点赞数
分类专栏： Ceph 文章标签： ceph
于 2023-09-20 09:28:11 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wqxl01/article/details/132195707
版权

Ceph版本：14.2.22

 

前言

Ceph多站点功能是Ceph对象存储的一个功能，它是在两个独立的集群（主集群和备集群）之间实现数据同步功能。在Ceph 14.2.22版本中两个集群都可以接受读写请求，但是second集群无法直接处理写请求，而是将写请求转发到master集群中处理。second集群无法创建bucket，只能操作object。两个集群之间的数据同步包括元数据和数据的同步。

 

集群规划

节点ip角色
node1192.168.3.11master集群
node2192.168.3.12second集群

 

Master集群配置

创建realm并设置为default

radosgw-admin realm create --rgw-realm=movies --default
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "name": "movies",
    "current_period": "dec345b1-dfee-4cec-832e-2d3797ad7012",
    "epoch": 1
}

其中--rgw-realm的值为rgw realm的名字，--default表示将movies设置为多站点中默认的rgw realm，该命令将会创建名为movies的rgw realm，并将movies设置为默认的rgw realm。

创建master zonegroup并设置为default

radosgw-admin zonegroup create --rgw-realm={realm name} --rgw-zonegroup={zonegroup name} --endpoints={url} --master --default

radosgw-admin zonegroup create --rgw-realm=movies --rgw-zonegroup=us --endpoints=http://192.168.3.12:7480 --master --default
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "09143c33-a26f-449a-94c9-11c41ce79ac9",
    "name": "us",
    "api_name": "us",
    "is_master": "true",
    "endpoints": [
        "http://192.168.3.12:7480"
    ],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "",
    "zones": [],
    "placement_targets": [],
    "default_placement": "",
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
}

创建master zone并设置为default

radosgw-admin zone create --rgw-zonegroup={zonegroup name} --rgw-zone={zone name} --endpoints={url} --master --default

radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east --endpoints=http://192.168.3.12:7480 --master --default
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "b711f710-37c1-4532-984b-5073c052be28",
    "name": "us-east",
    "domain_root": "us-east.rgw.meta:root",
    "control_pool": "us-east.rgw.control",
    "gc_pool": "us-east.rgw.log:gc",
    "lc_pool": "us-east.rgw.log:lc",
    "log_pool": "us-east.rgw.log",
    "intent_log_pool": "us-east.rgw.log:intent",
    "usage_log_pool": "us-east.rgw.log:usage",
    "reshard_pool": "us-east.rgw.log:reshard",
    "user_keys_pool": "us-east.rgw.meta:users.keys",
    "user_email_pool": "us-east.rgw.meta:users.email",
    "user_swift_pool": "us-east.rgw.meta:users.swift",
    "user_uid_pool": "us-east.rgw.meta:users.uid",
    "otp_pool": "us-east.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "us-east.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "us-east.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "us-east.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
}

创建system用户

radosgw-admin user create --uid={user name} --display-name={user name} --system

radosgw-admin user create --uid="muti-site" --display-name="muti-site" --system
-----------------------------------------------------------------------------------------------------------------------
{
    "user_id": "muti-site",
    "display_name": "muti-site",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "muti-site",
            "access_key": "RR1SZ9BPSGXBQQ78ZHLF",
            "secret_key": "HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "system": "true",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

access_key和secret_key在后续创建zone以及拉去realm配置信息时需要。

更新master zone的access-key和secret-key

radosgw-admin zone modify --rgw-zone={zone name} --access-key={key} --secret-key={key}

radosgw-admin zone modify --rgw-zone=us-east --access-key=RR1SZ9BPSGXBQQ78ZHLF --secret-key=HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "b711f710-37c1-4532-984b-5073c052be28",
    "name": "us-east",
    "domain_root": "us-east.rgw.meta:root",
    "control_pool": "us-east.rgw.control",
    "gc_pool": "us-east.rgw.log:gc",
    "lc_pool": "us-east.rgw.log:lc",
    "log_pool": "us-east.rgw.log",
    "intent_log_pool": "us-east.rgw.log:intent",
    "usage_log_pool": "us-east.rgw.log:usage",
    "reshard_pool": "us-east.rgw.log:reshard",
    "user_keys_pool": "us-east.rgw.meta:users.keys",
    "user_email_pool": "us-east.rgw.meta:users.email",
    "user_swift_pool": "us-east.rgw.meta:users.swift",
    "user_uid_pool": "us-east.rgw.meta:users.uid",
    "otp_pool": "us-east.rgw.otp",
    "system_key": {
        "access_key": "RR1SZ9BPSGXBQQ78ZHLF",
        "secret_key": "HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "us-east.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "us-east.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "us-east.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
}

更新并提交period

radosgw-admin period update --commit

radosgw-admin period update --commit
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "539162d3-3c1f-41c9-8513-1947434949a8",
    "epoch": 1,
    "predecessor_uuid": "dec345b1-dfee-4cec-832e-2d3797ad7012",
    "sync_status": [],
    "period_map": {
        "id": "539162d3-3c1f-41c9-8513-1947434949a8",
        "zonegroups": [
            {
                "id": "09143c33-a26f-449a-94c9-11c41ce79ac9",
                "name": "us",
                "api_name": "us",
                "is_master": "true",
                "endpoints": [
                    "http://192.168.3.12:7480"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b711f710-37c1-4532-984b-5073c052be28",
                "zones": [
                    {
                        "id": "b711f710-37c1-4532-984b-5073c052be28",
                        "name": "us-east",
                        "endpoints": [
                            "http://192.168.3.12:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": [],
                        "storage_classes": [
                            "STANDARD"
                        ]
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
            }
        ],
        "short_zone_ids": [
            {
                "key": "b711f710-37c1-4532-984b-5073c052be28",
                "val": 2810335027
            }
        ]
    },
    "master_zonegroup": "09143c33-a26f-449a-94c9-11c41ce79ac9",
    "master_zone": "b711f710-37c1-4532-984b-5073c052be28",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "realm_name": "movies",
    "realm_epoch": 2
}

对于zone、zonegroup的任何修改，只有commit period才会生效，commit period过程中会将当前period时期的所有配置修改信息发送给多站点中的其他rgw实例，其他rgw实例收到后会更新本地配置信息。

更新rgw实例配置并与master zone绑定

[client.rgw.node1]
host = node1
rgw frontends = "civetweb port=7480"
rgw_zone=us-east

重启rgw服务

systemctl restart ceph-radosgw@rgw.node1.service

 

Second集群配置

拉取master zonegroup所属的realm配置信息

radosgw-admin realm pull --url={current master url} --access-key={key} --secret-key={key}

radosgw-admin realm pull --url=http://192.168.3.12:7480 --access-key=RR1SZ9BPSGXBQQ78ZHLF --secret-key=HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1
-----------------------------------------------------------------------------------------------------------------------
2023-08-06 15:25:05.504 7fd710eb2900  1 error read_lastest_epoch .rgw.root:periods.539162d3-3c1f-41c9-8513-1947434949a8.latest_epoch
2023-08-06 15:25:06.136 7fd710eb2900  1 Set the period's master zonegroup 09143c33-a26f-449a-94c9-11c41ce79ac9 as the default
{
    "id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "name": "movies",
    "current_period": "539162d3-3c1f-41c9-8513-1947434949a8",
    "epoch": 2
}

注：执行完上述命令之后，有时会遇到如下问题。通过查看日志后，目前发现有2个原因会导致这种问题：master zone的access-key和secret-key为空值，另外一个是时间不同步。其中最坑人就是key值的缺失。在创建system用户时，一定要等master zone创建完成之后再创建，这个顺序不能乱，如果在还没有创建master zone之前就创建system用户，等执行到提交period，之前创建的system用户竟然被删除了。

request failed: (13) Permission denied
If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user.

创建second zone并与master zonegroup绑定

radosgw-admin zone create --rgw-zonegroup={master zonegroup name} --rgw-zone={name} --endpoints={url} --access-key={key} --secret-key={key}

radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west --endpoints=http://192.168.3.13:7480 --access-key=RR1SZ9BPSGXBQQ78ZHLF --secret-key=HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1
-----------------------------------------------------------------------------------------------------------------------
2023-08-06 15:26:09.574 7f5399906900  0 failed reading obj info from .rgw.root:zone_info.b711f710-37c1-4532-984b-5073c052be28: (2) No such file or directory
2023-08-06 15:26:09.574 7f5399906900  0 WARNING: could not read zone params for zone id=b711f710-37c1-4532-984b-5073c052be28 name=us-east
{
    "id": "84592722-4f81-47ab-a0d4-06f946477a57",
    "name": "us-west",
    "domain_root": "us-west.rgw.meta:root",
    "control_pool": "us-west.rgw.control",
    "gc_pool": "us-west.rgw.log:gc",
    "lc_pool": "us-west.rgw.log:lc",
    "log_pool": "us-west.rgw.log",
    "intent_log_pool": "us-west.rgw.log:intent",
    "usage_log_pool": "us-west.rgw.log:usage",
    "reshard_pool": "us-west.rgw.log:reshard",
    "user_keys_pool": "us-west.rgw.meta:users.keys",
    "user_email_pool": "us-west.rgw.meta:users.email",
    "user_swift_pool": "us-west.rgw.meta:users.swift",
    "user_uid_pool": "us-west.rgw.meta:users.uid",
    "otp_pool": "us-west.rgw.otp",
    "system_key": {
        "access_key": "RR1SZ9BPSGXBQQ78ZHLF",
        "secret_key": "HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "us-west.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "us-west.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "us-west.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
}

更新并提交period

radosgw-admin period update --commit

radosgw-admin period update --commit
-----------------------------------------------------------------------------------------------------------------------
2023-08-06 15:26:51.957 7f7efdb77900  1 Cannot find zone id=84592722-4f81-47ab-a0d4-06f946477a57 (name=us-west), switching to local zonegroup configuration
Sending period to new master zone b711f710-37c1-4532-984b-5073c052be28
{
    "id": "539162d3-3c1f-41c9-8513-1947434949a8",
    "epoch": 2,
    "predecessor_uuid": "dec345b1-dfee-4cec-832e-2d3797ad7012",
    "sync_status": [],
    "period_map": {
        "id": "539162d3-3c1f-41c9-8513-1947434949a8",
        "zonegroups": [
            {
                "id": "09143c33-a26f-449a-94c9-11c41ce79ac9",
                "name": "us",
                "api_name": "us",
                "is_master": "true",
                "endpoints": [
                    "http://192.168.3.12:7480"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b711f710-37c1-4532-984b-5073c052be28",
                "zones": [
                    {
                        "id": "84592722-4f81-47ab-a0d4-06f946477a57",
                        "name": "us-west",
                        "endpoints": [
                            "http://192.168.3.13:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    },
                    {
                        "id": "b711f710-37c1-4532-984b-5073c052be28",
                        "name": "us-east",
                        "endpoints": [
                            "http://192.168.3.12:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": [],
                        "storage_classes": [
                            "STANDARD"
                        ]
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
            }
        ],
        "short_zone_ids": [
            {
                "key": "84592722-4f81-47ab-a0d4-06f946477a57",
                "val": 3637646402
            },
            {
                "key": "b711f710-37c1-4532-984b-5073c052be28",
                "val": 2810335027
            }
        ]
    },
    "master_zonegroup": "09143c33-a26f-449a-94c9-11c41ce79ac9",
    "master_zone": "b711f710-37c1-4532-984b-5073c052be28",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "realm_name": "movies",
    "realm_epoch": 2
}

从上面输出信息可以看到us-west和us-east都是归属于us，并且us-east是master zone。

更新rgw实例配置并与second zone绑定

[client.rgw.node2]
host = node2
rgw frontends = "civetweb port=7480"
rgw_zone=us-west

重启rgw服务

systemctl restart ceph-radosgw@rgw.node2.service

查看同步状态

radosgw-admin sync status

radosgw-admin sync status
-----------------------------------------------------------------------------------------------------------------------
          realm 821d976e-be24-48ef-aeb0-a9882085e7fb (movies)
      zonegroup 09143c33-a26f-449a-94c9-11c41ce79ac9 (us)
           zone 84592722-4f81-47ab-a0d4-06f946477a57 (us-west)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: b711f710-37c1-4532-984b-5073c052be28 (us-east)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

 

灾难恢复

当由于某些原因导致master出现故障，导致无法写入数据，此时可以将second zone升级为master zone。当多站点中存在新的master zone后，如果出现故障的master恢复了，此时恢复后的master已经不再担任master角色，而是second角色，同时数据同步功能恢复正常。

将second zone提升为master zone

radosgw-admin zone modify --rgw-zone={zone name} --master --default

radosgw-admin zone modify --rgw-zone=us-west --master --default
-----------------------------------------------------------------------------------------------------------------------
2023-08-06 15:33:02.052 7faad265a900  0 NOTICE: overriding master zone: b711f710-37c1-4532-984b-5073c052be28
2023-08-06 15:33:02.056 7faad265a900  0 failed reading obj info from .rgw.root:zone_info.b711f710-37c1-4532-984b-5073c052be28: (2) No such file or directory
2023-08-06 15:33:02.056 7faad265a900  0 WARNING: could not read zone params for zone id=b711f710-37c1-4532-984b-5073c052be28 name=us-east
{
    "id": "84592722-4f81-47ab-a0d4-06f946477a57",
    "name": "us-west",
    "domain_root": "us-west.rgw.meta:root",
    "control_pool": "us-west.rgw.control",
    "gc_pool": "us-west.rgw.log:gc",
    "lc_pool": "us-west.rgw.log:lc",
    "log_pool": "us-west.rgw.log",
    "intent_log_pool": "us-west.rgw.log:intent",
    "usage_log_pool": "us-west.rgw.log:usage",
    "reshard_pool": "us-west.rgw.log:reshard",
    "user_keys_pool": "us-west.rgw.meta:users.keys",
    "user_email_pool": "us-west.rgw.meta:users.email",
    "user_swift_pool": "us-west.rgw.meta:users.swift",
    "user_uid_pool": "us-west.rgw.meta:users.uid",
    "otp_pool": "us-west.rgw.otp",
    "system_key": {
        "access_key": "RR1SZ9BPSGXBQQ78ZHLF",
        "secret_key": "HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "us-west.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "us-west.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "us-west.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
}

更新并提交period

radosgw-admin period update --commit

radosgw-admin period update --commit
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "baf807ae-b904-4721-982d-eea5a751ff27",
    "epoch": 1,
    "predecessor_uuid": "539162d3-3c1f-41c9-8513-1947434949a8",
    "sync_status": [],
    "period_map": {
        "id": "baf807ae-b904-4721-982d-eea5a751ff27",
        "zonegroups": [
            {
                "id": "09143c33-a26f-449a-94c9-11c41ce79ac9",
                "name": "us",
                "api_name": "us",
                "is_master": "true",
                "endpoints": [
                    "http://192.168.3.12:7480"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "84592722-4f81-47ab-a0d4-06f946477a57",
                "zones": [
                    {
                        "id": "84592722-4f81-47ab-a0d4-06f946477a57",
                        "name": "us-west",
                        "endpoints": [
                            "http://192.168.3.13:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    },
                    {
                        "id": "b711f710-37c1-4532-984b-5073c052be28",
                        "name": "us-east",
                        "endpoints": [
                            "http://192.168.3.12:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": [],
                        "storage_classes": [
                            "STANDARD"
                        ]
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
            }
        ],
        "short_zone_ids": [
            {
                "key": "84592722-4f81-47ab-a0d4-06f946477a57",
                "val": 3637646402
            },
            {
                "key": "b711f710-37c1-4532-984b-5073c052be28",
                "val": 2810335027
            }
        ]
    },
    "master_zonegroup": "09143c33-a26f-449a-94c9-11c41ce79ac9",
    "master_zone": "84592722-4f81-47ab-a0d4-06f946477a57",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "realm_name": "movies",
    "realm_epoch": 3
}

重启rgw服务

systemctl restart ceph-radosgw@rgw.node2.service

如果想让出现故障的master zone在恢复正常后依然执行master角色，如果多站点中已经存在master，此时只能通过手动修改zone的配置。如果多站点中不存在master，此时恢复正常的master zone自动成为master角色，以下是假设多站点中已经存在master的相关处理。

拉取master zonegroup所属的realm配置信息

radosgw-admin realm pull --url={current master url} --access-key={key} --secret-key={key}

radosgw-admin realm pull --url=http://192.168.3.13:7480 --access-key=RR1SZ9BPSGXBQQ78ZHLF --secret-key=HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "name": "movies",
    "current_period": "baf807ae-b904-4721-982d-eea5a751ff27",
    "epoch": 3
}

恢复旧的master zone

radosgw-admin zone modify --rgw-zone={zone name} --master --default

radosgw-admin zone modify --rgw-zone=us-east --master --default
-----------------------------------------------------------------------------------------------------------------------
2023-08-06 15:44:05.186 7fb6ecea7900  0 NOTICE: overriding master zone: 84592722-4f81-47ab-a0d4-06f946477a57
2023-08-06 15:44:05.186 7fb6ecea7900  0 failed reading obj info from .rgw.root:zone_info.84592722-4f81-47ab-a0d4-06f946477a57: (2) No such file or directory
2023-08-06 15:44:05.186 7fb6ecea7900  0 WARNING: could not read zone params for zone id=84592722-4f81-47ab-a0d4-06f946477a57 name=us-west
{
    "id": "b711f710-37c1-4532-984b-5073c052be28",
    "name": "us-east",
    "domain_root": "us-east.rgw.meta:root",
    "control_pool": "us-east.rgw.control",
    "gc_pool": "us-east.rgw.log:gc",
    "lc_pool": "us-east.rgw.log:lc",
    "log_pool": "us-east.rgw.log",
    "intent_log_pool": "us-east.rgw.log:intent",
    "usage_log_pool": "us-east.rgw.log:usage",
    "reshard_pool": "us-east.rgw.log:reshard",
    "user_keys_pool": "us-east.rgw.meta:users.keys",
    "user_email_pool": "us-east.rgw.meta:users.email",
    "user_swift_pool": "us-east.rgw.meta:users.swift",
    "user_uid_pool": "us-east.rgw.meta:users.uid",
    "otp_pool": "us-east.rgw.otp",
    "system_key": {
        "access_key": "RR1SZ9BPSGXBQQ78ZHLF",
        "secret_key": "HbuDM9yJPrJvpg9mKuARHNpPhadcNLafZ3WqbUo1"
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "us-east.rgw.buckets.index",
                "storage_classes": {
                    "STANDARD": {
                        "data_pool": "us-east.rgw.buckets.data"
                    }
                },
                "data_extra_pool": "us-east.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
}

更新并提交period

radosgw-admin period update --commit

radosgw-admin period update --commit
-----------------------------------------------------------------------------------------------------------------------
{
    "id": "6157442a-fe91-44a7-9404-44ba291ba105",
    "epoch": 1,
    "predecessor_uuid": "baf807ae-b904-4721-982d-eea5a751ff27",
    "sync_status": [],
    "period_map": {
        "id": "6157442a-fe91-44a7-9404-44ba291ba105",
        "zonegroups": [
            {
                "id": "09143c33-a26f-449a-94c9-11c41ce79ac9",
                "name": "us",
                "api_name": "us",
                "is_master": "true",
                "endpoints": [
                    "http://192.168.3.12:7480"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "b711f710-37c1-4532-984b-5073c052be28",
                "zones": [
                    {
                        "id": "84592722-4f81-47ab-a0d4-06f946477a57",
                        "name": "us-west",
                        "endpoints": [
                            "http://192.168.3.13:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    },
                    {
                        "id": "b711f710-37c1-4532-984b-5073c052be28",
                        "name": "us-east",
                        "endpoints": [
                            "http://192.168.3.12:7480"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": ""
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": [],
                        "storage_classes": [
                            "STANDARD"
                        ]
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb"
            }
        ],
        "short_zone_ids": [
            {
                "key": "84592722-4f81-47ab-a0d4-06f946477a57",
                "val": 3637646402
            },
            {
                "key": "b711f710-37c1-4532-984b-5073c052be28",
                "val": 2810335027
            }
        ]
    },
    "master_zonegroup": "09143c33-a26f-449a-94c9-11c41ce79ac9",
    "master_zone": "b711f710-37c1-4532-984b-5073c052be28",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "821d976e-be24-48ef-aeb0-a9882085e7fb",
    "realm_name": "movies",
    "realm_epoch": 4
}

重启rgw服务

systemctl restart ceph-radosgw@rgw.node1.service
哼了个哈
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
Ceph部署和基础使用
06-25
简单介绍Ceph的组件，搭建Ceph分布式存储，基础使用，包含基础故障排查
对象存储多站点
QTM_Gitee的博客
05-30 1178
单个zone配置通常由一个zonegroup组成，该zonegroup包含一个zone和一个或多个ceph radosgw实例，可以在这些实例之间对网关客户端请求进行负载平衡。在单zone配置中，通常多个网关实例指向单个Ceph存储群集。但是，Red Hat支持Ceph对象网关的多种多站点配置选项： Multi-zone: 一种更高级的配置包括一个zone group和多个zone，每个zone有一个或多个ceph-radosgw实例。每个zone都有自己的Ceph存储集群作为后盾。zone group中
Ceph 多节点quick部署
u010685850的专栏
12-22 1392
ceph 多节节点的快速部署，源自ceph官方文档和自己的实践
ceph 对象网关多区部署
xin053
05-29 2008
概念 realm realm代表了全局唯一的命名空间，这个命名空间由一个或多个zonegroup组成, 必须指定一个 zonegroup 为 master zonegroup zonegroup zonegroup包含一个或多个zone，必须指定一个zone 为master zone。rgw 多活方式是在同一zonegroup的多个zone之间进行的，即同一zonegroup中多个zone之间的数据完全一致，用户可以通过任意zone读写用一份数据。但是，对元数据的操作，比如创建桶，创建用户，仍然只能在mas
对象存储单节点ceph环境部署，附移除osd并格式化osd所在磁盘脚本
02-19
3.运行 ceph_deploy.sh 脚本，部署【单节点】对象存储环境 4.检查系统状态 ceph -s 5.创建system用户 radosgw-admin user create 6.S3cmd 安装和配置 其他 ceph_rm_osd.sh用于移除osd并格式化osd所在磁盘
ceph系统配置和课程总结
04-12
ceph系统配置和课程总结
ceph安装部署使用文档
12-24
ceph安装部署文档。。
使用脚本和docker部署ceph集群
09-13
使用脚本和docker部署ceph集群. 不过得去每个机器上去指定执行一次(前提是先有docker, 并且有对应的docker- ceph镜像), 可以部署mon,osd,mgr,mds.这个是L版本的ceph, 支持多mds active. 也可以用ansible批量执行
Ceph集群扩容及数据再均衡原理分析
李姓门徒
05-14 544
用户文件在Ceph RADOS中存储、定位过程大概包括：用户文件切割成对象、对象映射到PG、PG分组PGP、PG映射到OSD。这些过程中，可能涉及了大量概念和变量，而其实它们大部分是通过HASH、CRUSH等算法计算出来的，初始参数可能也就只有这么几个：用户文件inode、layout、crush_map等。Ceph所有组件和客户端通过计算就能得到所有对象的Location，而且只要输入的参数信息不变，Location就不会变，另外通过PG作为中间层将对象与OSD解绑，使得对象在Ceph RADOS分布更均
最新版Ceph( Reef版本) nfs导出
weixin_42562106的博客
05-14 239
nfs挂载方式。
分布式存储系统ceph
文思源想的技术空间
05-17 413
Ceph是一个自由软件存储平台，为统一的分布式对象、块和文件存储提供了一个高性能的解决方案。Ceph的目标是通过将存储数据分布在多个硬盘、计算机或数据中心上来提供高性能、可扩展和冗余的存储。它通过CRUSH算法来确定数据应该存储在哪个位置，从而实现负载均衡和容错。Ceph云计算平台：一些大型云计算平台使用 Ceph 作为存储解决方案。大数据分析：在大数据处理场景中，Ceph 被用于存储和管理数据。企业存储系统：不少企业采用 Ceph 构建自己的存储系统。以下是一些可能使用到CephOpenStack。
android手机应用源码Imsdroid语音视频通话源码.rar
05-20
android手机应用源码Imsdroid语音视频通话源码.rar
营销计划汇报PPT，市场品牌 推广渠道 产品 营销策略tbb.pptx
05-20
营销计划汇报PPT，市场品牌 推广渠道 产品 营销策略tbb.pptx
JavaScript_超过100种语言的纯Javascript OCR.zip
05-20
JavaScript
JavaScript_跨平台React UI包.zip
05-20
JavaScript
node-v16.17.0-headers.tar.xz
最新发布
05-20
Node.js，简称Node，是一个开源且跨平台的JavaScript运行时环境，它允许在浏览器外运行JavaScript代码。Node.js于2009年由Ryan Dahl创立，旨在创建高性能的Web服务器和网络应用程序。它基于Google Chrome的V8 JavaScript引擎，可以在Windows、Linux、Unix、Mac OS X等操作系统上运行。 Node.js的特点之一是事件驱动和非阻塞I/O模型，这使得它非常适合处理大量并发连接，从而在构建实时应用程序如在线游戏、聊天应用以及实时通讯服务时表现卓越。此外，Node.js使用了模块化的架构，通过npm（Node package manager，Node包管理器）,社区成员可以共享和复用代码，极大地促进了Node.js生态系统的发展和扩张。 Node.js不仅用于服务器端开发。随着技术的发展，它也被用于构建工具链、开发桌面应用程序、物联网设备等。Node.js能够处理文件系统、操作数据库、处理网络请求等，因此，开发者可以用JavaScript编写全栈应用程序，这一点大大提高了开发效率和便捷性。 在实践中，许多大型企业和组织已经采用Node.js作为其Web应用程序的开发平台，如Netflix、PayPal和Walmart等。它们利用Node.js提高了应用性能，简化了开发流程，并且能更快地响应市场需求。
520表白代码.rar
05-20
520表白html5爱心代码
一个简单的HTML5和CSS代码示例，用于创建一个动态的爱心形状，并在网页上展示一个类似520表白的消息 这个示例使用了CSS的
05-20
520表白html5爱心代码 一个简单的HTML5和CSS代码示例，用于创建一个动态的爱心形状，并在网页上展示一个类似520表白的消息。这个示例使用了CSS的动画效果和HTML的结构。
ceph分布式存储部署
09-15
ceph分布式存储的部署需要按照以下步骤进行： 1. 创建存储池：在ceph集群部署好之后，首先需要创建存储池，才能向ceph写入数据。通过一致性hash计算，文件将被保存在对应的PG中，而这个文件将属于某个存储池的一个PG。然后，数据对象将被写入主OSD并同步到从OSD上，以实现数据的高可用性。 2. 使用ceph-deploy部署ceph存储：可以使用ceph-deploy工具来简化ceph集群的部署工作。通过运行命令"ceph-deploy mgr create cephnode1 cephnode2 cephnode3"，可以管理集群并在指定的节点上部署rgw。 3. RADOS存储集群：对象数据的底层存储服务由多个存储主机组成的存储集群，也被称为RADOS（可靠的、自动化的、分布式的对象存储系统）。这个存储集群提供了可靠和高性能的存储服务，用于管理ceph分布式存储系统中的对象数据。 总结起来，ceph分布式存储的部署包括创建存储池、使用ceph-deploy工具部署存储和管理集群，以及使用RADOS存储集群来提供可靠和高性能的存储服务。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值