Ceph中查找BUCKET INDEX所在位置的方法

##1.引言 本文说明了如何查看每个BUCKET对应的索引信息，以及所在的OSD位置，为以后运维做参考。

##2.实验过程

###2.1 罗列出当前系统下所有的bucket信息

[root@node1 ~]# radosgw-admin bucket list
[
    "ntc-oss-large-file-bda",
    "ntc-oss-large-file-html",
    "ntc-oss-large-file",
    "ntc-oss-large-file-eml"
]

上述命令展示了当前Ceph 对象存储系统中涉及到的所有的BUCKET。

###2.2 查看具体某个BUCKET属性

[root@node1 ~]# radosgw-admin bucket stats --bucket=ntc-oss-large-file
{
    "bucket": "ntc-oss-large-file",
    "zonegroup": "c0d87b6c-defb-4101-a5f2-278074b3880f",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "d65ec313-61d5-46b9-9199-2c8fcc401bdc.24144.4",
    "marker": "d65ec313-61d5-46b9-9199-2c8fcc401bdc.24144.4",
    "index_type": "Normal",
    "owner": "admin",
    "ver": "0#27,1#25,2#27,3#19,4#25,5#23,6#29,7#21,8#21,9#15,10#21,11#19,12#23,13#31,14#15,15#17,16#13,17#23,18#21,19#27,20#23,21#25,22#25,23#17,24#25,25#11,26#23,27#21,28#27,29#17,30#17,31#15,32#31,33#27,34#23,35#21,36#19,37#19,38#19,39#27,40#19,41#2025,42#17,43#27,44#21,45#21,46#31,47#21,48#33,49#27,50#29,51#19,52#21,53#17,54#19,55#19,56#17,57#15,58#19,59#31,60#19,61#27,62#19,63#25,64#15,65#21,66#17,67#31,68#19,69#19,70#23,71#13,72#19,73#19,74#25,75#29,76#19,77#17,78#15,79#21,80#15,81#27,82#11,83#19,84#13,85#23,86#243,87#27,88#23,89#17,90#29,91#23,92#21,93#21,94#23,95#19,96#23,97#11,98#19,99#17",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0",
    "mtime": "2021-08-06 17:45:34.007557",
    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#",
    "usage": {
        "rgw.main": {
            "size": 10630744290,
            "size_actual": 10630848512,
            "size_utilized": 10630744290,
            "size_kb": 10381587,
            "size_kb_actual": 10381688,
            "size_kb_utilized": 10381587,
            "num_objects": 1016
        },
        "rgw.multimeta": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 0
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

该命令展示了BUCKET的名称，所在的data pool, index pool. BUCKET ID.查看不到

[root@node1 ~]# radosgw-admin zone get
{
    "id": "d65ec313-61d5-46b9-9199-2c8fcc401bdc",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "tier_config": [],
    "realm_id": ""
}

该命令展示了BUCKET的名称，所在的data pool, index pool. BUCKET ID.
2.3 检查对应BUCKET在index中是否存在

[root@node1 ~]# rados -p default.rgw.buckets.index ls - | grep "d65ec313-61d5-46b9-9199-2c8fcc401bdc.24144.4"
.dir.d65ec313-61d5-46b9-9199-2c8fcc401bdc.24144.4.43

注：此处需要在BUCKET ID前面加上.dir. 才是它在INDEX POOL中的索引

###2.4 查看对应INDEX中记录的key

[root@k19-bigdata-5 ~]# rados -p default.rgw.buckets.data listomapkeys .dir.d65ec313-61d5-46b9-9199-2c8fcc401bdc
02/2.txt
113.txt
125.txt
203.txt
234.txt
321.txt
584.txt
60.txt
981.txt

统计文件数量

rados -p default.rgw.buckets.data listomapkeys .dir.d65ec313-61d5-46b9-9199-2c8fcc401bdc | wc -l
988847
###2.5 查看对应索引信息存放的物理位置

# ceph osd map default.rgw.buckets.data   .dir.d65ec313-61d5-46b9-9199-2c8fcc401bdc

osdmap e5176 pool ‘default.rgw.buckets.data’ (6) object ‘.dir.d65ec313-61d5-46b9-9199-2c8fcc401bdc’ -> pg 6.578ebf78 (6.178) -> up ([1,2], p1) acting ([1,2], p1)
通过命令可以看到，BUCKET default.rgw.buckets.data的index信息落在OSD 1,2上面，其中1为主osd.

单个pg迁移
设置集群仅支持 Luminous（或者L之后的）客户端
具体命令：

ceph osd set-require-min-compat-client luminous

ceph osd pg-upmap-items 6.b0  0 2

从上述数据结果中
ceph osd pg-upmap-items 6.178 2 0
表明pg 1.1需从osd.0重新映射到osd.2