ceph命令总结
一、集群
1、启动一个ceph 进程
2、查看机器的监控状态[查看集群健康状态细节]
ceph health [detail]
3、查看ceph的实时运行状态(常用)
ceph -w
4、检查信息状态信息(常用)
ceph -s
5、查看ceph存储空间
ceph df
10、查看ceph集群中的认证用户及相关的key(常用)
ceph auth list
11、查看ceph log日志所在的目录
ceph-conf --name mon.node1 --show-config-value log_file
二、mon
1、查看mon的状态信息
#ceph mon stat
输出信息：
e2: 3 mons at {10.186.9.13=10.186.9.13:6789/0,10.186.9.14=10.186.9.14:6789/0,10.186.9.16=10.186.9.16:6789/0}, election epoch 46, leader 0 10.186.9.13, quorum 0,1,2 10.186.9.13,10.186.9.14,10.186.9.16
2、查看mon的选举状态
# ceph quorum_status
输出信息：
{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["10.186.9.13","10.186.9.14","10.186.9.16"],"quorum_leader_name":"10.186.9.13","monmap":{"epoch":2,"fsid":"e6e8accb-6f7d-460e-93fe-06d49b1fde83","modified":"2019-11-19 16:36:44.137334","created":"2019-11-19 16:36:30.935660","features":{"persistent":["kraken","luminous"],"optional":[]},"mons":[{"rank":0,"name":"10.186.9.13","addr":"10.186.9.13:6789/0","public_addr":"10.186.9.13:6789/0"},{"rank":1,"name":"10.186.9.14","addr":"10.186.9.14:6789/0","public_addr":"10.186.9.14:6789/0"},{"rank":2,"name":"10.186.9.16","addr":"10.186.9.16:6789/0","public_addr":"10.186.9.16:6789/0"}]}}
3、查看mon的映射信息
# ceph mon dump
输出信息：
dumped monmap epoch 1
epoch 1
fsid c71f378b-e86b-4c49-8b54-a9d85a6122c2
last_changed 2020-08-03 16:50:50.647195
created 2020-08-03 16:50:50.647195
0: 10.5.29.54:6789/0 mon.10.5.29.54
4、删除一个mon节点
# ceph mon remove node1
5、获得一个正在运行的mon map，为二进制mon.bin
# ceph mon getmap -o mon.bin
三、msd
1、查看msd状态
# ceph mds stat
输出结果：
cephfs-1/1/1 up {0=openstack1=up:active}
2、查看msd的映射信息
# ceph mds dump
输出结果：
dumped fsmap epoch 45552
fs_name cephfs
epoch 45552
flags c
created 2020-09-08 18:03:49.514279
modified 2020-09-25 09:25:56.099961
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 210
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
max_mds 1
in 0
up {0=2084098}
failed
damaged
stopped
data_pools [7]
metadata_pool 8
inline_data disabled
balancer
standby_count_wanted 0
2084098: 10.5.29.54:6800/1252110594 'openstack1' mds.0.45540 up:active seq 23
3、删除一个mds节点
# ceph mds rm 0 mds.node1
四、osd
1、查看ceph osd运行状态
# ceph osd stat
输出信息：
1 osds: 1 up, 1 in
2、查看osd映射信息
# ceph osd dump
输出信息：
epoch 218
fsid c71f378b-e86b-4c49-8b54-a9d85a6122c2
created 2020-08-03 16:50:50.839507
modified 2020-09-30 13:18:29.712934
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 7
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
pool 1 'images' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 118 lfor 0/116 flags hashpspool stripe_width 0 application rbd
removed_snaps [1~3,8~2]
pool 2 'volumes' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 218 lfor 0/119 flags hashpspool stripe_width 0 application rbd
removed_snaps [1~3]
pool 3 'backups' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 200 lfor 0/125 flags hashpspool stripe_width 0 application rbd
pool 4 'vms' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 16 pgp_num 16 last_change 124 lfor 0/122 flags hashpspool stripe_width 0 application rbd
removed_snaps [1~3]
pool 5 'k8s' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 60 flags hashpspool stripe_width 0
pool 6 '.rgw.root' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 132 flags hashpspool stripe_width 0 application rgw
pool 7 'cephfs_data' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 173 flags hashpspool stripe_width 0 application cephfs
pool 8 'cephfs_metadata' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 last_change 174 flags hashpspool stripe_width 0 application cephfs
pool 9 'default.rgw.control' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 151 flags hashpspool stripe_width 0 application rgw
pool 10 'default.rgw.meta' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 156 flags hashpspool stripe_width 0 application rgw
pool 11 'default.rgw.log' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 160 flags hashpspool stripe_width 0 application rgw
max_osd 1
osd.0 up in weight 1 up_from 212 up_thru 212 down_at 211 last_clean_interval [207,209) 10.5.29.54:6802/5668 10.5.29.54:6803/5668 10.5.29.54:6804/5668 10.5.29.54:6805/5668 exists,up 9fd52adc-c323-42a0-b130-a779bbdaf0b4
3、查看osd的目录树
# ceph osd tree
输出信息：
输出信息：
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.64000 root default
-2 0 host 10.5.29.54
-5 0.64000 host openstack1
0 hdd 0.64000 osd.0 up 1.00000 1.00000
4、查看osd各硬盘使用率
# ceph osd df
输出信息：
输出信息：
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 hdd 0.64000 1.00000 600GiB 11.0GiB 589GiB 1.83 1.00 120
TOTAL 600GiB 11.0GiB 589GiB 1.83
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
5、查看osd延时
# ceph osd perf
输出信息：
osd commit_latency(ms) apply_latency(ms)
0 0 0
五、PG组
1、查看pg组的映射信息
# ceph pg dump
输出信息：
dumped all
version 607481
stamp 2020-10-09 11:16:32.865738
last_osdmap_epoch 0
last_pg_scan 0
full_ratio 0
nearfull_ratio 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN
11.3 22 0 0 0 0 0 1554 1554 active+clean 2020-10-08 23:40:08.999691 218'65354 218:98261 [0] 0 [0] 0 218'63422 2020-10-08 23:40:08.999638 218'58648 2020-10-07 19:21:09.086796 0
10.2 0 0 0 0 0 0 0 0 active+clean 2020-10-08 17:03:50.360386 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 17:03:50.360073 0'0 2020-10-03 16:33:37.680982 0
9.1 1 0 0 0 0 0 15 15 active+clean 2020-10-08 08:10:13.634676 213'15 218:261552 [0] 0 [0] 0 213'15 2020-10-08 08:10:13.634540 213'15 2020-10-07 07:55:43.794747 0
3.b 0 0 0 0 0 0 0 0 active+clean 2020-10-09 08:05:07.603764 0'0 218:441 [0] 0 [0] 0 0'0 2020-10-09 08:05:07.603570 0'0 2020-10-08 04:10:55.985889 0
2.a 3 0 0 0 0 117 696 696 active+clean 2020-10-08 01:01:15.979950 218'696 218:124863 [0] 0 [0] 0 218'695 2020-10-08 01:01:15.979563 218'695 2020-10-05 09:23:23.610962 0
1.9 86 0 0 0 0 684376727 188 188 active+clean 2020-10-08 08:27:35.916078 71'188 218:836 [0] 0 [0] 0 71'188 2020-10-08 08:27:35.915814 71'188 2020-10-08 08:27:35.915814 0
4.c 7 0 0 0 0 8402530 1563 1563 active+clean 2020-10-09 05:48:38.892392 218'394363 218:672698 [0] 0 [0] 0 217'394362 2020-10-09 05:48:38.892240 217'394362 2020-10-08 04:31:51.649845 0
8.0 4 0 0 0 0 0 8 8 active+clean 2020-10-09 05:11:56.978009 162'8 218:267 [0] 0 [0] 0 162'8 2020-10-09 05:11:56.977695 162'8 2020-10-05 12:27:57.351136 0
11.2 26 0 0 0 0 0 1594 1594 active+clean 2020-10-08 12:58:06.268311 218'90994 218:136755 [0] 0 [0] 0 218'85762 2020-10-08 12:58:06.268206 218'52208 2020-10-02 13:31:54.916159 0
10.3 0 0 0 0 0 0 0 0 active+clean 2020-10-09 00:13:54.479007 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-09 00:13:54.478738 0'0 2020-10-02 12:03:50.578100 0
9.0 1 0 0 0 0 0 15 15 active+clean 2020-10-09 05:20:58.550242 213'15 218:268427 [0] 0 [0] 0 213'15 2020-10-09 05:20:58.550128 213'15 2020-10-07 21:28:01.376248 0
3.a 0 0 0 0 0 0 0 0 active+clean 2020-10-08 18:27:30.668569 0'0 218:439 [0] 0 [0] 0 0'0 2020-10-08 18:27:30.668417 0'0 2020-10-07 16:04:00.976150 0
2.b 1 0 0 0 0 98 106 106 active+clean 2020-10-08 10:05:35.206002 217'106 218:703 [0] 0 [0] 0 217'106 2020-10-08 10:05:35.205802 217'106 2020-10-06 03:38:00.975439 0
1.8 86 0 0 0 0 717931008 287 287 active+clean 2020-10-08 07:06:10.779285 70'287 218:1569 [0] 0 [0] 0 70'287 2020-10-08 07:06:10.778936 70'287 2020-10-08 07:06:10.778936 0
4.d 5 0 0 0 0 16388608 1546 1546 active+clean 2020-10-09 08:30:59.011121 218'624246 218:727723 [0] 0 [0] 0 217'624240 2020-10-09 08:30:59.010908 217'624240 2020-10-06 23:57:54.087248 0
8.1 0 0 0 0 0 0 0 0 active+clean 2020-10-09 02:16:40.314110 0'0 218:249 [0] 0 [0] 0 0'0 2020-10-09 02:16:40.313909 0'0 2020-10-05 03:37:09.108887 0
3.5 0 0 0 0 0 0 0 0 active+clean 2020-10-09 10:25:57.831498 0'0 218:426 [0] 0 [0] 0 0'0 2020-10-09 10:25:57.830926 0'0 2020-10-02 20:46:38.411226 0
1.7 81 0 0 0 0 679477248 418 418 active+clean 2020-10-08 14:05:10.402235 66'418 218:1093 [0] 0 [0] 0 66'418 2020-10-08 14:05:10.402109 66'418 2020-10-07 06:01:39.461254 0
2.4 1 0 0 0 0 18 498 498 active+clean 2020-10-08 20:12:13.357134 217'498 218:1247560 [0] 0 [0] 0 217'498 2020-10-08 20:12:13.356924 217'498 2020-10-05 00:45:28.059351 0
4.2 3 0 0 0 0 176180 1543 1543 active+clean 2020-10-08 16:42:20.137802 217'298843 218:399234 [0] 0 [0] 0 217'298843 2020-10-08 16:42:20.137431 217'298843 2020-10-06 06:30:49.343340 0
5.3 0 0 0 0 0 0 0 0 active+clean 2020-10-09 02:53:51.921079 0'0 218:329 [0] 0 [0] 0 0'0 2020-10-09 02:53:51.920938 0'0 2020-10-02 20:04:26.034336 0
6.0 1 0 0 0 0 688 1 1 active+clean 2020-10-08 07:23:36.310287 142'1 218:277 [0] 0 [0] 0 142'1 2020-10-08 07:23:36.310122 142'1 2020-10-08 07:23:36.310122 0
7.1 0 0 0 0 0 0 0 0 active+clean 2020-10-08 17:49:11.537675 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 17:49:11.537376 0'0 2020-10-06 03:19:35.732187 0
3.4 0 0 0 0 0 0 0 0 active+clean 2020-10-09 04:31:15.247127 0'0 218:435 [0] 0 [0] 0 0'0 2020-10-09 04:31:15.246848 0'0 2020-10-06 16:48:31.939261 0
1.6 82 0 0 0 0 679477283 310 310 active+clean 2020-10-08 20:19:21.537754 70'310 218:1686 [0] 0 [0] 0 70'310 2020-10-08 20:19:21.537612 70'310 2020-10-08 20:19:21.537612 0
2.5 1 0 0 0 0 4055040 90 90 active+clean 2020-10-09 04:23:31.750333 218'90 218:70691 [0] 0 [0] 0 217'60 2020-10-09 04:23:31.750105 217'60 2020-10-05 08:26:56.419549 0
4.3 3 0 0 0 0 188433 1531 1531 active+clean 2020-10-08 12:08:22.190091 217'212431 218:920082 [0] 0 [0] 0 217'212431 2020-10-08 12:08:22.189777 217'212431 2020-10-08 12:08:22.189777 0
5.2 0 0 0 0 0 0 0 0 active+clean 2020-10-09 09:28:43.292211 0'0 218:336 [0] 0 [0] 0 0'0 2020-10-09 09:28:43.292031 0'0 2020-10-05 19:37:54.878218 0
6.1 0 0 0 0 0 0 0 0 active+clean 2020-10-08 13:36:13.932128 0'0 218:260 [0] 0 [0] 0 0'0 2020-10-08 13:36:13.931794 0'0 2020-10-08 13:36:13.931794 0
7.0 0 0 0 0 0 0 0 0 active+clean 2020-10-09 09:29:09.307764 0'0 218:251 [0] 0 [0] 0 0'0 2020-10-09 09:29:09.307454 0'0 2020-10-09 09:29:09.307454 0
3.7 0 0 0 0 0 0 0 0 active+clean 2020-10-08 14:08:17.318780 0'0 218:426 [0] 0 [0] 0 0'0 2020-10-08 14:08:17.318563 0'0 2020-10-04 18:27:13.385545 0
2.6 2 0 0 0 0 8376320 393 393 active+clean 2020-10-09 01:22:42.643154 218'393 218:1367 [0] 0 [0] 0 217'337 2020-10-09 01:22:42.642792 217'337 2020-10-03 15:53:46.935028 0
1.5 86 0 0 0 0 707526946 167 167 active+clean 2020-10-09 03:44:52.360481 63'167 218:63972 [0] 0 [0] 0 63'167 2020-10-09 03:44:52.360344 63'167 2020-10-08 02:00:07.024073 0
4.0 6 0 0 0 0 12559360 1562 1562 active+clean 2020-10-08 19:01:16.152927 101'318862 218:828536 [0] 0 [0] 0 101'318862 2020-10-08 19:01:16.152716 101'318862 2020-10-07 08:36:13.213465 0
5.1 0 0 0 0 0 0 0 0 active+clean 2020-10-09 00:29:55.218759 0'0 218:335 [0] 0 [0] 0 0'0 2020-10-09 00:29:55.218326 0'0 2020-10-05 17:39:25.346109 0
6.2 1 0 0 0 0 333 1 1 active+clean 2020-10-08 14:37:19.490217 142'1 218:274 [0] 0 [0] 0 142'1 2020-10-08 14:37:19.490131 142'1 2020-10-08 14:37:19.490131 0
7.3 0 0 0 0 0 0 0 0 active+clean 2020-10-08 20:58:04.469121 0'0 218:245 [0] 0 [0] 0 0'0 2020-10-08 20:58:04.468788 0'0 2020-10-07 13:04:26.017451 0
3.6 0 0 0 0 0 0 0 0 active+clean 2020-10-09 04:07:18.496808 0'0 218:443 [0] 0 [0] 0 0'0 2020-10-09 04:07:18.496599 0'0 2020-10-02 17:40:58.371033 0
2.7 3 0 0 0 0 4194339 378 378 active+clean 2020-10-08 03:23:28.591046 217'378 218:655744 [0] 0 [0] 0 217'378 2020-10-08 03:23:28.590692 217'378 2020-10-03 01:41:55.349009 0
1.4 90 0 0 0 0 723589246 1587 1587 active+clean 2020-10-08 09:44:38.932913 71'2487 218:50536 [0] 0 [0] 0 71'2487 2020-10-08 09:44:38.932699 71'2487 2020-10-08 09:44:38.932699 0
4.1 5 0 0 0 0 4382772 1542 1542 active+clean 2020-10-08 22:57:48.978911 217'352542 218:808146 [0] 0 [0] 0 217'352542 2020-10-08 22:57:48.978519 217'352542 2020-10-02 10:26:44.907168 0
5.0 0 0 0 0 0 0 0 0 active+clean 2020-10-09 02:14:50.464730 0'0 218:332 [0] 0 [0] 0 0'0 2020-10-09 02:14:50.464550 0'0 2020-10-09 02:14:50.464550 0
6.3 0 0 0 0 0 0 0 0 active+clean 2020-10-08 11:52:18.454136 0'0 218:257 [0] 0 [0] 0 0'0 2020-10-08 11:52:18.453892 0'0 2020-10-08 11:52:18.453892 0
7.2 0 0 0 0 0 0 0 0 active+clean 2020-10-09 10:26:00.722055 0'0 218:253 [0] 0 [0] 0 0'0 2020-10-09 10:26:00.721822 0'0 2020-10-03 21:27:39.219344 0
3.1 0 0 0 0 0 0 0 0 active+clean 2020-10-08 10:11:17.315356 0'0 218:434 [0] 0 [0] 0 0'0 2020-10-08 10:11:17.315070 0'0 2020-10-02 06:34:46.486690 0
1.3 73 0 0 0 0 603979792 1186 1186 active+clean 2020-10-08 15:11:27.101874 70'1186 218:528365 [0] 0 [0] 0 70'1186 2020-10-08 15:11:27.101729 70'1186 2020-10-03 06:36:46.159828 0
2.0 2 0 0 0 0 4194304 161 161 active+clean 2020-10-08 10:50:51.658886 218'161 218:3646 [0] 0 [0] 0 217'149 2020-10-08 10:50:51.658690 217'149 2020-10-02 05:54:00.944450 0
4.6 2 0 0 0 0 4055552 1593 1593 active+clean 2020-10-08 18:25:57.203958 217'407603 218:519423 [0] 0 [0] 0 217'407603 2020-10-08 18:25:57.203816 217'407603 2020-10-02 14:27:00.120976 0
5.7 0 0 0 0 0 0 0 0 active+clean 2020-10-09 02:07:56.120705 0'0 218:336 [0] 0 [0] 0 0'0 2020-10-09 02:07:56.120542 0'0 2020-10-09 02:07:56.120542 0
6.4 0 0 0 0 0 0 0 0 active+clean 2020-10-08 15:30:06.812974 0'0 218:256 [0] 0 [0] 0 0'0 2020-10-08 15:30:06.812920 0'0 2020-10-07 05:18:20.563881 0
7.5 0 0 0 0 0 0 0 0 active+clean 2020-10-08 22:01:37.445961 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 22:01:37.445720 0'0 2020-10-07 10:34:22.205740 0
3.0 0 0 0 0 0 0 0 0 active+clean 2020-10-08 19:33:48.443499 0'0 218:436 [0] 0 [0] 0 0'0 2020-10-08 19:33:48.443313 0'0 2020-10-08 19:33:48.443313 0
1.2 76 0 0 0 0 629145635 155 155 active+clean 2020-10-08 01:56:32.707423 70'155 218:731 [0] 0 [0] 0 70'155 2020-10-08 01:56:32.707293 70'155 2020-10-02 10:30:10.099163 0
2.1 2 0 0 0 0 4194402 218 218 active+clean 2020-10-09 04:11:49.268318 217'218 218:1281 [0] 0 [0] 0 217'218 2020-10-09 04:11:49.268031 217'218 2020-10-08 02:17:00.274874 0
4.7 6 0 0 0 0 4386868 1519 1519 active+clean 2020-10-08 07:59:17.454869 218'154320 218:739719 [0] 0 [0] 0 217'154316 2020-10-08 07:59:17.454735 217'154316 2020-10-05 14:12:12.252121 0
5.6 0 0 0 0 0 0 0 0 active+clean 2020-10-09 02:09:26.702351 0'0 218:325 [0] 0 [0] 0 0'0 2020-10-09 02:09:26.702215 0'0 2020-10-09 02:09:26.702215 0
6.5 0 0 0 0 0 0 0 0 active+clean 2020-10-08 04:00:02.687366 0'0 218:258 [0] 0 [0] 0 0'0 2020-10-08 04:00:02.687219 0'0 2020-10-03 06:26:57.008192 0
7.4 0 0 0 0 0 0 0 0 active+clean 2020-10-08 02:59:38.635329 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 02:59:38.635023 0'0 2020-10-05 09:48:19.960110 0
3.3 0 0 0 0 0 0 0 0 active+clean 2020-10-08 10:02:27.292244 0'0 218:436 [0] 0 [0] 0 0'0 2020-10-08 10:02:27.291922 0'0 2020-10-08 10:02:27.291922 0
1.1 77 0 0 0 0 645922816 188 188 active+clean 2020-10-09 01:29:46.645099 71'188 218:812 [0] 0 [0] 0 71'188 2020-10-09 01:29:46.644840 71'188 2020-10-09 01:29:46.644840 0
2.2 2 0 0 0 0 8348160 732 732 active+clean 2020-10-08 08:16:14.045553 218'732 218:185914 [0] 0 [0] 0 217'695 2020-10-08 08:16:14.045314 217'695 2020-10-07 02:53:34.753245 0
4.4 4 0 0 0 0 4018211 1556 1556 active+clean 2020-10-08 06:18:49.134949 217'394356 218:532058 [0] 0 [0] 0 217'394356 2020-10-08 06:18:49.134707 217'394356 2020-10-01 20:05:17.730157 0
5.5 0 0 0 0 0 0 0 0 active+clean 2020-10-09 07:02:06.450455 0'0 218:329 [0] 0 [0] 0 0'0 2020-10-09 07:02:06.450175 0'0 2020-10-02 19:14:45.702111 0
6.6 0 0 0 0 0 0 0 0 active+clean 2020-10-09 01:51:28.416654 0'0 218:261 [0] 0 [0] 0 0'0 2020-10-09 01:51:28.416441 0'0 2020-10-09 01:51:28.416441 0
7.7 0 0 0 0 0 0 0 0 active+clean 2020-10-08 09:34:00.752395 0'0 218:243 [0] 0 [0] 0 0'0 2020-10-08 09:34:00.752151 0'0 2020-10-05 23:30:16.410891 0
3.2 0 0 0 0 0 0 0 0 active+clean 2020-10-08 21:30:12.649567 0'0 218:437 [0] 0 [0] 0 0'0 2020-10-08 21:30:12.649340 0'0 2020-10-02 11:24:48.064097 0
1.0 74 0 0 0 0 612368384 287 287 active+clean 2020-10-08 12:10:51.317165 70'287 218:1574 [0] 0 [0] 0 70'287 2020-10-08 12:10:51.317042 70'287 2020-10-08 12:10:51.317042 0
2.3 4 0 0 0 0 8388804 119 119 active+clean 2020-10-08 14:37:17.781382 218'119 218:1700 [0] 0 [0] 0 218'104 2020-10-08 14:37:17.781097 218'104 2020-10-08 14:37:17.781097 0
4.5 3 0 0 0 0 4198400 1532 1532 active+clean 2020-10-08 23:00:49.156878 217'624232 218:708960 [0] 0 [0] 0 217'624232 2020-10-08 23:00:49.156523 217'624232 2020-10-04 02:26:30.123265 0
5.4 0 0 0 0 0 0 0 0 active+clean 2020-10-08 18:14:55.237209 0'0 218:334 [0] 0 [0] 0 0'0 2020-10-08 18:14:55.236936 0'0 2020-10-07 14:58:11.211567 0
6.7 2 0 0 0 0 92 2 2 active+clean 2020-10-09 05:05:07.463340 142'2 218:308 [0] 0 [0] 0 142'2 2020-10-09 05:05:07.463026 142'2 2020-10-04 07:44:47.516921 0
7.6 0 0 0 0 0 0 0 0 active+clean 2020-10-09 09:26:40.918563 0'0 218:244 [0] 0 [0] 0 0'0 2020-10-09 09:26:40.918469 0'0 2020-10-09 09:26:40.918469 0
11.0 26 0 0 0 0 0 1580 1580 active+clean 2020-10-08 06:59:28.073175 218'82480 218:124014 [0] 0 [0] 0 218'76498 2020-10-08 06:59:28.073021 218'61986 2020-10-05 10:44:27.787918 0
10.1 0 0 0 0 0 0 0 0 active+clean 2020-10-08 08:31:48.712338 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 08:31:48.712031 0'0 2020-10-07 03:44:54.397047 0
9.2 0 0 0 0 0 0 0 0 active+clean 2020-10-08 15:54:41.163026 0'0 218:158 [0] 0 [0] 0 0'0 2020-10-08 15:54:41.162795 0'0 2020-10-07 09:26:01.113031 0
3.8 0 0 0 0 0 0 0 0 active+clean 2020-10-08 14:59:07.063854 0'0 218:431 [0] 0 [0] 0 0'0 2020-10-08 14:59:07.063526 0'0 2020-10-07 03:33:40.865941 0
2.9 3 0 0 0 0 12582912 229 229 active+clean 2020-10-09 07:40:35.238749 218'229 218:1358 [0] 0 [0] 0 217'199 2020-10-09 07:40:35.238655 217'199 2020-10-04 09:17:39.275859 0
1.a 77 0 0 0 0 637534227 155 155 active+clean 2020-10-08 22:51:26.423092 70'155 218:713 [0] 0 [0] 0 70'155 2020-10-08 22:51:26.422734 70'155 2020-10-08 22:51:26.422734 0
4.f 6 0 0 0 0 6488243 1508 1508 active+clean 2020-10-09 05:49:19.373954 218'154309 218:739752 [0] 0 [0] 0 217'154308 2020-10-09 05:49:19.373283 217'154308 2020-10-06 14:40:37.175097 0
8.3 3 0 0 0 0 34 11 11 active+clean 2020-10-08 21:36:08.224389 213'11 218:257 [0] 0 [0] 0 213'11 2020-10-08 21:36:08.224263 213'11 2020-10-08 21:36:08.224263 0
11.1 32 0 0 0 0 0 1556 1556 active+clean 2020-10-08 06:48:18.371502 218'96556 218:145224 [0] 0 [0] 0 218'89476 2020-10-08 06:48:18.371360 218'89476 2020-10-08 06:48:18.371360 0
10.0 0 0 0 0 0 0 0 0 active+clean 2020-10-08 19:00:45.314686 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 19:00:45.314460 0'0 2020-10-07 12:15:27.025212 0
9.3 1 0 0 0 0 0 15 15 active+clean 2020-10-09 05:58:16.090892 213'15 218:266867 [0] 0 [0] 0 213'15 2020-10-09 05:58:16.090541 213'15 2020-10-02 12:43:51.931371 0
3.9 0 0 0 0 0 0 0 0 active+clean 2020-10-08 13:57:56.740117 0'0 218:433 [0] 0 [0] 0 0'0 2020-10-08 13:57:56.739922 0'0 2020-10-02 09:09:43.302252 0
2.8 5 0 0 0 0 16580096 242 242 active+clean 2020-10-08 20:43:07.343247 218'242 218:218351 [0] 0 [0] 0 218'161 2020-10-08 20:43:07.342942 218'161 2020-10-02 09:23:21.553739 0
1.b 79 0 0 0 0 629145650 1186 1186 active+clean 2020-10-08 11:07:32.897503 70'1186 218:662508 [0] 0 [0] 0 70'1186 2020-10-08 11:07:32.897338 70'1186 2020-10-01 20:00:17.716690 0
4.e 3 0 0 0 0 176164 1507 1507 active+clean 2020-10-08 09:48:44.386416 217'407617 218:534387 [0] 0 [0] 0 217'407617 2020-10-08 09:48:44.386260 217'407617 2020-10-08 09:48:44.386260 0
8.2 0 0 0 0 0 0 4 4 active+clean 2020-10-08 12:11:18.403767 213'4 218:252 [0] 0 [0] 0 213'4 2020-10-08 12:11:18.403584 213'4 2020-10-08 12:11:18.403584 0
11.6 24 0 0 0 0 0 1554 1554 active+clean 2020-10-08 01:28:07.112299 218'85554 218:128593 [0] 0 [0] 0 218'78106 2020-10-08 01:28:07.112159 218'71040 2020-10-06 17:16:30.513491 0
10.7 0 0 0 0 0 0 0 0 active+clean 2020-10-09 02:35:31.561908 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-09 02:35:31.561519 0'0 2020-10-02 15:26:03.205788 0
9.4 1 0 0 0 0 0 15 15 active+clean 2020-10-08 23:34:53.915915 213'15 218:261068 [0] 0 [0] 0 213'15 2020-10-08 23:34:53.915555 213'15 2020-10-05 10:08:15.797193 0
2.f 0 0 0 0 0 0 387 387 active+clean 2020-10-08 15:39:20.897579 218'387 218:392027 [0] 0 [0] 0 217'374 2020-10-08 15:39:20.897417 217'374 2020-10-07 06:39:03.564104 0
1.c 77 0 0 0 0 629145890 1587 1587 active+clean 2020-10-08 09:26:22.092018 71'2487 218:66700 [0] 0 [0] 0 71'2487 2020-10-08 09:26:22.091911 71'2487 2020-10-02 07:45:20.134932 0
3.e 0 0 0 0 0 0 0 0 active+clean 2020-10-08 00:48:47.232535 0'0 218:442 [0] 0 [0] 0 0'0 2020-10-08 00:48:47.232458 0'0 2020-10-03 10:17:34.079407 0
4.9 8 0 0 0 0 12448901 1527 1527 active+clean 2020-10-08 07:39:19.650861 217'352527 218:793189 [0] 0 [0] 0 217'352527 2020-10-08 07:39:19.650602 217'352527 2020-10-02 02:37:53.666969 0
8.5 4 0 0 0 0 136 6 6 active+clean 2020-10-08 10:31:37.257934 162'6 218:262 [0] 0 [0] 0 162'6 2020-10-08 10:31:37.257782 162'6 2020-10-03 08:19:31.836694 0
11.7 23 0 0 0 0 0 1588 1588 active+clean 2020-10-09 09:10:21.852553 218'73888 218:111064 [0] 0 [0] 0 218'73482 2020-10-09 09:10:21.852344 218'56354 2020-10-05 15:02:49.374283 0
10.6 0 0 0 0 0 0 0 0 active+clean 2020-10-08 10:38:52.801017 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 10:38:52.800708 0'0 2020-10-07 09:15:18.381402 0
9.5 2 0 0 0 0 0 30 30 active+clean 2020-10-08 18:12:26.845373 213'30 218:537512 [0] 0 [0] 0 213'30 2020-10-08 18:12:26.845071 213'30 2020-10-05 09:46:03.133136 0
2.e 1 0 0 0 0 0 335 335 active+clean 2020-10-08 05:05:16.823255 217'335 218:268964 [0] 0 [0] 0 217'335 2020-10-08 05:05:16.823130 217'335 2020-10-04 07:47:15.933299 0
1.d 81 0 0 0 0 679477248 167 167 active+clean 2020-10-09 08:47:39.232720 63'167 218:48251 [0] 0 [0] 0 63'167 2020-10-09 08:47:39.232597 63'167 2020-10-04 04:23:20.525375 0
3.f 0 0 0 0 0 0 0 0 active+clean 2020-10-09 07:39:45.124798 0'0 218:437 [0] 0 [0] 0 0'0 2020-10-09 07:39:45.124460 0'0 2020-10-06 14:37:28.122757 0
4.8 6 0 0 0 0 8519750 1564 1564 active+clean 2020-10-09 10:26:02.645356 218'318864 218:828555 [0] 0 [0] 0 218'318864 2020-10-09 10:26:02.645216 217'318863 2020-10-06 20:12:21.375773 0
8.4 5 0 0 0 0 1518 11 11 active+clean 2020-10-08 23:57:36.427105 213'11 218:267 [0] 0 [0] 0 213'11 2020-10-08 23:57:36.426909 213'11 2020-10-06 06:56:40.729308 0
11.4 29 0 0 0 0 0 1506 1506 active+clean 2020-10-08 20:23:25.016955 218'105706 218:158789 [0] 0 [0] 0 218'101664 2020-10-08 20:23:25.016836 218'84154 2020-10-06 03:55:52.470750 0
10.5 0 0 0 0 0 0 0 0 active+clean 2020-10-09 05:36:32.329904 0'0 218:147 [0] 0 [0] 0 0'0 2020-10-09 05:36:32.329666 0'0 2020-10-09 05:36:32.329666 0
9.6 1 0 0 0 0 0 15 15 active+clean 2020-10-08 07:32:40.729876 213'15 218:261128 [0] 0 [0] 0 213'15 2020-10-08 07:32:40.729567 213'15 2020-10-06 03:35:00.785216 0
3.c 0 0 0 0 0 0 0 0 active+clean 2020-10-09 07:26:06.064060 0'0 218:439 [0] 0 [0] 0 0'0 2020-10-09 07:26:06.063823 0'0 2020-10-04 05:06:59.242164 0
2.d 5 0 0 0 0 12324 74 74 active+clean 2020-10-08 08:42:00.640450 218'74 218:586741 [0] 0 [0] 0 217'69 2020-10-08 08:42:00.640281 217'69 2020-10-08 08:42:00.640281 0
1.e 79 0 0 0 0 654311424 310 310 active+clean 2020-10-09 08:06:25.412511 70'310 218:1681 [0] 0 [0] 0 70'310 2020-10-09 08:06:25.412370 70'310 2020-10-09 08:06:25.412370 0
4.b 1 0 0 0 0 35 1524 1524 active+clean 2020-10-09 09:17:12.386608 217'212424 218:905081 [0] 0 [0] 0 217'212424 2020-10-09 09:17:12.386265 217'212424 2020-10-09 09:17:12.386265 0
8.7 4 0 0 0 0 558 10 10 active+clean 2020-10-08 10:59:31.054899 213'10 218:268 [0] 0 [0] 0 213'10 2020-10-08 10:59:31.054717 213'10 2020-10-04 11:57:57.741507 0
11.5 25 0 0 0 0 0 1506 1506 active+clean 2020-10-09 00:35:25.756950 218'91706 218:137757 [0] 0 [0] 0 218'89194 2020-10-09 00:35:25.756819 218'89194 2020-10-09 00:35:25.756819 0
10.4 0 0 0 0 0 0 0 0 active+clean 2020-10-08 13:00:40.455775 0'0 218:145 [0] 0 [0] 0 0'0 2020-10-08 13:00:40.455565 0'0 2020-10-05 20:18:37.700719 0
9.7 1 0 0 0 0 0 15 15 active+clean 2020-10-08 01:44:33.857684 213'15 218:266769 [0] 0 [0] 0 213'15 2020-10-08 01:44:33.856247 213'15 2020-10-05 14:22:35.612337 0
3.d 0 0 0 0 0 0 0 0 active+clean 2020-10-08 14:37:18.523328 0'0 218:430 [0] 0 [0] 0 0'0 2020-10-08 14:37:18.523219 0'0 2020-10-04 15:16:33.537922 0
2.c 2 0 0 0 0 98 314 314 active+clean 2020-10-08 12:17:04.319742 218'314 218:1013452 [0] 0 [0] 0 217'311 2020-10-08 12:17:04.319418 217'311 2020-10-03 15:08:22.893984 0
1.f 76 0 0 0 0 620757056 438 438 active+clean 2020-10-09 06:58:50.424828 218'438 218:1222 [0] 0 [0] 0 217'434 2020-10-09 06:58:50.424527 217'434 2020-10-09 06:58:50.424527 0
4.a 9 0 0 0 0 16774198 1553 1553 active+clean 2020-10-09 00:59:09.998945 217'298853 218:554641 [0] 0 [0] 0 217'298853 2020-10-09 00:59:09.998692 217'298853 2020-10-04 04:20:44.069459 0
8.6 1 0 0 0 0 0 6 6 active+clean 2020-10-09 01:15:56.969147 213'6 218:252 [0] 0 [0] 0 213'6 2020-10-09 01:15:56.968950 213'6 2020-10-05 12:10:56.851219 0
11 207 0 0 0 0 0 12438 12438
1 1280 0 0 0 0 10534166580 8616 8616
2 37 0 0 0 0 70927032 4972 4972
3 0 0 0 0 0 0 0 0
4 77 0 0 0 0 103164205 24670 24670
5 0 0 0 0 0 0 0 0
6 4 0 0 0 0 1113 4 4
7 0 0 0 0 0 0 0 0
8 21 0 0 0 0 2246 56 56
9 8 0 0 0 0 0 120 120
10 0 0 0 0 0 0 0 0
sum 1634 0 0 0 0 10708261176 50876 50876
OSD_STAT USED AVAIL TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM
0 11.0GiB 589GiB 600GiB [] 120 120
sum 11.0GiB 589GiB 600GiB
2、查看一个PG的map
# ceph pg map 2.c
输出信息：
osdmap e218 pg 2.c (2.c) -> up [0] acting [0]
六、pool
1、查看ceph集群中的pool数量
# ceph osd lspools
输出结果：
1 images,2 volumes,3 backups,4 vms,5 k8s,6 .rgw.root,7 cephfs_data,8 cephfs_metadata,9 default.rgw.control,10 default.rgw.meta,11 default.rgw.log,
2、在ceph集群中创建一个pool
ceph osd pool create jiayuan 100 #这里的100指的是PG组
3、为一个ceph pool配置配额
ceph osd pool set-quota data max_objects 10000
4、在集群中删除一个pool
ceph osd pool delete testpool testpool --yes-i-really-really-mean-it #集群名字需要重复两次
备注：删除pool需要在配置文件ceph.conf中 [mon]添加mon allow pool delete = true并重启mon服务, 如systemctl restart ceph-mon.target
5、显示集群中pool的详细信息
# rados df
8、查看data池的pg数量
# ceph osd pool get volumes pg_num
输出信息：
pg_num: 16
9、设置data池的最大存储空间为100T（默认是1T)
# ceph osd pool set data target_max_bytes 100000000000000
10、设置data池的副本数是1
ceph osd pool set jiayuan size 1
11、设置data池能接受写操作的最小副本为1
ceph osd pool set jiayuan min_size 1
12、查看集群中所有pool的副本尺寸
ceph osd dump | grep 'replicated size'
13、设置一个pool的pg数量
ceph osd pool set jiayuan pg_num 128
14、设置一个pool的pgp数量
ceph osd pool set jiayuan pgp_num 128
15、查询public与cluster是否分开，在存储节点执行以下命令：
ps aux | grep osd
输出信息：
root 87153 39.3 3.1 6846900 4132380 ? Sl Jan09 155212:56 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 73
root 87293 8.0 2.8 6667388 3778780 ? Sl Jan09 31817:50 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 67
root 87478 10.8 2.7 6425832 3557612 ? Sl Jan09 42731:55 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 61
root 87660 9.8 2.7 7128020 3670144 ? Sl Jan09 38932:51 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 55
root 87834 7.8 2.8 6864492 3739852 ? Sl Jan09 30920:15 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 48
root 88074 8.0 2.6 6817916 3528300 ? Sl Jan09 31631:47 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 43
root 88247 9.6 2.7 6984468 3565332 ? Sl Jan09 38053:06 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 36
root 88384 7.5 2.7 6774176 3654044 ? Sl Jan09 29702:47 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 30
root 88594 8.1 2.6 6961384 3539420 ? Sl Jan09 32002:42 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 86
root 88812 8.4 2.8 7164612 3718620 ? Sl Jan09 33381:07 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 78
root 88988 8.3 2.8 6862320 3704392 ? Sl Jan09 32853:56 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 12
root 89164 8.3 2.7 6807364 3675788 ? Sl Jan09 32893:14 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 6
root 89300 9.4 2.7 6323416 3638680 ? Sl Jan09 37184:16 /usr/bin/ceph-osd -f --public-addr 10.186.9.13 --cluster-addr 10.186.8.13 -i 2
root 161819 0.0 0.0 50772 6912 ? S 14:27 0:00 /usr/bin/python /usr/local/bin/zabbix-ceph get --target osd.num.in
root 161943 0.0 0.0 112716 956 pts/0 S+ 14:27 0:00 grep --color=auto osd
七、rados
八、rbd
1、查看ceph中一个pool里的所有镜像
# rbd ls volumes
输出信息：
volume-2203c1c8-64ea-4da2-934c-011ba9f99603
volume-6a5ea26a-377b-4bee-84c2-d8b95b58ca04
volume-b3b7e767-b63b-46ab-9f24-e52496551d6f
volume-c7c6642e-186d-4153-a2e1-8fdbd422c13c
volume-d9d935b8-74cd-4d7c-8c75-9cc2ef76051a
2 、查看ceph pool中一个镜像的信息
rbd info -p volumes --image volume-2203c1c8-64ea-4da2-934c-011ba9f99603
输出信息：
rbd image 'volume-2203c1c8-64ea-4da2-934c-011ba9f99603':
size 1GiB in 256 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.2238386b8b4567
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
create_timestamp: Fri Oct 9 09:23:02 2020
3、在test池中创建一个命名为test2的10000M的镜像
# rbd create -p volumes --size 10000 test2
备注：有些操作系统需要强制指定特性, 加上–image-feature layering ：
# rbd create -p volumes --size 10000 test2 --image-feature layering
4、删除一个镜像
# rbd rm -p volumes test2
5、调整一个镜像的尺寸
# rbd resize -p volumes --size 20000 test2
6、镜像快照的创建、查询、删除
创建：
# rbd snap create volumes/test2@snap1
查询：
# rbd snap -p volumes ls test2
输出：
SNAPID NAME SIZE TIMESTAMP
5 snap1 19.5GiB Fri Oct 9 14:51:14 2020
删除：
# rbd snap rm volumes/test2@snap1
删除一个镜像文件的所有快照：
# rbd snap purge -p volumes test2
7、导出/导入镜像
导出：
# rbd export -p volumes test2 /root/test2.raw
导入：
# rbd import /root/test2.raw -p volumes --image test-import
1111
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
