ceph 旧OSD 节点 格式化 数据，加入新 ceph集群

旧有的ceph osd ， 想要格式化之后加入新的ceph节点。

 

查询 osd 旧有 数据，后面的步骤会用到。

[root@ceph-207 ~]# ceph-volume lvm list
====== osd.1 =======
  [block]       /dev/ceph-58ef1d0f-272b-4273-82b1-689946254645/osd-block-e0efe172-778e-46e1-baa2-cd56408aac34

      block device              /dev/ceph-58ef1d0f-272b-4273-82b1-689946254645/osd-block-e0efe172-778e-46e1-baa2-cd56408aac34
      block uuid                hCx4XW-OjKC-OC8Y-jEg2-NKYo-Pb6f-y9Nfl3
      cephx lockbox secret      
      cluster fsid              b7e4cb56-9cc8-4e44-ab87-24d4253d0951
      cluster name              ceph
      crush device class        None
      encrypted                 0
      osd fsid                  e0efe172-778e-46e1-baa2-cd56408aac34
      osd id                    1
      osdspec affinity          
      type                      block
      vdo                       0
      devices                   /dev/sdb

 

直接加入集群报错：

ceph-volume lvm activate 1 e0efe172-778e-46e1-baa2-cd56408aac34

目前遇到了两类报错：

osd.1 21 heartbeat_check: no reply from 192.168.8.206:6804 osd.0 ever on either front or back, first ping sent 2020-11-26T16:00:04.842947+0800 (oldest deadline 2020-11-26T16:00:24.842947+0800)

stderr: Calculated size of logical volume is 0 extents. Needs to be larger.

--> Was unable to complete a new OSD, will rollback changes

 

格式化数据，重新加入新的ceph集群：

 

1、停止osd 服务 ， @ 后面的 1 为 ceph-volume lvm list 命令查询出的 osd id

systemctl stop ceph-osd@1

2、重处理 osd lvm， 1 还是 osd id

ceph-volume lvm zap --osd-id 1

3、查询 lvs 信息， 删除 lv、pg 等信息

[root@ceph-207 ~]# lvs
  LV                                             VG                                        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-block-e0efe172-778e-46e1-baa2-cd56408aac34 ceph-58ef1d0f-272b-4273-82b1-689946254645 -wi-a-----  <16.00g                                                    
  home                                           cl                                        -wi-ao---- <145.12g                                                    
  root                                           cl                                        -wi-ao----   50.00g                                                    
  swap                                           cl                                        -wi-ao----   <3.88g                                                    
[root@ceph-207 ~]# vgremove ceph-58ef1d0f-272b-4273-82b1-689946254645
Do you really want to remove volume group "ceph-58ef1d0f-272b-4273-82b1-689946254645" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume ceph-58ef1d0f-272b-4273-82b1-689946254645/osd-block-e0efe172-778e-46e1-baa2-cd56408aac34? [y/n]: y
  Logical volume "osd-block-e0efe172-778e-46e1-baa2-cd56408aac34" successfully remove

4、将主机上的磁盘重新加入新的ceph集群

ceph-volume lvm create --data /dev/sdb

5、查询下 osd tree ， 磁盘挂载情况

[root@ceph-207 ~]# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME          STATUS  REWEIGHT  PRI-AFF
-1         0.07997  root default                                
-3         0.03508      host ceph-206                           
 0    hdd  0.01559          osd.0          up   1.00000  1.00000
 1    hdd  0.01949          osd.1          up   1.00000  1.00000
-5         0.04489      host ceph-207                           
 2    hdd  0.01559          osd.2          up   1.00000  1.00000
[root@ceph-207 ~]# lsblk
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0    0   200G  0 disk 
├─sda1                                                                                                  8:1    0     1G  0 part /boot
└─sda2                                                                                                  8:2    0   199G  0 part 
  ├─cl-root                                                                                           253:0    0    50G  0 lvm  /
  ├─cl-swap                                                                                           253:1    0   3.9G  0 lvm  [SWAP]
  └─cl-home                                                                                           253:2    0 145.1G  0 lvm  /home
sdb                                                                                                     8:16   0    16G  0 disk 
└─ceph--c221ed63--d87a--4bbd--a503--d8f2ed9e806b-osd--block--530376b8--c7bc--4d64--bc0c--4f8692559562 253:3    0    16G  0 lvm  
sr0