Ceph集群搭建（octopus版）之 CephFS导出nfs共享

当前ceph版本估计只有octopus是最新最稳也是最好用的

Linux系统选择CentOS Stream 8因为他维护到2024年，还是很好用的

准备3台机所有搭建在192.168.10.25这台机完成

192.168.10.25
192.168.10.26
192.168.10.27

静态ip 设置

nmcli con add con-name $NetName ifname $NetName autoconnect yes type ethernet \
ip4 192.168.10.25/24 ipv4.dns "114.114.114.114" ipv4.gateway "192.168.10.254"
#设置完成后执行下reload网络
nmcli c reload

有必要的话。可以做个时间同步

timedatectl set-timezone Asia/Shanghai
ntpdate ntp3.aliyun.com

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
dnf  install wntp
ntpdate ntp3.aliyun.com 

可科学上网的同学可以使用官网的方式去简单初始化集群

https://docs.ceph.com/en/pacific/cephadm/install/#install-cephadm

我的用是国内源比较快

1.源配置

tee /etc/yum.repos.d/ceph.repo <<-'EOF'
[Ceph-noarch]
name=Ceph noarch
baseurl=http://mirrors.163.com/ceph/rpm-octopus/el8/noarch/
enabled=1
gpgcheck=0

[Ceph-source]
name=Ceph SRPMS
baseurl=http://mirrors.163.com/ceph/rpm-octopus/el8/x86_64/
enabled=1
gpgcheck=0
EOF

2 安装包

dnf install cephadm  -y
yum info cephadm
#安装源
cephadm install
#开始初始化一个最小的集群
cephadm bootstrap --mon-ip 192.168.10.25

3 拷贝公钥其他节点

ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.10.26
ssh-copy-id -f -i /etc/ceph/ceph.pub root@192.168.10.27

---------其他节点安装cephadm-------

dnf install cephadm -y

[root@ceph25 ~]# cephadm install
Installing packages ['cephadm']...

4再回到安装节点25添加节点。将自己也添加进去

ceph orch host add ceph26 192.168.10.26
ceph orch host add ceph27 192.168.10.27
ceph orch host add ceph25 192.168.10.25 `

--------------遵循官网的意见打标签--------------------

 ceph orch host label add ceph25 _admin
 ceph orch host label add ceph26 _admin
 ceph orch host label add ceph27 _admin

[root@ceph25 ~]# ceph orch host ls
HOST    ADDR           LABELS  STATUS
ceph25  192.168.10.25  _admin
ceph26  192.168.10.26  _admin
ceph27  192.168.10.27  _admin

mon和mgr自动添加,只需要调整mon的个数即可

[root@ceph25 ~]# ceph -s
 cluster:
   id:     0fcb69c6-a50c-11ec-b3a0-000c297b25be
   health: HEALTH_WARN
           OSD count 0 < osd_pool_default_size 3

 services:
   mon: 3 daemons, quorum ceph25,ceph26,ceph27 (age 4m)
   mgr: ceph25.bassgu(active, since 17m), standbys: ceph26.bgrvgp
   osd: 0 osds: 0 up, 0 in

 data:
   pools:   0 pools, 0 pgs
   objects: 0 objects, 0 B
   usage:   0 B used, 0 B / 0 B avail
   pgs:

mon有点多现在我们调整为3个，因为只有3个节点

[root@ceph25 ~]# ceph orch ls
NAME           RUNNING  REFRESHED  AGE  PLACEMENT  IMAGE NAME                                IMAGE ID
alertmanager       1/1  3m ago     19m  count:1    quay.io/prometheus/alertmanager:v0.20.0   0881eb8f169f
crash              3/3  3m ago     19m  *          quay.io/ceph/ceph:v15                     24f313b0d66d
grafana            1/1  3m ago     19m  count:1    quay.io/ceph/ceph-grafana:6.7.4           557c83e11646
mgr                2/2  3m ago     19m  count:2    quay.io/ceph/ceph:v15                     24f313b0d66d
mon                3/5  3m ago     19m  count:5    quay.io/ceph/ceph:v15                     24f313b0d66d
node-exporter      3/3  3m ago     19m  *          quay.io/prometheus/node-exporter:v0.18.1  e5a616e4b9cf
prometheus         1/1  3m ago     19m  count:1    quay.io/prometheus/prometheus:v2.18.1     de242295e225

调整为3个

[root@ceph25 ~]# ceph orch apply mon 3
Scheduled mon update...

[root@ceph25 ~]# ceph orch ls
NAME           RUNNING  REFRESHED  AGE  PLACEMENT  IMAGE NAME                                IMAGE ID
alertmanager       1/1  4m ago     19m  count:1    quay.io/prometheus/alertmanager:v0.20.0   0881eb8f169f
crash              3/3  4m ago     19m  *          quay.io/ceph/ceph:v15                     24f313b0d66d
grafana            1/1  4m ago     19m  count:1    quay.io/ceph/ceph-grafana:6.7.4           557c83e11646
mgr                2/2  4m ago     19m  count:2    quay.io/ceph/ceph:v15                     24f313b0d66d
mon                3/3  4m ago     1s   count:3    quay.io/ceph/ceph:v15                     24f313b0d66d
node-exporter      3/3  4m ago     19m  *          quay.io/prometheus/node-exporter:v0.18.1  e5a616e4b9cf
prometheus         1/1  4m ago     19m  count:1    quay.io/prometheus/prometheus:v2.18.1     de242295e225

自动添加所有osd级初始化磁盘比以前的版本方便多了，

ceph orch apply osd --all-available-devices

[root@ceph25 ~]# ceph -s
  cluster:
    id:     0fcb69c6-a50c-11ec-b3a0-000c297b25be
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 44 sec, mon.ceph26 has slow ops

  services:
    mon: 3 daemons, quorum ceph25,ceph26,ceph27 (age 11m)
    mgr: ceph25.bassgu(active, since 23m), standbys: ceph26.bgrvgp
    osd: 3 osds: 3 up (since 47s), 3 in (since 47s)

  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 117 GiB / 120 GiB avail
    pgs:     1 active+clean

开始部署nfs服务

1 使用nfs需要创建cephFS因为nfs功能工作在cephFS之上的

创建一个fs服务创建再25，26的服务器上

ceph fs volume create cephfs-demo "ceph25,ceph26"

[root@ceph25 ~]# ceph fs ls
name: cephfs-demo, metadata pool: cephfs.cephfs-demo.meta, data pools: [cephfs.cephfs-demo.data ]

查看

[root@ceph25 ~]# ceph orch ls
NAME                       RUNNING  REFRESHED  AGE  PLACEMENT             IMAGE NAME                                IMAGE ID
alertmanager                   1/1  5m ago     74m  count:1               quay.io/prometheus/alertmanager:v0.20.0   0881eb8f169f
crash                          3/3  5m ago     75m  *                     quay.io/ceph/ceph:v15                     24f313b0d66d
grafana                        1/1  5m ago     74m  count:1               quay.io/ceph/ceph-grafana:6.7.4           557c83e11646
mds.cephfs-demo                3/3  5m ago     28s  ceph25;ceph26;ceph27  quay.io/ceph/ceph:v15                     24f313b0d66d

[root@ceph25 mnt]# ceph mds stat
cephfs-demo:1 {0=cephfs-demo.ceph25.cqfkji=up:active} 2 up:standby

调整为一主2从

[root@ceph25 ~]# ceph orch apply mds cephfs-demo "ceph25,ceph26,ceph27"
Scheduled mds.cephfs-demo update...

创建一个nfs服务

 ceph orch apply nfs nfs-share

创建一个池

ceph osd pool create nfs-pools

[root@ceph25 ~]# ceph osd lspools
1 device_health_metrics
2 nfs-pools

创建一个高可用nfs分别运行在ceph26,ceph27上

ceph orch apply nfs my-nfs nfs-pools --placement="ceph26,ceph27"

[root@ceph25 ~]# ceph orch apply nfs my-nfs nfs-pools --placement="ceph26,ceph27"
Scheduled nfs.my-nfs update...

查看

[root@ceph25 ~]# ceph orch ls
NAME                       RUNNING  REFRESHED  AGE   PLACEMENT      IMAGE NAME                                IMAGE ID
alertmanager                   1/1  6m ago     55m   count:1        quay.io/prometheus/alertmanager:v0.20.0   0881eb8f169f
crash                          3/3  6m ago     55m   *              quay.io/ceph/ceph:v15                     24f313b0d66d
grafana                        1/1  6m ago     55m   count:1        quay.io/ceph/ceph-grafana:6.7.4           557c83e11646
mgr                            2/2  6m ago     55m   count:2        quay.io/ceph/ceph:v15                     24f313b0d66d
mon                            3/3  6m ago     35m   count:3        quay.io/ceph/ceph:v15                     24f313b0d66d
nfs.my-nfs                     2/2  106s ago   118s  ceph26;ceph27  quay.io/ceph/ceph:v15                     24f313b0d66d

[root@ceph25 ~]# ceph nfs cluster ls
my-nfs
[root@ceph25 ~]# ceph nfs cluster info nfs-share
{}

本次使用的是UI创建也可以使用命令行

客户机安装nfs

 yum -y install nfs-utils
 #使用刚才的伪装目录挂载
 mount -t nfs 192.168.10.26:/nfsdemo /mnt/

开机挂载使用

[root@aws ~]# tail -n 2 /etc/fstab

192.168.10.26:/nfsdemo  /mnt  nfs  _netdev,defaults  0 0

mount -a

其他挂载

1.内核挂载

客户端安装cpeh内核包

yum -y install ceph-common ceph-fuse

[root@aws ceph]# lsmod | grep ceph
ceph                  438272  1
libceph               413696  1 ceph
fscache               385024  2 ceph,nfs
netfs                  45056  2 ceph,fscache
libcrc32c              16384  4 nf_conntrack,nf_nat,xfs,libceph

挂载方法1

[root@ceph25 mnt]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
        key = AQDPrjFiaOFaDxAA9cJ2ksRjXyLXFTgV7LwIew==

mount -t ceph 192.168.10.25:6789,192.168.10.26:6789,192.168.10.27:6789:/ /mnt/ -o name=admin,secret=AQDPrjFiaOFaDxAA9cJ2ksRjXyLXFTgV7LwIew==

挂载方法2

需要从服务端拷贝相应文件

scp /etc/ceph/ceph.client.admin.keyring root@192.168.10.20:/etc/ceph/
scp /etc/ceph/ceph.conf  root@192.168.10.20:/etc/ceph/

mount -t ceph 192.168.10.25:6789,192.168.10.26:6789,192.168.10.27:6789:/ /mnt/ -o name=admin

服务端访问

[root@ceph25 mnt]# ceph df
--- RAW STORAGE ---
CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
ssd    120 GiB  117 GiB  900 KiB   3.0 GiB       2.50
TOTAL  120 GiB  117 GiB  900 KiB   3.0 GiB       2.50

--- POOLS ---
POOL                     ID  PGS  STORED  OBJECTS  USED     %USED  MAX AVAIL
device_health_metrics     1    1     0 B        0      0 B      0     37 GiB
nfs-pools                 2   32   404 B        5   36 KiB      0     37 GiB
cephfs.cephfs-demo.meta   3   32  38 KiB       23  216 KiB      0     37 GiB
cephfs.cephfs-demo.data   4   32    18 B        1   12 KiB      0     37 GiB

更多挂载姿势可以参考官网

另外一直针对较老的系统，比如内核也比较佬，就使用这种用户空间挂载的方式
安装包

 yum -y install ceph-fuse
 
 #拷贝相应文件
scp /etc/ceph/ceph.client.admin.keyring root@192.168.10.20:/etc/ceph/
scp /etc/ceph/ceph.conf  root@192.168.10.20:/etc/ceph/

#加cli方式
ceph-fuse -n client.admin  -m  192.168.10.25:6789,192.168.10.26:6789,192.168.10.27:6789 /mnt/
#直接admin
 ceph-fuse   -m  192.168.10.25:6789,192.168.10.26:6789,192.168.10.27:6789 /mnt/

官网参考价格
持久化自己官网看了