本文尝试在龙蜥Anolis OS8.6QU1上通过cephadm部署ceph17.2.0分布式块存储,实现国产化操作系统上的存储系统构建,本步骤为完成首节点mon节点和图形界面后在其它节点进行的基础安装,以ceph2节点为例。
一、按兼容模式安装Anolis OS8.6QU1系统
参照本人前面写的文档,先行兼容模式最小化安装Anolis OS8.6QU1系统
确认系统运行在RHCK内核下(内核版本为4.18.0)
[root@ceph2 opt]# rpm -qa |grep kernel
kernel-modules-4.18.0-372.16.1.an8_6.x86_64
kernel-tools-libs-4.18.0-372.16.1.an8_6.x86_64
kernel-4.18.0-372.16.1.an8_6.x86_64
kernel-core-4.18.0-372.16.1.an8_6.x86_64
kernel-devel-4.18.0-372.16.1.an8_6.x86_64
kernel-headers-4.18.0-372.16.1.an8_6.x86_64
kernel-tools-4.18.0-372.16.1.an8_6.x86_64
二、完成镜像准备
1、下载所需容器镜像:
podman pull docker.io/library/haproxy:2.3
podman pull quay.io/ceph/ceph:v17.2.0
podman pull quay.io/ceph/ceph-grafana:8.3.5
podman pull quay.io/prometheus/prometheus:v2.33.4
podman pull quay.io/prometheus/node-exporter:v1.3.1
podman pull quay.io/prometheus/alertmanager:v0.23.0
podman pull docker.io/maxwo/snmp-notifier:v1.2.1
podman pull docker.io/arcts/keepalived:latest
2、保存所拉取的镜像:
podman save -o images/cephv17.2.0.tar quay.io/ceph/ceph:v17.2.0
podman save -o images/prometheus-v2.33.4.tar quay.io/prometheus/prometheus:v2.33.4
podman save -o images/node-exporter-v1.3.1.tar quay.io/prometheus/node-exporter:v1.3.1
podman save -o images/alertmanager-v0.23.0.tar quay.io/prometheus/alertmanager:v0.23.0
podman save -o images/ceph-grafana-8.3.5.tar quay.io/ceph/ceph-grafana:8.3.5
podman save -o images/haproxy-2.3.tar docker.io/library/haproxy:2.3
podman save -o images/keepalived.tar docker.io/arcts/keepalived:latest
podman save -o images/snmp-notifier-v1.2.1.tar docker.io/maxwo/snmp-notifier:v1.2.1
3、查看确认相关镜像文件:
[root@localhost images]# ll
总用量 2304528
-rw-r--r-- 1 AAAA AAAA 58962432 5月 27 2022 alertmanager-v0.23.0.tar
-rw-r--r-- 1 AAAA AAAA 570742784 5月 27 2022 ceph-grafana-8.3.5.tar
-rw-r--r-- 1 AAAA AAAA 1368927232 5月 27 2022 cephv17.2.0.tar
-rw-r--r-- 1 AAAA AAAA 102969856 5月 27 2022 haproxy-2.3.tar
-rw-r--r-- 1 AAAA AAAA 16243712 5月 27 2022 keepalived.tar
-rw-r--r-- 1 AAAA AAAA 22312960 5月 27 2022 node-exporter-v1.3.1.tar
-rw-r--r-- 1 AAAA AAAA 204994048 5月 27 2022 prometheus-v2.33.4.tar
-rw-r--r-- 1 AAAA AAAA 14664192 5月 27 2022 snmp-notifier-v1.2.1.tar
三、准备ceph-admin\ceph-common相关公共文件包
如下文件清单:
[root@localhost ceph-commonV17.2.0]# ll
总用量 65304
-rw-r--r-- 1 AAAA AAAA 72292 5月 27 2022 cephadm-17.2.0-0.el8.noarch.rpm
-rw-r--r-- 1 AAAA AAAA 5513604 5月 27 2022 ceph-base-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 24141596 5月 27 2022 ceph-common-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 17611856 5月 27 2022 ceph-osd-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 24716 5月 27 2022 ceph-selinux-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 322596 5月 27 2022 gperftools-libs-2.7-6.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 205164 5月 27 2022 libbabeltrace-1.5.4-3.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 809020 5月 27 2022 libcephfs2-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 393712 5月 27 2022 libibverbs-37.2-1.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 60164 5月 27 2022 liboath-2.6.2-3.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 47804 5月 27 2022 librabbitmq-0.9.0-2.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 3604092 5月 27 2022 librados2-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 494348 5月 27 2022 libradosstriper1-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 4060516 5月 27 2022 librbd1-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 361824 5月 27 2022 librdkafka-0.11.4-1.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 79628 5月 27 2022 librdmacm-37.2-1.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 5573832 5月 27 2022 librgw2-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 76804 5月 27 2022 libunwind-1.3.1-3.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 265716 5月 27 2022 lttng-ust-2.8.1-11.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 45608 5月 27 2022 python3-ceph-argparse-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 105960 5月 27 2022 python3-ceph-common-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 181704 5月 27 2022 python3-cephfs-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 44920 5月 27 2022 python3-prettytable-0.7.2-14.el8.noarch.rpm
-rw-r--r-- 1 AAAA AAAA 374300 5月 27 2022 python3-rados-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 353620 5月 27 2022 python3-rbd-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 111944 5月 27 2022 python3-rgw-17.2.0-0.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 64676 5月 27 2022 rdma-core-29.0-3.el8.x86_64.rpm
-rw-r--r-- 1 AAAA AAAA 1818384 5月 27 2022 thrift-0.13.0-2.el8.x86_64.rpm
四、主机配置:
-
配置hosts文件:
[root@localhost opt]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.188.1 ceph1
192.168.188.2 ceph2
192.168.188.3 ceph3
192.168.188.4 ceph4
192.168.188.203 iscsi1
192.168.188.204 iscsi2
-
修改chronyd服务配置,该服务默认已安装并启用,注意配置和首节点有区别的,192.168.188.1为集群首节点作时钟源地址
[root@cehp2 opt]# vi /etc/chrony.conf
[root@cehp2 opt] # cat /etc/chrony.conf
pool 192.168.188.1
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
[root@cehp1 opt] # systemctl restart chronyd
[root@ceph2 opt]# systemctl status chronyd
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-03-10 16:08:42 CST; 6s ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 2827 ExecStopPost=/usr/libexec/chrony-helper remove-daemon-state (code=exited, status=0/SUCCESS)
Process: 2837 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 2832 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 2836 (chronyd)
Tasks: 1 (limit: 1649436)
Memory: 860.0K
CGroup: /system.slice/chronyd.service
└─2836 /usr/sbin/chronyd
3月 10 16:08:42 ceph2 systemd[1]: Starting NTP client/server...
3月 10 16:08:42 ceph2 chronyd[2836]: chronyd version 4.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 >
3月 10 16:08:42 ceph2 chronyd[2836]: Frequency 0.000 +/- 1000000.000 ppm read from /var/lib/chrony/drift
3月 10 16:08:42 ceph2 systemd[1]: Started NTP client/server.
五、正式安装
-
安装ceph-admin\ceph-common相关公共文件包
[root@cehp2 ceph-commonV17.2.0]# dnf install *
上次元数据过期检查:0:04:30 前,执行于 2023年03月10日 星期五 13时10分39秒。
依赖关系解决。
=========================================================================================================================================================
软件包 架构 版本 仓库 大小
=========================================================================================================================================================
安装:
ceph-base x86_64 2:17.2.0-0.el8 @commandline 5.3 M
ceph-common x86_64 2:17.2.0-0.el8 @commandline 23 M
ceph-osd x86_64 2:17.2.0-0.el8 @commandline 17 M
ceph-selinux x86_64 2:17.2.0-0.el8 @commandline 24 k
cephadm noarch 2:17.2.0-0.el8 @commandline 71 k
gperftools-libs x86_64 2.7-6.el8 @commandline 315 k
libcephfs2 x86_64 2:17.2.0-0.el8 @commandline 790 k
liboath x86_64 2.6.2-3.el8 @commandline 59 k
librabbitmq x86_64 0.9.0-2.el8 @commandline 47 k
librados2 x86_64 2:17.2.0-0.el8 @commandline 3.4 M
libradosstriper1 x86_64 2:17.2.0-0.el8 @commandline 483 k
librbd1 x86_64 2:17.2.0-0.el8 @commandline 3.9 M
librdkafka x86_64 0.11.4-1.el8 @commandline 353 k
librdmacm x86_64 37.2-1.el8 @commandline 78 k
librgw2 x86_64 2:17.2.0-0.el8 @commandline 5.3 M
libunwind x86_64 1.3.1-3.el8 @commandline 75 k
lttng-ust x86_64 2.8.1-11.el8 @commandline 259 k
python3-ceph-argparse x86_64 2:17.2.0-0.el8 @commandline 45 k
python3-ceph-common x86_64 2:17.2.0-0.el8 @commandline 103 k
python3-cephfs x86_64 2:17.2.0-0.el8 @commandline 177 k
python3-prettytable noarch 0.7.2-14.el8 @commandline 44 k
python3-rados x86_64 2:17.2.0-0.el8 @commandline 366 k
python3-rbd x86_64 2:17.2.0-0.el8 @commandline 345 k
python3-rgw x86_64 2:17.2.0-0.el8 @commandline 109 k
rdma-core x86_64 29.0-3.el8 @commandline 63 k
thrift x86_64 0.13.0-2.el8 @commandline 1.7 M
升级:
libbabeltrace x86_64 1.5.4-3.el8 @commandline 200 k
安装依赖关系:
checkpolicy x86_64 2.9-1.el8 BaseOS 345 k
conmon x86_64 2:2.1.0-1.module+an8.6.0+10751+21528717 AppStream 54 k
container-selinux noarch 2:2.179.1-1.module+an8.6.0+10751+21528717 AppStream 57 k
containernetworking-plugins x86_64 1:1.0.1-2.module+an8.6.0+10751+21528717 AppStream 18 M
containers-common x86_64 2:1-27.module+an8.6.0+10751+21528717 AppStream 94 k
criu x86_64 3.15-3.module+an8.6.0+10751+21528717 AppStream 517 k
fuse-common x86_64 3.3.0-15.an8 BaseOS 21 k
fuse-overlayfs x86_64 1.8.2-1.module+an8.6.0+10751+21528717 AppStream 72 k
fuse3 x86_64 3.3.0-15.an8 BaseOS 53 k
fuse3-libs x86_64 3.3.0-15.an8 BaseOS 94 k
libconfig x86_64 1.5-9.el8 BaseOS 68 k
libicu x86_64 60.3-2.el8 BaseOS 8.8 M
libnet x86_64 1.1.6-15.el8 AppStream 66 k
libslirp x86_64 4.4.0-1.module+an8.5.0+10624+8ab00a81 AppStream 69 k
libstoragemgmt x86_64 1.9.1-3.an8 BaseOS 245 k
pciutils x86_64 3.7.0-1.an8 BaseOS 104 k
podman-catatonit x86_64 2:4.0.2-6.module+an8.6.0+10751+21528717 AppStream 353 k
policycoreutils-python-utils noarch 2.9-19.an8 BaseOS 252 k
protobuf-c x86_64 1.3.0-6.an8 AppStream 36 k
python3-audit x86_64 3.0.7-2.0.1.an8.2 BaseOS 86 k
python3-libsemanage x86_64 2.9-8.an8 BaseOS 127 k
python3-libstoragemgmt x86_64 1.9.1-3.an8 BaseOS 175 k
python3-pip noarch 9.0.3-22.an8 BaseOS 19 k
python3-policycoreutils noarch 2.9-19.an8 BaseOS 2.2 M
python3-pyyaml x86_64 3.12-12.el8 BaseOS 192 k
python3-setools x86_64 4.3.0-3.an8 BaseOS 623 k
python3-setuptools noarch 39.2.0-6.an8 BaseOS 162 k
python36 x86_64 3.6.8-38.module+an8.5.0+10578+67483ed2 AppStream 18 k
runc x86_64 1:1.0.3-2.module+an8.6.0+10751+21528717 AppStream 3.0 M
shadow-utils-subid x86_64 2:4.6-16.0.1.an8 BaseOS 111 k
slirp4netns x86_64 1.1.8-2.module+an8.6.0+10751+21528717 AppStream 50 k
userspace-rcu x86_64 0.10.1-4.an8 BaseOS 100 k
安装弱的依赖:
nvme-cli x86_64 1.16-3.0.1.an8 BaseOS 499 k
podman x86_64 2:4.0.2-6.module+an8.6.0+10751+21528717 AppStream 13 M
smartmontools x86_64 1:7.1-1.0.1.an8 BaseOS 511 k
降级:
libibverbs x86_64 37.2-1.el8 @commandline 384 k
启用模块流:
container-tools an8
python36 3.6
..... 65/65
已升级:
libbabeltrace-1.5.4-3.el8.x86_64
已降级:
libibverbs-37.2-1.el8.x86_64
已安装:
ceph-base-2:17.2.0-0.el8.x86_64 ceph-common-2:17.2.0-0.el8.x86_64
ceph-osd-2:17.2.0-0.el8.x86_64 ceph-selinux-2:17.2.0-0.el8.x86_64
cephadm-2:17.2.0-0.el8.noarch checkpolicy-2.9-1.el8.x86_64
conmon-2:2.1.0-1.module+an8.6.0+10751+21528717.x86_64 container-selinux-2:2.179.1-1.module+an8.6.0+10751+21528717.noarch
containernetworking-plugins-1:1.0.1-2.module+an8.6.0+10751+21528717.x86_64 containers-common-2:1-27.module+an8.6.0+10751+21528717.x86_64
criu-3.15-3.module+an8.6.0+10751+21528717.x86_64 fuse-common-3.3.0-15.an8.x86_64
fuse-overlayfs-1.8.2-1.module+an8.6.0+10751+21528717.x86_64 fuse3-3.3.0-15.an8.x86_64
fuse3-libs-3.3.0-15.an8.x86_64 gperftools-libs-2.7-6.el8.x86_64
libcephfs2-2:17.2.0-0.el8.x86_64 libconfig-1.5-9.el8.x86_64
libicu-60.3-2.el8.x86_64 libnet-1.1.6-15.el8.x86_64
liboath-2.6.2-3.el8.x86_64 librabbitmq-0.9.0-2.el8.x86_64
librados2-2:17.2.0-0.el8.x86_64 libradosstriper1-2:17.2.0-0.el8.x86_64
librbd1-2:17.2.0-0.el8.x86_64 librdkafka-0.11.4-1.el8.x86_64
librdmacm-37.2-1.el8.x86_64 librgw2-2:17.2.0-0.el8.x86_64
libslirp-4.4.0-1.module+an8.5.0+10624+8ab00a81.x86_64 libstoragemgmt-1.9.1-3.an8.x86_64
libunwind-1.3.1-3.el8.x86_64 lttng-ust-2.8.1-11.el8.x86_64
nvme-cli-1.16-3.0.1.an8.x86_64 pciutils-3.7.0-1.an8.x86_64
podman-2:4.0.2-6.module+an8.6.0+10751+21528717.x86_64 podman-catatonit-2:4.0.2-6.module+an8.6.0+10751+21528717.x86_64
policycoreutils-python-utils-2.9-19.an8.noarch protobuf-c-1.3.0-6.an8.x86_64
python3-audit-3.0.7-2.0.1.an8.2.x86_64 python3-ceph-argparse-2:17.2.0-0.el8.x86_64
python3-ceph-common-2:17.2.0-0.el8.x86_64 python3-cephfs-2:17.2.0-0.el8.x86_64
python3-libsemanage-2.9-8.an8.x86_64 python3-libstoragemgmt-1.9.1-3.an8.x86_64
python3-pip-9.0.3-22.an8.noarch python3-policycoreutils-2.9-19.an8.noarch
python3-prettytable-0.7.2-14.el8.noarch python3-pyyaml-3.12-12.el8.x86_64
python3-rados-2:17.2.0-0.el8.x86_64 python3-rbd-2:17.2.0-0.el8.x86_64
python3-rgw-2:17.2.0-0.el8.x86_64 python3-setools-4.3.0-3.an8.x86_64
python3-setuptools-39.2.0-6.an8.noarch python36-3.6.8-38.module+an8.5.0+10578+67483ed2.x86_64
rdma-core-29.0-3.el8.x86_64 runc-1:1.0.3-2.module+an8.6.0+10751+21528717.x86_64
shadow-utils-subid-2:4.6-16.0.1.an8.x86_64 slirp4netns-1.1.8-2.module+an8.6.0+10751+21528717.x86_64
smartmontools-1:7.1-1.0.1.an8.x86_64 thrift-0.13.0-2.el8.x86_64
userspace-rcu-0.10.1-4.an8.x86_64
完毕!
可以看到容器平台podman-4.0.2也一并安装完成。
-
导入镜像文件
podman load -i images/cephv17.2.0.tar
podman load -i images/prometheus-v2.33.4.tar
podman load -i images/node-exporter-v1.3.1.tar
podman load -i images/alertmanager-v0.23.0.tar
podman load -i images/ceph-grafana-8.3.5.tar
podman load -i images/haproxy-2.3.tar
podman load -i images/keepalived.tar
podman load -i images/snmp-notifier-v1.2.1.tar
-
核对查看镜像文件
[root@ceph2 opt]# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/haproxy 2.3 6717330cf869 10 months ago 103 MB
quay.io/ceph/ceph v17.2.0 132aa60d26c9 10 months ago 1.37 GB
quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 11 months ago 571 MB
quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 12 months ago 205 MB
quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 15 months ago 22.3 MB
quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 18 months ago 58.9 MB
docker.io/maxwo/snmp-notifier v1.2.1 7ca9dd8b3f09 18 months ago 14.6 MB
docker.io/arcts/keepalived latest 073e0c3cd1b9 4 years ago 16.2 MB
请确认ceph1至ceph2节点root用户免密ssh登录配置正常,则ceph2节点的基础安装工作即完成,其它节点参照进行。后续部署操作均在ceph1节点进行。