k8s的ceph

fengge55

已于 2022-03-30 11:51:53 修改

阅读量4.5k

点赞数 18

分类专栏： k8s 文章标签： linux

于 2022-03-30 11:40:09 首次发布

本文链接：https://blog.csdn.net/fengge55/article/details/123838961

版权

k8s 专栏收录该内容

9 篇文章 0 订阅

订阅专栏

ceph安装
地址：https://rook.io/docs/rook/v1.8/quickstart.html

特性丰富
1，支持三种存储接口：块存储、文件存储、对象存储。
2，支持自定义接口，支持多种语言驱动。

基本概念
Ceph OSD
Object Storage Device是ceph的核心组件，用于存储数据，处理数据的复制、恢复、回填、再均衡，并通过检查其他OSD守护进程的心跳来向Ceph Monitors提供一些监控信息。ceph集群每台机器上的每块盘需要运行一个OSD进程。

每个OSD有自己日志，生产环境一般要求OSD的journal日志单独放在ssd上，数据对象放在机械硬盘，以提升性能。而OSD使用日志的原因有二：速度和一致性。
1，速度：日志使得OSD可以快速地提交小块数据的写入，Ceph把小片、随机IO依次写入日志，这样，后端文件系统就有可能归并写入动作，并最终提升并发承载力。因此，使用OSD日志能展现出优秀的突发写性能，实际上数据还没有写入OSD，因为文件系统把它们捕捉到了日志。
2，一致性： Ceph的OSD守护进程需要一个能保证原子操作的文件系统接口。OSD把一个操作的描述写入日志，然后把操作应用到文件系统，这需要原子更新一个对象（例如归置组元数据）。每隔一段 filestore max sync interval 和 filestore min sync interval之间的时间，OSD停止写入、把日志同步到文件系统，这样允许 OSD 修整日志里的操作并重用空间。若失败， OSD 从上个同步点开始重放日志。

Monitor
Ceph Monitor维护着展示集群状态的各种图表，包括监视器图、 OSD图、归置组（ PG ）图、和 CRUSH 图。 Ceph 保存着发生在Monitors、OSD和PG上的每一次状态变更的历史信息（称为epoch ）。Monitor支持高可用部署，可以运行多个Monitor组成一个集群，一个监视器集群确保了当某个监视器失效时的高可用性。存储集群客户端向Ceph监视器索取集群运行图的最新副本，而集群通过Paxos算法就集群当前状态保持一致。

MDS
Ceph Metadata Server为Ceph文件系统存储元数据（也就是说，Ceph 块设备和 Ceph 对象存储不使用MDS ）。Metadata Server使得 POSIX 文件系统的用户们，可以在不对 Ceph 存储集群造成负担的前提下，执行诸如 ls、find 等基本命令。

CRUSH
CRUSH 是 Ceph 使用的数据分布算法，类似一致性哈希，让数据分配到预期的地方。Ceph 客户端和 OSD 守护进程都用 CRUSH 算法来计算对象的位置信息，而不是依赖于一个中心化的查询表。与以往方法相比， CRUSH 的数据管理机制更好，它很干脆地把工作分配给集群内的所有客户端和 OSD 来处理，因此具有极大的伸缩性。 CRUSH 用智能数据复制确保弹性，更能适应超大规模存储。

Object
Ceph 最底层的存储单元是 Object 对象，每个 Object 包含元数据和原始数据。

PG
PG 全称 Placement Grouops，是一个逻辑的概念，一个 PG 包含多个Object。引入PG这一层其实是为了更好的分配数据和定位数据。

存储pool
Ceph 存储系统支持“池”概念，它是存储对象的逻辑分区，集群部署起来之后会有一个rdb的默认存储池。一个存储池可以设置PG的数量，CRUSH规则集，访问控制等。

RADOS
RADOS 全称 Reliable Autonomic Distributed Object Store，是 Ceph 集群的核心，用户实现数据分配、Failover 等集群操作。

Libradio
Librados 是 RADOS 的接口库，因为 RADOS 是协议很难直接访问，因此上层的 RBD、RGW 和 CephFS 都是通过 librados 访问的。Ceph的客户端通过一套名为librados的接口进行集群的访问，这里的访问包括对集群的整体访问和对象的访问两类接口。这套接口（API）包括C、C++和Python常见语言的实现，接口通过网络实现对Ceph集群的访问。在用户层面，可以在自己的程序中调用该接口，从而集成Ceph集群的存储功能，或者在监控程序中实现对Ceph集群状态的监控。

RBD
RBD 全称 RADOS block device，是 Ceph 对外提供的块设备服务。Ceph块设备是精简配置的、大小可调且将数据条带化存储到集群内的多个 OSD 。 Ceph 块设备利用 RADOS 的多种能力，如快照、复制和一致性。 Ceph 的 RADOS 块设备（ RBD ）使用内核模块或 librbd 库与 OSD 交互。

RGW
RGW 全称 RADOS gateway，是 Ceph 对外提供的对象存储服务，接口与 S3 和 Swift 兼容。

CephFS
CephFS 全称 Ceph File System，是Ceph对外提供的文件系统服务。

一：k8s集群节点都可以新加一块没有格式化的磁盘，三个节点都新加一块磁盘20G。只是测试而已。
# 扫描 SCSI总线并添加 SCSI 设备
for host in $(ls /sys/class/scsi_host) ; do echo "- - -">/sys/class/scsi_host/$host/scan; done

# 查看已添加的磁盘 sdb
[root@master01 ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 20G 0 disk
sr0 11:0 1 4.5G 0 rom
sda 8:0 0 100G 0 disk
├─sda2 8:2 0 99G 0 part
│ ├─centos-swap 253:1 0 2G 0 lvm
│ ├─centos-home 253:2 0 47G 0 lvm /home
│ └─centos-root 253:0 0 50G 0 lvm /
└─sda1 8:1 0 1G 0 part /boot

1：新磁盘清零，防止磁盘有数据导致pvc创建有问题
[root@master01 kubernetes]# dd if=/dev/zero of=/dev/sdb bs=1M status=progress

2，下载rook
https://github.com/rook/rook/tree/v1.5.5

3.解压，构建
[root@master01 ~]# cd rook/
[root@master01 rook]# unzip rook-1.5.5.zip
[root@master01 rook]# cd rook-1.5.5/cluster/examples/kubernetes/ceph/

#所有节点加载rdb模块
[root@master01 ~]# modprobe rbd
[root@master01 ~]# cat > /etc/sysconfig/modules/rbd.modules << EOF
modprobe rbd
EOF

#加载模块
[root@master01 ~]# chmod 755 /etc/sysconfig/modules/rbd.modules
[root@master01 ~]# source /etc/sysconfig/modules/rbd.modules
[root@master01 ~]# source /etc/sysconfig/modules/rbd.modules
[root@master01 ~]# lsmod |grep rbd

#部署ceph集群服务osd，mon，mgr节点打上标签
[root@master01 ceph]# kubectl label nodes {master01,node01,node02} ceph-osd=enabled
[root@master01 ceph]# kubectl label nodes {master01,node01,node02} ceph-mon=enabled
[root@master01 ceph]# kubectl label nodes {node01,node02} ceph-mgr=enabled #只有一个mgr使用，但是就怕node01挂了，mgr的pod无法应用到k8s节点。

#创建rbac，serviceaccount，clusterrole，clusterrolebinding等资源
[root@master01 ceph]# kubectl apply -f common.yaml

#查看
[root@master01 ceph]# kubectl get sa -n rook-ceph

[root@master01 ceph]# kubectl get clusterrole -n rook-ceph

[root@master01 ceph]# kubectl get clusterrolebinding -n rook-ceph

[root@master01 ceph]# kubectl get role -n rook-ceph

#创建自定义扩展资源对象
[root@master01 ceph]# kubectl apply -f crds.yaml

#镜像下载脚本.推送到私有仓库harbor.od.com
[root@master01 ceph]# vi ceph-images-download.sh
#/bin/bash

#下面镜像要根据ceph的operator.yaml文件要求
image_list=(
ceph:v15.2.8
rook-ceph:v1.5.5
cephcsi:v3.2.0
csi-node-driver-registrar:v2.0.1
csi-resizer:v1.0.0
csi-provisioner:v2.0.0
csi-snapshotter:v3.0.0
csi-attacher:v3.0.0
)

aliyuncs="registry.aliyuncs.com/it00021hot"
google_gcr="k8s.gcr.io/sig-storage"
harbor="harbor.od.com/ceph"
for image in ${image_list[*]}
do
docker image pull ${aliyuncs}/${image}

#下面这行下载镜像后，不需要修改operator.yaml文件
# docker image tag ${aliyuncs}/${image} ${google_gcr}/${image}

docker image tag ${aliyuncs}/${image} ${harbor}/${image}
docker image rm ${aliyuncs}/${image}
docker push ${harbor}/${image}
echo "${aliyuncs}/${image} ${google_gcr}/${image} downloaded"
done

#下载镜像
[root@yunwei ~]# sh ceph-images-download.sh

#修改operator.yaml文件，主要是下面两个地方。
[root@master01 ceph]# vi operator.yaml
spec:
serviceAccountName: rook-ceph-system
containers:
- name: rook-ceph-operator
image: harbor.od.com/ceph/rook-ceph:v1.5.5 #这里也要修改
args: ["ceph", "operator"]
volumeMounts:
- mountPath: /var/lib/rook
name: rook-config
- mountPath: /etc/ceph
name: default-config-dir
.............................................................................
...............................................................................
# ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v2.0.0"
# ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.0"
# ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.0.0"

#下面添加内容，要注意缩进
ROOK_CSI_CEPH_IMAGE: "harbor.od.com/ceph/cephcsi:v3.2.0"
ROOK_CSI_REGISTRAR_IMAGE: "harbor.od.com/ceph/csi-node-driver-registrar:v2.0.1"
ROOK_CSI_RESIZER_IMAGE: "harbor.od.com/ceph/csi-resizer:v1.0.0"
ROOK_CSI_PROVISIONER_IMAGE: "harbor.od.com/ceph/csi-provisioner:v2.0.0"
ROOK_CSI_SNAPSHOTTER_IMAGE: "harbor.od.com/ceph/csi-snapshotter:v3.0.0"
ROOK_CSI_ATTACHER_IMAGE: "harbor.od.com/ceph/csi-attacher:v3.0.0"

#创建operator.yaml
[root@master01 ceph]# kubectl apply -f operator.yaml
configmap/rook-ceph-operator-config created
deployment.apps/rook-ceph-operator created

[root@master01 ceph]# kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-64cbcf46cb-5hd9f 1/1 Running 0 12s

[root@master01 ceph]# kubectl get deployment -n rook-ceph
NAME READY UP-TO-DATE AVAILABLE AGE
rook-ceph-operator 1/1 1 1 27s

#查看operator容器
[root@master01 ceph]# kubectl get pods -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-64cbcf46cb-5hd9f 1/1 Running 0 13m

#查看operator容器日志，可以查看ceph初始化。
[root@master01 ceph]# kubectl logs -f rook-ceph-operator-64cbcf46cb-5hd9f -n rook-ceph

#由于只有三个节点，所以需要打标签。在cluster.yaml配置亲和性调到pod
[root@master01 ceph]# kubectl label nodes {master01,node02,node01} ceph-mon=enabled

[root@master01 ceph]# kubectl label nodes {master01,node02,node01} ceph-osd=enabled

[root@master01 ceph]# kubectl label nodes master01 ceph-mgr=enabled

#创建ceph的集群,下面是主要配置信息
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
# 命名空间的名字，同一个命名空间只支持一个集群
name: rook-ceph
namespace: rook-ceph
spec:
# ceph版本说明
# v13 is mimic, v14 is nautilus, and v15 is octopus.
cephVersion:
#修改ceph镜像，加速部署时间
image: harbor.foxchan.com/google_containers/ceph/ceph:v15.2.5
# 是否允许不支持的ceph版本
allowUnsupported: false
#指定rook数据在节点的保存路径
dataDirHostPath: /data/rook
# 升级时如果检查失败是否继续
skipUpgradeChecks: false
# 从1.5开始，mon的数量必须是奇数
mon:
count: 3
# 是否允许在单个节点上部署多个mon pod
allowMultiplePerNode: false
mgr:
modules:
- name: pg_autoscaler
enabled: true
# 开启dashboard，禁用ssl，指定端口是7000，你可以默认https配置。我是为了ingress配置省事。
dashboard:
enabled: true
port: 7000
ssl: false
# 开启prometheusRule
monitoring:
enabled: true
# 部署PrometheusRule的命名空间，默认此CR所在命名空间
rulesNamespace: rook-ceph
# 开启网络为host模式，解决无法使用cephfs pvc的bug
network:
provider: host
# 开启crash collector，每个运行了Ceph守护进程的节点上创建crash collector pod
crashCollector:
disable: false
# 设置node亲缘性，指定节点安装对应组件
placement:
mon:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-mon
operator: In
values:
- enabled

osd:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-osd
operator: In
values:
- enabled

mgr:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-mgr
operator: In
values:
- enabled
# 存储的设置，默认都是true，意思是会把集群所有node的设备清空初始化。
storage: # cluster level storage configuration and selection
useAllNodes: false #关闭使用所有Node
useAllDevices: false #关闭使用所有设备
nodes:
- name: "192.168.1.162" #指定存储节点主机
devices:
- name: "nvme0n1p1" #指定磁盘为nvme0n1p1
- name: "192.168.1.163"
devices:
- name: "nvme0n1p1"
- name: "192.168.1.164"
devices:
- name: "nvme0n1p1"
- name: "192.168.1.213"
devices:
- name: "nvme0n1p1"

#修改cluster.yaml配置文件,注意缩进
[root@master01 ceph]# cat cluster.yaml|grep -v '#'
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
spec:
cephVersion:
image: ceph/ceph:v15.2.8 #镜像
allowUnsupported: false
dataDirHostPath: /var/lib/rook #rook集群的配置数据存在主机上路径
skipUpgradeChecks: false
continueUpgradeAfterChecksEvenIfNotHealthy: false
mon:
count: 3
allowMultiplePerNode: false
mgr:
modules:
- name: pg_autoscaler
enabled: true
dashboard:
enabled: true
ssl: true
monitoring:
enabled: false
rulesNamespace: rook-ceph
network:
crashCollector:
disable: false
cleanupPolicy:
confirmation: ""
sanitizeDisks:
method: quick
dataSource: zero
iteration: 1
allowUninstallWithVolumes: false
placement:
mon:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-mon
operator: In
values:
- enabled
tolerations: ##容忍master污点，可以部署pod
- key: node-role.kubernetes.io/master
effect: NoSchedule
- key: ceph-mon #部署节点有标签为 ceph-mon节点上
operator: Exists

osd:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-osd
operator: In
values:
- enabled
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
- key: ceph-osd
operator: Exists

mgr:
nodeAffinity: #配置节点亲和度
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: ceph-mgr
operator: In
values:
- enabled
tolerations: #容忍master污点，可以部署pod
- key: node-role.kubernetes.io/master
effect: NoSchedule
- key: ceph-mgr
operator: Exists

annotations:
labels:
resources:
removeOSDsIfOutAndSafeToRemove: false
storage:
useAllNodes: false #关闭使用所有Node
useAllDevices: false #关闭使用所有设备
config:
   osdsPerDevice: "1"
nodes:
   - name: "master01" #指定节点主机，这个名称是根据节点标签labels的 kubernetes.io/hostname=master01
config:
storeType: bluestore #指定类型为裸磁盘
devices: #指定磁盘为sdb
- name: "sdb"
- name: "node01"
config:
storeType: bluestore #指定类型为裸磁盘
devices:
- name: "sdb" #指定磁盘名称
- name: "node02"
config:
storeType: bluestore
devices:
- name: "sdb"

disruptionManagement:
managePodBudgets: false
osdMaintenanceTimeout: 30
pgHealthCheckTimeout: 0
manageMachineDisruptionBudgets: false
machineDisruptionBudgetNamespace: openshift-machine-api

healthCheck:
daemonHealth:
mon:
disabled: false
interval: 45s
osd:
disabled: false
interval: 60s
status:
disabled: false
interval: 60s
livenessProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false


#创建ceph的集群
[root@master01 ceph]# kubectl apply -f cluster.yaml

#csi-rbdplugin，csi-cephfsplugin 是daemonset控制器，每个节点都要部署，但是他们都只有两个pod，master节点无法部署，需要下面修改配置，部署到master节点。
[root@master01 ceph]# kubectl get pods -n rook-ceph -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
csi-cephfsplugin-8v7hs 3/3 Running 0 40m 10.4.7.51 node02 <none> <none>
csi-cephfsplugin-cfq78 3/3 Running 0 40m 10.4.7.50 node01 <none> <none>
csi-cephfsplugin-provisioner-745fcb7868-mqrhd 6/6 Running 0 40m 10.244.2.14 node02 <none> <none>
csi-cephfsplugin-provisioner-745fcb7868-x8sww 6/6 Running 0 40m 10.244.1.11 node01 <none> <none>
csi-rbdplugin-bwjgr 3/3 Running 0 41m 10.4.7.51 node02 <none> <none>
csi-rbdplugin-gg28f 3/3 Running 0 41m 10.4.7.50 node01 <none> <none>
csi-rbdplugin-provisioner-7fdb4675dc-7m7gf 6/6 Running 0 41m 10.244.2.13 node02 <none> <none>
csi-rbdplugin-provisioner-7fdb4675dc-krt8d 6/6 Running 0 41m 10.244.1.10 node01 <none> <none>
rook-ceph-crashcollector-master01-5f7dbf46fc-lfzkf 1/1 Running 0 9m37s 10.244.0.18 master01 <none> <none>
rook-ceph-crashcollector-node01-7ffdfd64c8-2cksj 1/1 Running 0 11m 10.244.1.27 node01 <none> <none>
rook-ceph-crashcollector-node02-8889897f4-x6lcp 1/1 Running 0 11m 10.244.2.26 node02 <none> <none>
rook-ceph-mgr-a-5dcb79d55d-zzpms 1/1 Running 0 12m 10.244.1.23 node01 <none> <none>
rook-ceph-mon-a-68fb45ddb-9ltqt 1/1 Running 0 13m 10.244.2.22 node02 <none> <none>
rook-ceph-mon-b-749978f45f-nhxqv 1/1 Running 0 13m 10.244.0.15 master01 <none> <none>
rook-ceph-mon-c-59fff7cfc5-tn6gt 1/1 Running 0 13m 10.244.1.21 node01 <none> <none>
rook-ceph-operator-d459696cf-x8zq9 1/1 Running 0 64m 10.244.1.9 node01 <none> <none>
rook-ceph-osd-0-68585fc65-2xvj2 1/1 Running 0 11m 10.244.2.25 node02 <none> <none>
rook-ceph-osd-1-6b65c4f6c4-s2rrq 1/1 Running 0 11m 10.244.1.26 node01 <none> <none>
rook-ceph-osd-2-78788d78b6-vjsgb 1/1 Running 0 9m37s 10.244.0.17 master01 <none> <none>
rook-ceph-osd-prepare-master01-7wm87 0/1 Completed 0 12m 10.244.0.16 master01 <none> <none>
rook-ceph-osd-prepare-node01-mbjbj 0/1 Completed 0 12m 10.244.1.25 node01 <none> <none>
rook-ceph-osd-prepare-node02-gfkpq 0/1 Completed 0 12m 10.244.2.24 node02 <none> <none>

2、检查csi-rbdplugin，csi-cephfsplugin，csi-rbdplugin-provisioner，csi-cephfsplugin-provisioner的调度控制器

[root@master01 ~]# kubectl get deployment,daemonset,statefulset -n rook-ceph | grep csi-rbdplugin
deployment.apps/csi-rbdplugin-provisioner 2/2 2 2 23h
daemonset.apps/csi-rbdplugin 3 3 2 1 2 <none> 23h

[root@master01 ~]# kubectl get deployment,daemonset,statefulset -n rook-ceph | grep csi-cephfsplugin
deployment.apps/csi-cephfsplugin-provisioner 2/2 2 2 23h
daemonset.apps/csi-cephfsplugin 3 3 2 3 2 <none> 23h

注解：
检查发现csi-rbdplugin-provisioner，csi-cephfsplugin-provisioner是deployment控制器，
csi-rbdplugin，csi-cephfsplugin 是daemonset控制器，daemonset控制器可以保障每个可调度节点运行一份副本，
做为节点csi驱动daemonset相比deployment更适合一些，那么csi-rbdplugin更有可能是csi驱动管理pod，
那么我们就先将它调度到master节点试试。

1.修改csi-rbdplugin的daemonset配置，可以部署到master节点。
[root@master01 ~]# kubectl edit daemonset csi-rbdplugin -n rook-ceph
。。。。。。。。。。。
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: csi-rbdplugin
template:
metadata:
creationTimestamp: null
labels:
app: csi-rbdplugin
contains: csi-rbdplugin-metrics
spec:
affinity:
nodeAffinity: {}
tolerations: #添三行，pod部署到master
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- args:
- --v=0
- --csi-address=/csi/csi.sock

2.修改csi-cephfsplugin
[root@master01 ~]# kubectl edit daemonset csi-cephfsplugin -n rook-ceph
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
app: csi-cephfsplugin
template:
metadata:
creationTimestamp: null
labels:
app: csi-cephfsplugin
contains: csi-cephfsplugin-metrics
spec:
affinity:
nodeAffinity: {}
tolerations: #添三行，pod部署到master
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- args:
- --v=0
- --csi-address=/csi/csi.sock

#查看修改后pod，csi-rbdplugin，csi-cephfsplugin都是三个pod，
csi-cephfsplugin-provisioner，csi-rbdplugin-provisioner的deployment配置文件定义是2个副本，所以是对的。

[root@master01 template]# kubectl get pods -n rook-ceph -o wide
NAME READY STATUS RESTARTS AGE IP NODE
csi-cephfsplugin-5mll4 3/3 Running 0 25m 10.4.7.50 node01
csi-cephfsplugin-bz6v6 3/3 Running 0 25m 10.4.7.49 master01
csi-cephfsplugin-hgx4v 3/3 Running 0 25m 10.4.7.51 node02
csi-cephfsplugin-provisioner-745fcb7868-mqrhd 6/6 Running 37 24h 10.244.2.38 node02
csi-cephfsplugin-provisioner-745fcb7868-x8sww 6/6 Running 49 24h 10.244.1.38 node01
csi-rbdplugin-4mm7q 3/3 Running 0 26m 10.4.7.51 node02
csi-rbdplugin-9bv9k 3/3 Running 0 56m 10.4.7.49 master01
csi-rbdplugin-f5wrx 3/3 Running 0 26m 10.4.7.50 node01
csi-rbdplugin-provisioner-7fdb4675dc-7m7gf 6/6 Running 42 24h 10.244.2.40 node02
csi-rbdplugin-provisioner-7fdb4675dc-krt8d 6/6 Running 43 24h 10.244.1.34 node01
rook-ceph-crashcollector-master01-5f7dbf46fc-lfzkf 1/1 Running 2 23h 10.244.0.29 master01
rook-ceph-crashcollector-node01-7ffdfd64c8-2cksj 1/1 Running 1 23h 10.244.1.39 node01
rook-ceph-crashcollector-node02-8889897f4-x6lcp 1/1 Running 1 23h 10.244.2.43 node02
rook-ceph-mgr-a-5dcb79d55d-zzpms 1/1 Running 1 23h 10.244.1.36 node01
rook-ceph-mon-a-68fb45ddb-9ltqt 1/1 Running 1 23h 10.244.2.39 node02
rook-ceph-mon-b-749978f45f-nhxqv 1/1 Running 2 23h 10.244.0.30 master01
rook-ceph-mon-c-59fff7cfc5-tn6gt 1/1 Running 1 23h 10.244.1.35 node01
rook-ceph-operator-d459696cf-x8zq9 1/1 Running 1 24h 10.244.1.40 node01
rook-ceph-osd-0-68585fc65-2xvj2 1/1 Running 1 23h 10.244.2.42 node02
rook-ceph-osd-1-6b65c4f6c4-s2rrq 1/1 Running 1 23h 10.244.1.37 node01
rook-ceph-osd-2-78788d78b6-vjsgb 1/1 Running 3 23h 10.244.0.34 master01
rook-ceph-osd-prepare-master01-kl2gp 0/1 Completed 0 101m 10.244.0.36 master01
rook-ceph-osd-prepare-node01-mxnn6 0/1 Completed 0 101m 10.244.1.41 node01
rook-ceph-osd-prepare-node02-4sm6n 0/1 Completed 0 101m 10.244.2.44 node02

#查看主机上的磁盘是否被调用，如果没有，就要查看下cluster.yaml配置
[root@master01 static]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sdb LVM2_member 3PQAnX-rehr-fxu0-bIMJ-fYWS-Zcaz-ublUvk
└─ceph--4ce439e6--f251--4fc3--9e75--375ed61d6d8b-osd--block--4e99fb5a--9c32--482d--897f--6b689f60c946

#查看查看rook-ceph-operatorpod的log
[root@master01 ceph]# kubectl logs -f rook-ceph-operator-d459696cf-x8zq9 -n rook-ceph

#查看osd
[root@master01 ceph]# kubectl logs -f rook-ceph-osd-prepare-master01-7wm87 -n rook-ceph provision

#查看ceph集群的svc，其中rook-ceph-mgr-dashboard服务是后面配置ingress，外面域名可以访问的。
[root@master01 ceph]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-cephfsplugin-metrics ClusterIP 10.103.24.98 <none> 8080/TCP,8081/TCP 93m
csi-rbdplugin-metrics ClusterIP 10.108.163.76 <none> 8080/TCP,8081/TCP 93m
rook-ceph-mgr ClusterIP 10.97.146.102 <none> 9283/TCP 90m
rook-ceph-mgr-dashboard ClusterIP 10.106.30.224 <none> 8443/TCP 90m
rook-ceph-mon-a ClusterIP 10.107.214.89 <none> 6789/TCP,3300/TCP 93m
rook-ceph-mon-b ClusterIP 10.98.9.202 <none> 6789/TCP,3300/TCP 92m
rook-ceph-mon-c ClusterIP 10.97.87.235 <none> 6789/TCP,3300/TCP 91m

11、安装Toolbox
toolbox是一个rook的工具集容器，该容器中的命令可以用来调试、测试Rook，对Ceph临时测试的操作一般在这个容器内执行。
[root@master01 ceph]# kubectl apply -f toolbox.yaml

#测试 Toolbox 是否生成
[root@master01 ceph]# kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME READY STATUS RESTARTS AGE
rook-ceph-tools-58d7dbc69f-d5kz4 1/1 Running 0 6m41s

#进入rook-tools 容器
[root@master01 ceph]# kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
sh-4.4#
sh-4.4# ceph status #查看ceph集群状态
cluster:
id: 4dc66671-6fe6-4dee-9a09-cdb24bc481e9
health: HEALTH_OK

services:
mon: 3 daemons, quorum a,b,c (age 72m)
mgr: a(active, since 71m)
osd: 3 osds: 3 up (since 69m), 3 in (since 73m)

data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 57 GiB / 60 GiB avail
pgs: 1 active+clean

sh-4.4#
sh-4.4# ceph osd status #查看osd状态
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node02 1030M 18.9G 0 0 0 0 exists,up
1 node01 1030M 18.9G 0 0 0 0 exists,up
2 master01 1030M 18.9G 0 0 0 0 exists,up
sh-4.4#
sh-4.4# rados df # 查看ceph集群的存储情况
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
device_health_metrics 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B

total_objects 0
total_used 3.0 GiB
total_avail 57 GiB
total_space 60 GiB

sh-4.4# ceph auth ls #ceph集群各个服务的认证信息
installed auth entries:

osd.0
key: AQDML0FikQLhCRAAUw9YbHN0X22HktxtC4k0lA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQDTL0FiQ1WQNRAAdo77BUT8aV8KjIHz3SCPhA==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.2
key: AQC7MkFihCAfBxAAlD/ik5g9Xm1dZNgvpDCaIQ==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQALL0Fi8fnyIhAAIrUfMZiAcJ3Y2QNhozEAbA==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQAuL0FiXA+SBxAA2O4l5/BkZyXsFSRou4iF5A==
caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
key: AQAuL0FitCqSBxAAUlOJxEngZisvyN0pL3dFiQ==
caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
key: AQAuL0FiCUCSBxAAsOu8GxJH6/8Ea+7EAzMRDA==
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rbd
key: AQAuL0FiN1eSBxAAcPauOXaUXogYXx3dgKwNKw==
caps: [mon] allow profile bootstrap-rbd
client.bootstrap-rbd-mirror
key: AQAuL0FioGySBxAAaApeNBU/JgyLnHyYjtPGhQ==
caps: [mon] allow profile bootstrap-rbd-mirror
client.bootstrap-rgw
key: AQAuL0FixIGSBxAAWhnt7ozxhd/DyDsLU+4Alw==
caps: [mon] allow profile bootstrap-rgw
client.crash
key: AQCnL0Fi+43UDxAAKkcYOsbycCtXAvxqmGPIqA==
caps: [mgr] allow profile crash
caps: [mon] allow profile crash
client.csi-cephfs-node
key: AQCmL0Fiw5W5LhAAMINFmCHXKmofF7Cf50SpdQ==
caps: [mds] allow rw
caps: [mgr] allow rw
caps: [mon] allow r
caps: [osd] allow rw tag cephfs *=*
client.csi-cephfs-provisioner
key: AQCmL0FiJCvXEhAA9a3JKSNZRsTamAq37CnDBw==
caps: [mgr] allow rw
caps: [mon] allow r
caps: [osd] allow rw tag cephfs metadata=*
client.csi-rbd-node
key: AQClL0Fic1evMBAAksit07nE0srJCj4jTgR+bg==
caps: [mgr] allow rw
caps: [mon] profile rbd
caps: [osd] profile rbd
client.csi-rbd-provisioner
key: AQClL0FiUKncGRAA4C4iw9FoNgZPVHq0ygrltw==
caps: [mgr] allow rw
caps: [mon] profile rbd
caps: [osd] profile rbd
mgr.a
key: AQCoL0FiRTfqDhAA5YtsWGFA/AJHuV4Cdw857w==
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *

12.部署ceph的dashboard

#创建自签tls公私秘钥对
[root@master01 ceph]# mkdir tls
[root@master01 ceph]# cd tls/
[root@master01 tls]# openssl genrsa -out tls.key 2048
[root@master01 tls]# openssl req -new -x509 -days 3650 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/O=DevOps/CN=*.od.com
[root@master01 tls]# ll
total 8
-rw-r--r-- 1 root root 1220 Mar 28 14:37 tls.crt
-rw-r--r-- 1 root root 1679 Mar 28 14:37 tls.key

#创建ceph的dashboard的ingress的secret，由于ceph在rook-ceph名称空间里，所以创建secret也要在这个名称空间里。
[root@master01 tls]# kubectl create secret tls ceph-ingress-secret --cert=tls.crt --key=tls.key -n rook-ceph
secret/ceph-ingress-secret created

#修改ceph的dashboard文件
[root@master01 tls]# cd ..
[root@master01 ceph]# cat dashboard-ingress-https.yaml
#
# This example is for Kubernetes running an ngnix-ingress
# and an ACME (e.g. Let's Encrypt) certificate service
#
# The nginx-ingress annotations support the dashboard
# running using HTTPS with a self-signed certificate
#
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: rook-ceph-mgr-ingress #定义ceph的ingress名字，
namespace: rook-ceph # namespace:cluster，名称空间
annotations: #注解，一定要有，否则会有问题
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/server-snippet: |
proxy_ssl_verify off;
spec:
tls:
- hosts:
- rook-ceph.od.com #自签tls的CN，建议一致跟下面的rules的host名字
secretName: ceph-ingress-secret #定义上面对od.com泛域名创建的secret的名字
rules:
- host: rook-ceph.od.com #定义访问的域名
http:
paths:
- path: /
backend:
serviceName: rook-ceph-mgr-dashboard #这是mgr的dashboard服务名字，前面在创建ceph集群的时候就已经创建好了。
servicePort: 8443 #rook-ceph-mgr-dashboard服务端口

#创建ceph的dashboard的ingress
[root@master01 ceph]# kubectl apply -f dashboard-ingress-https.yaml

[root@master01 ceph]# kubectl get ingress -n rook-ceph
NAME CLASS HOSTS ADDRESS PORTS AGE
rook-ceph-mgr-ingress <none> rook-ceph.od.com 80, 443 29m

[root@master01 ceph]# kubectl describe ingress rook-ceph-mgr-ingress -n rook-ceph
Name: rook-ceph-mgr-ingress
Namespace: rook-ceph
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
ceph-ingress-secret terminates rook-ceph.od.com
Rules:
Host Path Backends
---- ---- --------
rook-ceph.od.com
/ rook-ceph-mgr-dashboard:8443 (10.244.1.23:8443)
Annotations: kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: true
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/server-snippet: proxy_ssl_verify off;
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 29m nginx-ingress-controller Ingress rook-ceph/rook-ceph-mgr-ingress

#配置dns，#由于ingress-collector服务是hostnetwork，而且通过标签部署pod在master01，
如果有三台master，可以指定vip，高可用

[root@master01 tls]# vi /var/named/od.com.zone
$ORIGIN od.com.
$TTL 600 ; 10 minutes
@ IN SOA dns.od.com. dnsadmin.od.com. (
2020011201 ; serial
10800 ; refresh (3 hours)
900 ; retry (15 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS dns.od.com.
$TTL 60 ; 1 minute
dns A 10.4.7.49
harbor A 10.4.7.53
demo A 10.4.7.49
rook-ceph A 10.4.7.49

#加载一下named服务
[root@master01 tls]# service named restart

#另外一种方式就直接修改 rook-ceph-mgr-dashboard 的service，
[root@master01 ceph]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-cephfsplugin-metrics ClusterIP 10.100.248.219 <none> 8080/TCP,8081/TCP 58m
csi-rbdplugin-metrics ClusterIP 10.109.65.11 <none> 8080/TCP,8081/TCP 58m
rook-ceph-mgr ClusterIP 10.100.72.151 <none> 9283/TCP 53m
rook-ceph-mgr-dashboard ClusterIP 10.111.104.197 <none> 8443/TCP 53m
rook-ceph-mon-a ClusterIP 10.101.3.133 <none> 6789/TCP,3300/TCP 58m
rook-ceph-mon-b ClusterIP 10.111.38.134 <none> 6789/TCP,3300/TCP 55m
rook-ceph-mon-c ClusterIP 10.104.208.191 <none> 6789/TCP,3300/TCP 54m

#修改type类型，默认是ClusterIp，改为NodePort
[root@master01 tls]# kubectl edit svc rook-ceph-mgr-dashboard -n rook-ceph

#查看rook-ceph-mgr-dashboard 的service服务type为NodePort
[root@master01 tls]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-cephfsplugin-metrics ClusterIP 10.100.248.219 <none> 8080/TCP,8081/TCP 64m
csi-rbdplugin-metrics ClusterIP 10.109.65.11 <none> 8080/TCP,8081/TCP 64m
rook-ceph-mgr ClusterIP 10.100.72.151 <none> 9283/TCP 59m
rook-ceph-mgr-dashboard NodePort 10.111.104.197 <none> 8443:31004/TCP 59m
rook-ceph-mon-a ClusterIP 10.101.3.133 <none> 6789/TCP,3300/TCP 63m
rook-ceph-mon-b ClusterIP 10.111.38.134 <none> 6789/TCP,3300/TCP 60m
rook-ceph-mon-c ClusterIP 10.104.208.191 <none> 6789/TCP,3300/TCP 59m

#浏览器直接用node节点ip:31004访问

15、查看登录密码
[root@master01 ceph]# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 --decode && echo
8ynOK:Wb${/)/;0SY[9(

#修改密码为：admin123，但是重启后，密码又变了

16.浏览器登录 https://rook-ceph.od.com
用户：admin
密码：8ynOK:Wb${/)/;0SY[9(

#卸载ceph集群
1.删除之前执行的yaml文件
[root@master01 ceph]# kubectl delete -f crds.yaml -f common.yaml -f operator.yaml -f cluster.yaml

2.删除ceph集群的data目录
[root@master01 ceph]# rm /var/lib/rook/ -rf

3：查看资源
[root@master01 tmp]# kubectl -n rook-ceph get cephcluster

4.删除rook-ceph名称空间
[root@master01 ceph]# kubectl get ns rook-ceph -o json >/tmp/tmp.json
[root@master01 ceph]# kubectl delete namespace rook-ceph

5.删除名称空间rook-ceph时，出现Terminating，开启临时一个服务
[root@master01 ~]# kubectl proxy --port=9098
Starting to serve on 127.0.0.1:9098

6.开启另外一个终端，删除tmp.json文件里的spec内容
[root@master01 ceph]# cd /tmp/
[root@master01 tmp]# vi tmp.json
[root@master01 tmp]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:9098/api/v1/namespaces/rook-ceph/finalize

7.再次查看集群状态
[root@master01 tmp]# kubectl -n rook-ceph get cephcluster

8.之前的磁盘清零
[root@master01 ceph]# dd if=/dev/zero of=/dev/sdb bs=1M status=progress

17.ceph配置rdb，一般是配置有状态服务statefulset，
[root@master01 rbd]# cd /root/rook/rook-1.5.5/cluster/examples/kubernetes/ceph/csi/rbd
[root@master01 rbd]# cat storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool #块地址池
metadata:
name: replicapool #定义块地址池名称
namespace: rook-ceph #在rook-ceph名称空间创建块存储池。
spec:
failureDomain: osd #容灾模式host或者osd，
replicated:
size: 3 #数据副本个数，一个节点挂了，不会丢失数据，另外还有两个副本。
# Disallow setting pool with replica 1, this could lead to data loss without recovery.
# Make sure you're *ABSOLUTELY CERTAIN* that is what you want
requireSafeReplicaSize: true
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#targetSizeRatio: .5
---
apiVersion: storage.k8s.io/v1
kind: StorageClass #储存类型
metadata:
name: rook-ceph-block #创建一个块存储名字，储存是针对集群的，不针对名称空间的
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com #提供rdb块存储的供应商。
parameters:
# clusterID is the namespace where the rook cluster is running
# If you change this namespace, also change the namespace below where the secret namespaces are defined
clusterID: rook-ceph # namespace:cluster

# If you want to use erasure coded pool with RBD, you need to create
# two pools. one erasure coded and one replicated.
# You need to specify the replicated pool here in the `pool` parameter, it is
# used for the metadata of the images.
# The erasure coded pool must be set as the `dataPool` parameter below.
#dataPool: ec-data-pool
pool: replicapool #指定上面创建的块储存池名称

# (optional) mapOptions is a comma-separated list of map options.
# For krbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
# For nbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
# mapOptions: lock_on_read,queue_depth=1024

# (optional) unmapOptions is a comma-separated list of unmap options.
# For krbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd/#kernel-rbd-krbd-options
# For nbd options refer
# https://docs.ceph.com/docs/master/man/8/rbd-nbd/#options
# unmapOptions: force

# RBD image format. Defaults to "2".
imageFormat: "2"

# RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
imageFeatures: layering

# The secrets contain Ceph admin credentials. These are generated automatically by the operator
# in the same namespace as the cluster.
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster
# Specify the filesystem type of the volume. If not specified, csi-provisioner
# will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
# in hyperconverged settings where the volume is mounted on the same node as the osds.
csi.storage.k8s.io/fstype: ext4
# uncomment the following to use rbd-nbd as mounter on supported nodes
# **IMPORTANT**: If you are using rbd-nbd as the mounter, during upgrade you will be hit a ceph-csi
# issue that causes the mount to be disconnected. You will need to follow special upgrade steps
# to restart your application pods. Therefore, this option is not recommended.
#mounter: rbd-nbd
allowVolumeExpansion: true
reclaimPolicy: Delete

#可以默认参数，
[root@master01 rbd]# kubectl apply -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created

#查看创建的块池
[root@master01 rbd]# kubectl get cephblockpool -A
NAMESPACE NAME AGE
rook-ceph replicapool 92m

#查看刚才创建的存储类名称，也叫存储驱动，
[root@master01 rbd]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 27m

#在创建一个rook-cephfs的共享文件存储驱动
[root@master01 ~]# cd /root/rook/rook-1.5.5/cluster/examples/kubernetes/ceph/

#修改容灾模式为osd
[root@master01 ceph]# sed -i 's/failureDomain: host/failureDomain: osd/g' filesystem.yaml

#创建文件系统
[root@master01 ceph]# cat filesystem.yaml
#############################################################################
# Create a filesystem with settings with replication enabled for a production environment.
# A minimum of 3 OSDs on different nodes are required in this example.
# kubectl create -f filesystem.yaml
#############################################################################

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: myfs #创建fs的名字
namespace: rook-ceph # namespace:cluster
spec:
# The metadata pool spec. Must use replication.
metadataPool:
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
# Inline compression mode for the data pool
# Further reference: https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref/#inline-compression
compression_mode: none
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#target_size_ratio: ".5"
# The list of data pool specs. Can use replication or erasure coding.
dataPools:
- failureDomain: osd
replicated:
size: 3
# Disallow setting pool with replica 1, this could lead to data loss without recovery.
# Make sure you're *ABSOLUTELY CERTAIN* that is what you want
requireSafeReplicaSize: true
parameters:
# Inline compression mode for the data pool
# Further reference: https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref/#inline-compression
compression_mode: none
# gives a hint (%) to Ceph in terms of expected consumption of the total cluster capacity of a given pool
# for more info: https://docs.ceph.com/docs/master/rados/operations/placement-groups/#specifying-expected-pool-size
#target_size_ratio: ".5"
# Whether to preserve filesystem after CephFilesystem CRD deletion
preserveFilesystemOnDelete: true
# The metadata service (mds) configuration
metadataServer:
# The number of active MDS instances
activeCount: 1
# Whether each active MDS instance will have an active standby with a warm metadata cache for faster failover.
# If false, standbys will be available, but will not have a warm cache.
activeStandby: true
# The affinity rules to apply to the mds deployment
placement:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - mds-node
# topologySpreadConstraints:
# tolerations:
# - key: mds-node
# operator: Exists
# podAffinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-mds
# topologyKey: kubernetes.io/hostname will place MDS across different hosts
topologyKey: kubernetes.io/hostname
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- rook-ceph-mds
# topologyKey: */zone can be used to spread MDS across different AZ
# Use <topologyKey: failure-domain.beta.kubernetes.io/zone> in k8s cluster if your cluster is v1.16 or lower
# Use <topologyKey: topology.kubernetes.io/zone> in k8s cluster is v1.17 or upper
topologyKey: topology.kubernetes.io/zone
# A key/value list of annotations
annotations:
# key: value
# A key/value list of labels
labels:
# key: value
resources:
# The requests and limits set here, allow the filesystem MDS Pod(s) to use half of one CPU core and 1 gigabyte of memory
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# priorityClassName: my-priority-class

#创建 CephFilesystem
[root@master01 ceph]# kubectl apply -f filesystem.yaml

[root@master01 cephfs]# kubectl get CephFilesystem -n rook-ceph
NAME ACTIVEMDS AGE
myfs 1 72s

#这个时候会运行mds两个pod
[root@master01 ceph]# kubectl -n rook-ceph get pod -l app=rook-ceph-mds
NAME READY STATUS RESTARTS AGE
rook-ceph-mds-myfs-a-6d69895657-bl5cs 1/1 Running 0 4m28s
rook-ceph-mds-myfs-b-b6d546f87-pjm7p 1/1 Running 0 4m25s

#创建cephfs文件共享存储驱动
[root@master01 cephfs]# cd /root/rook/rook-1.5.5/cluster/examples/kubernetes/ceph/csi/cephfs
[root@master01 cephfs]# vi storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com # driver:namespace:operator
parameters:
# clusterID is the namespace where operator is deployed.
clusterID: rook-ceph # namespace:cluster

# CephFS filesystem name into which the volume shall be created
fsName: myfs #要指定已经创建好的CephFilesystem

# Ceph pool into which the volume shall be created
# Required for provisionVolume: "true"
pool: myfs-data0

# Root path of an existing CephFS volume
# Required for provisionVolume: "false"
# rootPath: /absolute/path

# (optional) The driver can use either ceph-fuse (fuse) or ceph kernel client (kernel)
# If omitted, default volume mounter will be used - this is determined by probing for ceph-fuse
# or by setting the default mounter explicitly via --volumemounter command-line argument.
mounter: kernel
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
# uncomment the following line for debugging
#- debug

#创建rook-cephfs存储驱动
[root@master01 cephfs]# kubectl apply -f storageclass.yaml
storageclass.storage.k8s.io/rook-cephfs created

#查看已经创建了两个存储驱动。需要存储可以调用这些驱动，自动创建pvc
[root@master01 cephfs]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 19h
rook-cephfs rook-ceph.cephfs.csi.ceph.com Delete Immediate true 5s

#测试rbd只支持有状态服务，
1.创建nginx的无状态应用
[root@master01 ceph]# cat /root/nginx-dp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-nginx
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.16
ports:
- containerPort: 80
volumeMounts:
- name: localtime
mountPath: /etc/localtime
- name: nginx-html-storage
mountPath: /usr/share/nginx/html
volumes:
- name: localtime
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
- name: nginx-html-storage
persistentVolumeClaim:
claimName: nginx-pv-claim #指定名为nginx-pv-claim的pvc

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-pv-claim #创建名为nginx-pv-claim的pvc
labels:
app: nginx
spec:
storageClassName: rook-ceph-block #指定上面创建存储引擎名为 rook-ceph-block
accessModes:
- ReadWriteMany #访问模式为多节点同时访问
resources:
requests:
storage: 1Gi #容量为1Gi

#创建nginx的应用，
[root@master01 ~]# kubectl apply -f nginx-dp.yaml
deployment.apps/test-nginx created
persistentvolumeclaim/nginx-pv-claim created

#查看pvc，状态为pending，肯定不正常。
[root@master01 ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nginx-pv-claim Pending rook-ceph-block 10s

2,。查看创建的pvc，状态为pending，肯定不正常。
[root@master01 ceph]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nginx-pv-claim Pending rook-ceph-block 4m48s

3.查看pvc，报错了
[root@master01 ~]# kubectl describe pvc nginx-pv-claim

# failed to provision volume with StorageClass "rook-ceph-block": rpc
error: code = InvalidArgument desc = multi node access modes are
only supported on rbd `block` type volumes
意思是说：意思是不推荐在ceph rbd模式下使用RWX访问控制，如果应用层没有访问锁机制，可能会造成数据损坏

#修改定义的pvc的访问模式为ReadWriteOnce
1.修改配置
[root@master01 ceph]# cat /root/nginx-dp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-nginx
labels:
app: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.16
ports:
- containerPort: 80
volumeMounts:
- name: localtime
mountPath: /etc/localtime
- name: nginx-html-storage
mountPath: /usr/share/nginx/html
volumes:
- name: localtime
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
- name: nginx-html-storage
persistentVolumeClaim:
claimName: nginx-pv-claim #指定名为nginx-pv-claim的pvc,或者已经存在的pvc的名字
readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-pv-claim #创建名为nginx-pv-claim的pvc
labels:
app: nginx
spec:
storageClassName: rook-ceph-block #指定上面创建存储引擎名为 rook-ceph-block
accessModes:
- ReadWriteOnce #访问模式为单节点访问
resources:
requests:
storage: 1Gi #容量为1Gi

2.创建pvc，nginx应用并查看pvc跟pod
[root@master01 ~]# kubectl apply -f nginx-dp.yaml
deployment.apps/test-nginx created
persistentvolumeclaim/nginx-pv-claim created

[root@master01 ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nginx-pv-claim Bound pvc-6d2d0264-48fc-4edf-9433-875b0512ec5f 1Gi RWO rook-ceph-block 6s

3.查看pod
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
test-nginx-6f6584b4b-zgksm 1/1 Running 0 61s

# 使用cephfs的存储驱动来存储共享文件cpch-cephFS测试
案例一:多容器共享同一个数据目录,部署多个私有仓库共享同一个数据目录进行测试
[root@master01 cephfs]# cd /root/rook/rook-1.5.5/cluster/examples/kubernetes/ceph/csi/cephfs
[root@master01 cephfs]# ls
kube-registry.yaml pod.yaml pvc-clone.yaml pvc-restore.yaml pvc.yaml rook-cephfs.yaml snapshotclass.yaml snapshot.yaml storageclass.yaml

#创建test的名称空间
[root@master01 cephfs]# kubectl create ns test

#在test名称空间创建2个pod
[root@master01 cephfs]# cat kube-registry.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc #通过下面的cephfs存储驱动为rook-cephfs来创建pvc
namespace: test #pvc是针对名称空间的
spec:
accessModes:
- ReadWriteMany #访问模式为多节点读写访问
resources:
requests:
storage: 1Gi
storageClassName: rook-cephfs #指定存储驱动名为：rook-cephfs，这个存储驱动是cephfs驱动。
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-registry
namespace: test #要挂载test的pvc，所以就需要在test名称空间创建pod，才可以识别
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
spec:
replicas: 2 #用两个副本
selector:
matchLabels:
k8s-app: kube-registry
template:
metadata:
labels:
k8s-app: kube-registry
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2
imagePullPolicy: Always
resources:
limits:
cpu: 100m
memory: 100Mi
env:
# Configuration reference: https://docs.docker.com/registry/configuration/
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_HTTP_SECRET
value: "Ple4seCh4ngeThisN0tAVerySecretV4lue"
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store #这里指定挂载逻辑名字，意思就是把名为：cephfs-pvc的pvc挂载到容器里，
mountPath: /var/lib/registry #挂载到容器这个目录下
ports:
- containerPort: 5000
name: registry
protocol: TCP
livenessProbe:
httpGet:
path: /
port: registry
readinessProbe:
httpGet:
path: /
port: registry
volumes:
- name: image-store #定义挂载逻辑名字
persistentVolumeClaim:
claimName: cephfs-pvc #定义上面创建的pvc
readOnly: false


#创建挂载共享文件存储pod
[root@master01 cephfs]# kubectl apply -f kube-registry.yaml

[root@master01 cephfs]# kubectl get pvc -n test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-pvc Bound pvc-933ab84b-f298-4c42-946d-5560a367caf1 1Gi RWX rook-cephfs 19s

[root@master01 cephfs]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
kube-registry-58659ff99b-bvd52 1/1 Running 0 49s
kube-registry-58659ff99b-gvcbz 1/1 Running 0 49s

#进入其中一个pod。
[root@master01 cephfs]# kubectl exec -it kube-registry-58659ff99b-bvd52 -n test sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ #
/ # cd /var/lib/registry/
/var/lib/registry # echo "abc" > test.txt
/var/lib/registry # cat test.txt
abc

#进入另外一个pod
[root@master01 cephfs]# kubectl exec -it kube-registry-58659ff99b-gvcbz -n test sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
/ #
/ # cat /var/lib/registry/test.txt
abc