Kubenetes使用Ceph集群的rbd进行数据持久化

Kubenetes使用Ceph集群的rbd进行数据持久化

1. Ceph集群配置

1.1 创建存储池

ceph osd pool create kubernetes

1.2 初始化rbd存储池

rbd pool init kubernetes

1.3 创建访问密钥

ceph auth get-or-create client.kubernetes mon ‘profile rbd’ osd ‘profile rbd pool=kubernetes’ mgr ‘profile rbd pool=kubernetes’
[client.kubernetes]
key = AQAMYQthBwXJFRAASOYVG7gzvPoB7ZtrDPX4Vg==

1.4 获取访问信息

ceph mon dump
<…>
fsid 00853702-edc0-11eb-b8c2-fa163e9fb3ad
<…>
0: [v2:10.30.101.201:3300/0,v1:10.30.101.201:6789/0] mon.UST-ceph01
1: [v2:10.30.101.202:3300/0,v1:10.30.101.202:6789/0] mon.UST-ceph02
2: [v2:10.30.101.203:3300/0,v1:10.30.101.203:6789/0] mon.UST-ceph03

在配置kubernetes访问ceph时需要用这些信息

2. k8s部署使用rbd

2.1 ceph集群配置

创建部署文件

cat < csi-config-map.yaml

apiVersion: v1
kind: ConfigMap
data:
config.json: |-
[
{
“clusterID”: “00853702-edc0-11eb-b8c2-fa163e9fb3ad”,
“monitors”: [
“10.30.101.201:6789”,
“10.30.101.202:6789”,
“10.30.101.203:6789”
]
}
]
metadata:
name: ceph-csi-config
EOF

应用配置

kubectl apply -f csi-config-map.yaml

2.2 kms配置

创建部署文件

cat < csi-kms-config-map.yaml

apiVersion: v1
kind: ConfigMap
data:
config.json: |-
{}
metadata:
name: ceph-csi-encryption-kms-config
EOF

应用配置

kubectl apply -f csi-kms-config-map.yaml

2.3 配置密钥

创建部署文件

cat < csi-rbd-secret.yaml

apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: default
stringData:
userID: kubernetes
userKey: AQAMYQthBwXJFRAASOYVG7gzvPoB7ZtrDPX4Vg==
EOF

应用配置

kubectl apply -f csi-rbd-secret.yaml

2.3 依赖服务

kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

注：也可以将官方部署文件先下载到本地，修改配置后再部署应用。

2.4 创建StorageClass

创建部署文件

cat < csi-rbd-sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
clusterID: b9127830-b0cc-4e34-aa47-9d1a2e9949a8
pool: kubernetes
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:

• discard
  EOF

部署

kubectl apply -f csi-rbd-sc.yaml

2.5 应用rbd创建服务

创建PVC

cat < pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 1Gi
storageClassName: csi-rbd-sc
EOF
kubectl apply -f pvc.yaml

应用PVC

cat < pod.yaml

apiVersion: v1
kind: Pod
metadata:
name: csi-rbd-demo-pod
spec:
containers:
- name: web-server
image: nginx
volumeMounts:
- name: mypvc
mountPath: /var/lib/www/html
volumes:
- name: mypvc
persistentVolumeClaim:
claimName: rbd-pvc
readOnly: false
EOF
kubectl apply -f pod.yaml