Kubernetes集群Rook部署Ceph集群
环境详情
| 主机名 | IP地址 | CPU | 内存 |
|---|---|---|---|
| k8s-master01 | 192.168.46.3 | 4C | 5G |
| k8s-master02 | 192.168.46.4 | 4C | 5G |
| k8s-master03 | 192.168.46.5 | 4C | 5G |
| k8s-node01 | 192.168.46.6 | 4C | 5G |
| k8s-node02 | 192.168.46.7 | 4C | 5G |
| Pod网段 | 172.16.0.0/12 |
|---|---|
| Service网段 | 10.0.0.0/16 |
| VIP地址 | 192.168.46.10/24 |
安装前准备
集群至少需要三个工作节点（由于机器资源有限，所以准备在k8s-master03、k8s-node01和k8s-node02安装）
每个需要安装的节点准备一块裸设备（未分区、未进行格式化）
添加裸设备
所有添加了裸设备的节点重启机器
检查是否添加裸设备成功
lsblk -f
k8s-master03去除污点
由于k8s-master03节点需要安装pod，所以需要去除污点
kubectl taint nodes k8s-master03 node-role.kubernetes.io/control-plane:NoSchedule-
安装Rook
下载源代码
GitHub链接：https://github.com/rook/rook
git clone --single-branch --branch v1.12.7 https://github.com/rook/rook
cd rook/deploy/examples/
修改operator.yaml文件
vim operator.yaml
# 取消CSI镜像地址的注释并求改镜像地址
# ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.9.0"
# ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0"
# ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.8.0"
# ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0"
# ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2"
# ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.3.0"
ROOK_CSI_CEPH_IMAGE: "swr.cn-north-4.myhuaweicloud.com/ctl456/cephcsi:v3.9.0"
ROOK_CSI_REGISTRAR_IMAGE: "swr.cn-north-4.myhuaweicloud.com/ctl456/csi-node-driver-registrar:v2.8.0"
ROOK_CSI_RESIZER_IMAGE: "swr.cn-north-4.myhuaweicloud.com/ctl456/csi-resizer:v1.8.0"
ROOK_CSI_PROVISIONER_IMAGE: "swr.cn-north-4.myhuaweicloud.com/ctl456/csi-provisioner:v3.5.0"
ROOK_CSI_SNAPSHOTTER_IMAGE: "swr.cn-north-4.myhuaweicloud.com/ctl456/csi-snapshotter:v6.2.2"
ROOK_CSI_ATTACHER_IMAGE: "swr.cn-north-4.myhuaweicloud.com/ctl456/csi-attacher:v4.3.0"
# 开启自动发现磁盘
ROOK_ENABLE_DISCOVERY_DAEMON: "true"
# 修改image:地址
image: rook/ceph:v1.12.7
# 改为
image: swr.cn-north-4.myhuaweicloud.com/ctl456/rook-ceph:v1.12.7
部署Rook
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
部署成功结果
kubectl get pod -n rook-ceph -o wide
部署Ceph集群
修改cluster.yaml文件
vim cluster.yaml
# 修改image地址
image: quay.io/ceph/ceph:v17.2.6
# 改为
image: swr.cn-north-4.myhuaweicloud.com/ctl456/ceph:v17.2.6
# 修改useAllNodes和useAllDevices
useAllNodes: false
useAllDevices: false
# 配置nodes
nodes:
- name: "k8s-master03"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "k8s-node01"
devices:
- name: "sdb"
- name: "k8s-node02"
devices:
- name: "sdb"
部署Ceph集群
kubectl create -f cluster.yaml
部署成功结果
kubectl get pod -n rook-ceph
kubectl get cephcluster -n rook-ceph
安装Ceph客户端工具和Dashboard
安装Ceph客户端
修改toolbox.yaml文件
vim toolbox.yaml
# 修改image地址
image: quay.io/ceph/ceph:v17.2.6
# 改为
image: swr.cn-north-4.myhuaweicloud.com/ctl456/ceph:v17.2.6
部署Ceph客户端
kubectl create -f toolbox.yaml
部署成功结果
kubectl get pod -n rook-ceph -l app=rook-ceph-tools
进入容器查看Ceph
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
ceph status
ceph osd status
ceph df
安装Dashboard
默认情况下，Dashboard已经安装
只需要暴露服务即可
创建编辑dashboard-np.yaml
vim dashboard.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: rook-ceph-mgr
ceph_daemon-id: a
rook_cluster: rook-ceph
name: rook-ceph-mgr-dashboard-np
namespace: rook-ceph
spec:
ports:
- name: http-dashboard
port: 8443
protocol: TCP
targetPort: 8443
selector:
app: rook-ceph-mgr
ceph_daemon_id: a
rook_cluster: rook-ceph
sessionAffinity: None
type: NodePort
运行yaml文件
kubectl create -f dashboard.yaml
访问Dashboard
kubectl get svc -n rook-ceph
https://VIP地址:暴露出的port
账号：admin
密码：
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
创建StorageClass和Ceph的存储池
修改storageclass.yaml文件
vim csi/rbd/storageclass.yaml
# 修改size
size: 3
# 改为
size: 2
# 生产环境最少为3，并且要小于等于OSD的数量
运行yaml文件
kubectl create -f csi/rbd/storageclass.yaml -n rook-ceph
查看cephblockpool和StorageClass
kubectl get cephblockpool -n rook-ceph
kubectl get sc
挂载测试
修改mysql.yaml文件
vim mysql.yaml
# 修改image地址
image: mysql:5.6
# 改为
image: swr.cn-north-4.myhuaweicloud.com/ctl456/mysql:5.6
yaml文件有关存储配置
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
containers:
- image: swr.cn-north-4.myhuaweicloud.com/ctl456/mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: changeme
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
运行yaml文件
kubectl create -f mysql.yaml
运行成功结果
kubectl get pod
kubectl get pvc
kubectl get pv
StatefulSet volumeClaimTemplates
创建StatefulSet-nginx.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: swr.cn-north-4.myhuaweicloud.com/ctl456/nginx:latest
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "rook-ceph-block"
resources:
requests:
storage: 1Gi
成功结果
共享型文件系统
创建共享型文件系统
kubectl create -f filesystem.yaml
成功结果
创建共享型文件系统的StorageClass
kubectl create -f csi/cephfs/storageclass.yaml
挂载测试
修改kube-registry.yaml文件
vim csi/cephfs/kube-registry.yaml
# 修改image地址
image: registry:2
# 改为
image: swr.cn-north-4.myhuaweicloud.com/ctl456/registry:2
yaml文件有关存储配置
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc
namespace: kube-system
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: rook-cephfs
containers:
......
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
......
volumes:
- name: image-store
persistentVolumeClaim:
claimName: cephfs-pvc
readOnly: false
运行yaml文件
kubectl create -f csi/cephfs/kube-registry.yaml
成功结果
PVC扩容
扩容文件共享型PVC（需要等待，有延迟）
kubectl edit pvc NAME -n 命名空间名字
# 例子
kubectl edit pvc cephfs-pvc -n kube-system
# 修改storage的值
storage: 1Gi
# 改为
storage: 2Gi
成功结果
扩容快存储（需要等待，有延迟）
kubectl edit pvc mysql-pv-claim
# 扩容10G
storage: 20Gi
# 改为
storage: 30Gi
成功结果
PVC快照
安装Snapshot控制器
下载源代码
GitHub链接：https://github.com/kubernetes-csi/external-snapshotter
git clone --single-branch --branch v6.3.1 https://github.com/kubernetes-csi/external-snapshotter
安装 Snapshot CRD
cd external-snapshotter/
kubectl kustomize client/config/crd |kubectl create -f -
Install Common Snapshot Controller
cd deploy/kubernetes/snapshot-controller
vim setup-snapshot-controller.yaml
# 修改image地址
image: registry.k8s.io/sig-storage/snapshot-controller:v6.2.1
# 改为
image: swr.cn-north-4.myhuaweicloud.com/ctl456/snapshot-controller:v6.2.1
kubectl -n kube-system kustomize ./ |kubectl create -f -
检查是否安装成功
kubectl get pod -n kube-system | grep snap
创建SnapshotClass
kubectl create -f csi/rbd/snapshotclass.yaml
创建快照
首先在之前所创建的mysql容器创建一个文件夹
# 查看mysql容器的名字
kubectl get pod
kubectl exec -it pod名称 -- bash
cd var/lib/mysql
mkdir test_snapshot
echo "test for snapshot" > test_snapshot/1.txt
exit
修改snapshot.yaml文件
# 查看PVC名称
kubectl get pvc
vim csi/rbd/snapshot.yaml
# 改为mysql的PVC
persistentVolumeClaimName: mysql的pvc
创建快照及查看状态
kubectl create -f csi/rbd/snapshot.yaml
kubectl get volumesnapshotclass
kubectl get volumesnapshot
使用快照恢复数据
修改pvc-restore.yaml
vim csi/rbd/pvc-restore.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-restore
spec:
storageClassName: rook-ceph-block # 新建PVC的storageClass
dataSource:
name: rbd-pvc-snapshot # 快照的名称
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi # 大小不能低于原pvc的大小
运行yaml文件
kubectl create -f csi/rbd/pvc-restore.yaml
kubectl get pvc
数据校验
创建一个容器挂载这个PVC
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: check-snapshot-restore
spec:
selector:
matchLabels:
app: check
strategy:
type: Recreate
template:
metadata:
labels:
app: check
spec:
containers:
- image: swr.cn-north-4.myhuaweicloud.com/ctl456/alpine:3.6
name: check
command:
- sh
- -c
- sleep 36000
volumeMounts:
- name: check-mysql-persistent-storage
mountPath: /mnt
volumes:
- name: check-mysql-persistent-storage
persistentVolumeClaim:
claimName: rbd-pvc-restore # 新PVC的NAME
kubectl create -f restore-check-snaphost-rbd.yaml
查看数据是否存在
# 查看pod名称
kubectl get pod
kubectl exec -it pod名称 -- ls /mnt/test_snapshot
kubectl exec -it pod名称 -- cat /mnt/test_snapshot/1.txt
清理数据
kubectl delete -f restore-check-snaphost-rbd.yaml -f csi/rbd/pvc-restore.yaml -f csi/rbd/snapshot.yaml
PVC克隆
yaml文件
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-clone
spec:
storageClassName: rook-ceph-block
dataSource:
name: rbd-pvc # 被克隆PVC的名称
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi # 大小不能低于之前PVC的大小
1327
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
