kubernetes部署cephcsi链接外部ceph

藏经老曾

已于 2023-04-21 13:32:34 修改

阅读量1.2k

点赞数

分类专栏： kubernetes 文章标签： kubernetes ceph docker

于 2023-04-21 13:19:59 首次发布

本文链接：https://blog.csdn.net/zhangkangren/article/details/130282188

版权

kubernetes 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

一、Ceph CIS 介绍

1.1 功能介绍

Ceph Container Storage Interface（ceph CSI）是 Ceph 存储集群提供给 Kubernetes 的一种插件式接口，用于 Kubernetes 中动态配置和管理 Ceph 存储。

CSI 是 Kubernetes 的第三方存储插件，为 Kubernetes 集群中运行的 Pod 提供了直接访问 Ceph 存储集群的能力。通过 CSI 接口，Kubernetes 用户可以使用 Ceph 存储集群作为 Kubernetes 集群中的持久化存储，而不需要深入了解 Ceph 存储集群的底层实现。

1.2 工作原理：

Kubernetes 用户需要通过 CSI 接口在 Kubernetes 集群中创建一个 PersistentVolumeClaim（PVC），并指定所需的存储容量和其他相关参数。
Kubernetes 会将 PVC 转换成 CSI NodePublishRequest，并发送到 CSI 插件。
CSI 插件会将 NodePublishRequest 转换成 Ceph 存储集群中的 API 请求，并将结果返回给 Kubernetes。
Kubernetes 将结果转换成 CSI NodePublishResponse，并将其发送到 Node 上运行的 CSI 插件。
CSI 插件会将 NodePublishResponse 转换成 Ceph 存储集群中的 API 请求，并将结果返回给 Kubernetes。

通过 CSI 接口，用户可以方便地在 Kubernetes 集群中使用 Ceph 存储集群，从而为 Kubernetes 应用程序提供持久化数据存储。同时，CSI 还提供了一系列的安全机制和管理功能，使 Kubernetes 用户可以更加灵活地管理和配置 Ceph 存储集群。

1.3 主要作用

配置管理 - Ceph CIS允许管理员在集群中配置和管理Ceph存储集群的各种操作。通过CIS，管理员可以执行诸如添加/删除/更新OSD（对象存储设备）、创建/删除CRUSH图等操作。
监控 - Ceph CIS提供了对Ceph存储集群的实时监控功能。它可以帮助管理员跟踪集群的使用情况、性能瓶颈以及故障状态。
故障排除 - Ceph CIS允许管理员检查集群各个组件的状态，以便快速发现和修复故障。例如，如果某个OSD停止工作，管理员可以使用CIS来诊断和解决该问题。
性能优化 - Ceph CIS可以收集并分析Ceph集群的性能数据，以便管理员进行优化。通过CIS，管理员可以了解集群中每个组件的负载情况，以及哪些操作可能会影响集群的性能。

总之，Ceph CIS是Ceph存储集群的重要组成部分，它可以帮助管理员轻松地管理、监控和维护整个集群，确保其高可用性和高性能。

二、部署Ceph CIS插件

ceph集群部署
镜像下载
本站镜像下载

docker pull quay.io/k8scsi/csi-resizer:v1.1.0
docker pull quay.io/k8scsi/csi-attacher:v3.1.0
docker pull quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
docker pull quay.io/k8scsi/csi-provisioner:v2.1.0
docker pull quay.io/k8scsi/csi-snapshotter:v3.0.3
docker pull quay.io/cephcsi/cephcsi:v3.2.0

2.1 创建Namespace

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: cephcsi
  name: cephcsi

2.2 创建ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
  namespace: cephcsi
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  config.json: |-
    [{"clusterID":"92bdd53a-c560-42ea-aecd-056c94808f17","monitors":["192.168.17.50:6789","192.168.17.60:6789","192.168.17.70:6789"],"cephFS":{}}]

# 以上配置信息
ceph mon dump

dumped monmap epoch 3
epoch 3
fsid 92bdd53a-c560-42ea-aecd-056c94808f17  # clusterID
last_changed 2023-04-11 16:44:54.723870
created 2023-04-11 16:43:36.126758
min_mon_release 14 (nautilus)
0: [v2:192.168.17.50:3300/0,v1:192.168.17.50:6789/0] mon.node50 #192.168.17.50:6789地址
1: [v2:192.168.17.60:3300/0,v1:192.168.17.60:6789/0] mon.node60
2: [v2:192.168.17.70:3300/0,v1:192.168.17.70:6789/0] mon.node70

2.3 创建Secret

apiVersion: v1
kind: Secret
metadata:
  name: cephfs-secret-ceph
  namespace: cephcsi
data:
  adminID: admin
  adminKey: AQC4HTVkrEaTJxAAsj16ACmuxiDCmlSDEmkInA==
  userID: admin
  userKey: AQC4HTVkrEaTJxAAsj16ACmuxiDCmlSDEmkInA==

# 获取admin用户的AdminKey，执行指令
ceph auth get client.admin

exported keyring for client.admin
[client.admin]
        key = AQC4HTVkrEaTJxAAsj16ACmuxiDCmlSDEmkInA== #adminKey和userKey
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

2.4 创建StorageClass

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    k8s.kuboard.cn/storageType: cephfs_provisioner
  name: ceph
parameters:
  clusterID: 92bdd53a-c560-42ea-aecd-056c94808f17 #与ConfigMap相同
  csi.storage.k8s.io/controller-expand-secret-name: cephfs-secret-ceph
  csi.storage.k8s.io/controller-expand-secret-namespace: cephcsi
  csi.storage.k8s.io/node-stage-secret-name: cephfs-secret-ceph
  csi.storage.k8s.io/node-stage-secret-namespace: cephcsi
  csi.storage.k8s.io/provisioner-secret-name: cephfs-secret-ceph
  csi.storage.k8s.io/provisioner-secret-namespace: cephcsi
  fsName: cephfs
  pool: cephfs_pool
provisioner: cephfs.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

2.5 创建VolumeSnapshotClass

apiVersion: snapshot.storage.k8s.io/v1beta1
deletionPolicy: Delete
driver: cephfs.csi.ceph.com
kind: VolumeSnapshotClass
metadata:
  annotations: {}
  name: ceph
parameters:
  clusterID: 92bdd53a-c560-42ea-aecd-056c94808f17 #与ConfigMap相同
  csi.storage.k8s.io/snapshotter-secret-name: cephfs-secret-ceph
  csi.storage.k8s.io/snapshotter-secret-namespace: cephcsi

2.6 创建csi-provisioner RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-csi-provisioner
  namespace: cephcsi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cephfs-external-provisioner-runner-clusterrole
rules:
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - list
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - list
      - watch
      - create
      - update
      - patch
  - apiGroups:
      - ''
    resources:
      - persistentvolumes
    verbs:
      - get
      - list
      - watch
      - create
      - delete
      - patch
  - apiGroups:
      - ''
    resources:
      - persistentvolumeclaims
    verbs:
      - get
      - list
      - watch
      - update
  - apiGroups:
      - storage.k8s.io
    resources:
      - storageclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - snapshot.storage.k8s.io
    resources:
      - volumesnapshots
    verbs:
      - get
      - list
  - apiGroups:
      - snapshot.storage.k8s.io
    resources:
      - volumesnapshotcontents
    verbs:
      - create
      - get
      - list
      - watch
      - update
      - delete
  - apiGroups:
      - snapshot.storage.k8s.io
    resources:
      - volumesnapshotclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - storage.k8s.io
    resources:
      - volumeattachments
    verbs:
      - get
      - list
      - watch
      - update
      - patch
  - apiGroups:
      - storage.k8s.io
    resources:
      - volumeattachments/status
    verbs:
      - patch
  - apiGroups:
      - ''
    resources:
      - persistentvolumeclaims/status
    verbs:
      - update
      - patch
  - apiGroups:
      - storage.k8s.io
    resources:
      - csinodes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - snapshot.storage.k8s.io
    resources:
      - volumesnapshotcontents/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cephfs-csi-provisioner-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cephfs-external-provisioner-runner-clusterrole
subjects:
  - kind: ServiceAccount
    name: cephfs-csi-provisioner
    namespace: cephcsi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-external-provisioner-cfg
  namespace: cephcsi
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - get
      - list
      - create
      - delete
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - watch
      - list
      - delete
      - update
      - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-csi-provisioner-role-cfg
  namespace: cephcsi
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-external-provisioner-cfg
subjects:
  - kind: ServiceAccount
    name: cephfs-csi-provisioner
    namespace: cephcsi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-csi-provisioner-psp
  namespace: cephcsi
rules:
  - apiGroups:
      - policy
    resourceNames:
      - cephfs-csi-provisioner-psp
    resources:
      - podsecuritypolicies
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-csi-provisioner-psp
  namespace: cephcsi
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-csi-provisioner-psp
subjects:
  - kind: ServiceAccount
    name: cephfs-csi-provisioner
    namespace: cephcsi

2.7 创建nodepluginRBAC

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-csi-nodeplugin
  namespace: cephcsi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cephfs-csi-nodeplugin
rules:
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cephfs-csi-nodeplugin-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cephfs-csi-nodeplugin-clusterrole
subjects:
  - kind: ServiceAccount
    name: cephfs-csi-nodeplugin
    namespace: cephcsi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-csi-nodeplugin-psp
  namespace: cephcsi
rules:
  - apiGroups:
      - policy
    resourceNames:
      - cephfs-csi-nodeplugin-psp
    resources:
      - podsecuritypolicies
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-csi-nodeplugin-psp
  namespace: cephcsi
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-csi-nodeplugin-psp
subjects:
  - kind: ServiceAccount
    name: cephfs-csi-nodeplugin
    namespace: cephcsi

2.8 创建provisioner Service

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: csi-metrics
  name: csi-cephfsplugin-provisioner
  namespace: cephcsi
spec:
  ports:
    - name: http-metrics
      port: 8080
      protocol: TCP
      targetPort: 8681
  selector:
    app: csi-cephfsplugin-provisioner

2.9 创建cephfsplugin Service

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: csi-metrics
  name: csi-metrics-cephfsplugin
spec:
  ports:
    - name: http-metrics
      port: 8080
      protocol: TCP
      targetPort: 8681
  selector:
    app: csi-cephfsplugin

2.9 创建provisioner Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: csi-cephfsplugin-provisioner
  namespace: cephcsi
spec:
  replicas: 2
  selector:
    matchLabels:
      app: csi-cephfsplugin-provisioner
  template:
    metadata:
      labels:
        app: csi-cephfsplugin-provisioner
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - csi-cephfsplugin-provisioner
              topologyKey: kubernetes.io/hostname
      containers:
        - args:
            - '--csi-address=$(ADDRESS)'
            - '--v=5'
            - '--timeout=150s'
            - '--leader-election=true'
            - '--retry-interval-start=500ms'
            - '--feature-gates=Topology=false'
          env:
            - name: ADDRESS
              value: 'unix:///csi/csi-provisioner.sock'
          image: 'csi-provisioner:v2.1.0'
          imagePullPolicy: IfNotPresent
          name: csi-provisioner
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
        - args:
            - '--csi-address=$(ADDRESS)'
            - '--v=5'
            - '--timeout=150s'
            - '--leader-election'
            - '--retry-interval-start=500ms'
            - '--handle-volume-inuse-error=false'
          env:
            - name: ADDRESS
              value: 'unix:///csi/csi-provisioner.sock'
          image: 'csi-resizer:v1.1.0'
          imagePullPolicy: IfNotPresent
          name: csi-resizer
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
        - args:
            - '--csi-address=$(ADDRESS)'
            - '--v=5'
            - '--timeout=150s'
            - '--leader-election=true'
          env:
            - name: ADDRESS
              value: 'unix:///csi/csi-provisioner.sock'
          image: 'csi-snapshotter:v3.0.3'
          imagePullPolicy: IfNotPresent
          name: csi-snapshotter
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
        - args:
            - '--v=5'
            - '--csi-address=$(ADDRESS)'
            - '--leader-election=true'
            - '--retry-interval-start=500ms'
          env:
            - name: ADDRESS
              value: /csi/csi-provisioner.sock
          image: 'csi-attacher:v3.1.0'
          imagePullPolicy: IfNotPresent
          name: csi-cephfsplugin-attacher
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
        - args:
            - '--nodeid=$(NODE_ID)'
            - '--type=cephfs'
            - '--controllerserver=true'
            - '--endpoint=$(CSI_ENDPOINT)'
            - '--v=5'
            - '--drivername=cephfs.csi.ceph.com'
            - '--pidlimit=-1'
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: 'unix:///csi/csi-provisioner.sock'
          image: 'cephcsi:v3.2.0'
          imagePullPolicy: IfNotPresent
          name: csi-cephfsplugin
          securityContext:
            capabilities:
              add:
                - SYS_ADMIN
            privileged: true
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
            - mountPath: /etc/ceph-csi-config/
              name: ceph-csi-config
            - mountPath: /tmp/csi/keys
              name: keys-tmp-dir
        - args:
            - '--type=liveness'
            - '--endpoint=$(CSI_ENDPOINT)'
            - '--metricsport=8681'
            - '--metricspath=/metrics'
            - '--polltime=60s'
            - '--timeout=3s'
          env:
            - name: CSI_ENDPOINT
              value: 'unix:///csi/csi-provisioner.sock'
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          image: 'cephcsi:v3.2.0'
          imagePullPolicy: IfNotPresent
          name: liveness-prometheus
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
      serviceAccount: cephfs-csi-provisioner
      volumes:
        - emptyDir:
            medium: Memory
          name: socket-dir
        - hostPath:
            path: /sys
          name: host-sys
        - hostPath:
            path: /lib/modules
          name: lib-modules
        - hostPath:
            path: /dev
          name: host-dev
        - configMap:
            name: ceph-csi-config
          name: ceph-csi-config
        - emptyDir:
            medium: Memory
          name: keys-tmp-dir

2.10 创建cephfsplugin DaemonSet

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: csi-cephfsplugin
  namespace: cephcsi
spec:
  selector:
    matchLabels:
      app: csi-cephfsplugin
  template:
    metadata:
      labels:
        app: csi-cephfsplugin
    spec:
      containers:
        - args:
            - '--v=5'
            - '--csi-address=/csi/csi.sock'
            - >-
              --kubelet-registration-path=/var/lib/kubelet/plugins/cephfs.csi.ceph.com/csi.sock
          env:
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          image: 'csi-node-driver-registrar:v2.1.0'
          name: driver-registrar
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
            - mountPath: /registration
              name: registration-dir
        - args:
            - '--nodeid=$(NODE_ID)'
            - '--type=cephfs'
            - '--nodeserver=true'
            - '--endpoint=$(CSI_ENDPOINT)'
            - '--v=5'
            - '--drivername=cephfs.csi.ceph.com'
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: 'unix:///csi/csi.sock'
          image: 'cephcsi:v3.2.0'
          imagePullPolicy: IfNotPresent
          name: csi-cephfsplugin
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - SYS_ADMIN
            privileged: true
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
            - mountPath: /var/lib/kubelet/pods
              mountPropagation: Bidirectional
              name: mountpoint-dir
            - mountPath: /var/lib/kubelet/plugins
              mountPropagation: Bidirectional
              name: plugin-dir
            - mountPath: /sys
              name: host-sys
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /dev
              name: host-dev
            - mountPath: /run/mount
              name: host-mount
            - mountPath: /etc/ceph-csi-config/
              name: ceph-csi-config
            - mountPath: /tmp/csi/keys
              name: keys-tmp-dir
        - args:
            - '--type=liveness'
            - '--endpoint=$(CSI_ENDPOINT)'
            - '--metricsport=8681'
            - '--metricspath=/metrics'
            - '--polltime=60s'
            - '--timeout=3s'
          env:
            - name: CSI_ENDPOINT
              value: 'unix:///csi/csi.sock'
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          image: 'cephcsi:v3.2.0'
          imagePullPolicy: IfNotPresent
          name: liveness-prometheus
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /csi
              name: socket-dir
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      serviceAccount: cephfs-csi-nodeplugin
      tolerations:
        - effect: ''
          key: node-role.kubernetes.io/master
          operator: Exists
      volumes:
        - hostPath:
            path: /var/lib/kubelet/plugins/cephfs.csi.ceph.com/
            type: DirectoryOrCreate
          name: socket-dir
        - hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
          name: registration-dir
        - hostPath:
            path: /var/lib/kubelet/pods
            type: DirectoryOrCreate
          name: mountpoint-dir
        - hostPath:
            path: /var/lib/kubelet/plugins
            type: Directory
          name: plugin-dir
        - hostPath:
            path: /sys
          name: host-sys
        - hostPath:
            path: /lib/modules
          name: lib-modules
        - hostPath:
            path: /dev
          name: host-dev
        - hostPath:
            path: /run/mount
          name: host-mount
        - configMap:
            name: ceph-csi-config
          name: ceph-csi-config
        - emptyDir:
            medium: Memory
          name: keys-tmp-dir

2.11 创建名称为rabbitmq数据卷

---
metadata:
  annotations:
    k8s.kuboard.cn/pvcType: Dynamic
  name: rabbitmq-ceph
  namespace: default    # 需要用到的命名空间
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: ceph

2.12 部署使用使用

# pod的volumes字段声明
volumes:
- name: rabbitmq
  persistentVolumeClaim:
  claimName: rabbitmq-ceph
# 在容器内使用
volumeMounts:
- mountPath: /data
  name: rabbitmq

藏经老曾

关注

0
点赞
踩
2

收藏

觉得还不错? 一键收藏
0
评论
kubernetes部署cephcsi链接外部ceph

Ceph Container Storage Interface（ceph CSI）是 Ceph 存储集群提供给 Kubernetes 的一种插件式接口，用于 Kubernetes 中动态配置和管理 Ceph 存储。CSI 是 Kubernetes 的第三方存储插件，为 Kubernetes 集群中运行的 Pod 提供了直接访问 Ceph 存储集群的能力。
复制链接

扫一扫