k8s之ceph分布式存储

（ceph集群安装请看上篇文章）

CEPH篇 块存储、文件存储和对象存储意义和差异及ubuntu20.4下 ceph安装_yaodunlin的博客-CSDN博客

各个work节点安装客户端

apt update

apt install ceph-common

K8S 客户端版本必须大于服务器版本   必须拷贝 这两个文件 ceph.client.admin.keyring  ceph.conf 到所有的K8S work节点

服务端拷贝配置文件过去

scp ceph.client.admin.keyring ceph.conf 这两个文件过去

chmod 644 /etc/ceph/ceph.c*

创建StorageClass

 cat class.yaml

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

  name: rbd

provisioner: ceph.com/rbd

parameters:

  monitors: 192.168.1.10:6789,192.168.1.11:6789,192.168.1.12:6789

  pool: kube

  adminId: admin

  adminSecretNamespace: default

  adminSecretName: ceph-secret-admin

  userId: kube

  userSecretNamespace: default

  userSecretName: ceph-secret

  imageFormat: "2"

  imageFeatures: layering

kubectl apply -f  class.yaml

创建Secret

 cat secrets.yaml

apiVersion: v1

kind: Secret

metadata:

  name: ceph-secret-admin

  namespace: default

type: "kubernetes.io/rbd"

data:

  # ceph auth get-key client.admin | base64

  key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

---

apiVersion: v1

kind: Secret

metadata:

  name: ceph-secret

  namespace: default

type: "kubernetes.io/rbd"

data:

  # ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube'

  # ceph auth get-key client.kube | base64

  key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

kubectl apply -f secrets.yaml

创建ClusterRoleBinding

cat clusterrolebinding.yaml

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: rbd-provisioner

subjects:

  - kind: ServiceAccount

    name: rbd-provisioner

    namespace: default

roleRef:

  kind: ClusterRole

  name: rbd-provisioner

  apiGroup: rbac.authorization.k8s.io

创建ClusterRole

cat clusterrole.yaml

kind: ClusterRole

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: rbd-provisioner

rules:

  - apiGroups: [""]

    resources: ["persistentvolumes"]

    verbs: ["get", "list", "watch", "create", "delete"]

  - apiGroups: [""]

    resources: ["persistentvolumeclaims"]

    verbs: ["get", "list", "watch", "update"]

  - apiGroups: ["storage.k8s.io"]

    resources: ["storageclasses"]

    verbs: ["get", "list", "watch"]

  - apiGroups: [""]

    resources: ["events"]

    verbs: ["create", "update", "patch"]

  - apiGroups: [""]

    resources: ["services"]

    resourceNames: ["kube-dns","coredns"]

    verbs: ["list", "get"]

  - apiGroups: [""]

    resources: ["endpoints"]

    verbs: ["get", "list", "watch", "create", "update", "patch"]

创建Deployment

cat deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: rbd-provisioner

spec:

  replicas: 1

  selector:

    matchLabels:

      app: rbd-provisioner

  strategy:

    type: Recreate

  template:

    metadata:

      labels:

        app: rbd-provisioner

    spec:

      containers:

      - name: rbd-provisioner

        image: "quay.io/external_storage/rbd-provisioner:latest"

        env:

        - name: PROVISIONER_NAME

          value: ceph.com/rbd

      serviceAccount: rbd-provisioner

创建RoleBinding

cat rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

  name: rbd-provisioner

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: Role

  name: rbd-provisioner

subjects:

- kind: ServiceAccount

  name: rbd-provisioner

  namespace: default

创建Role

cat role.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: Role

metadata:

  name: rbd-provisioner

rules:

- apiGroups: [""]

  resources: ["secrets"]

  verbs: ["get"]

- apiGroups: [""]

  resources: ["endpoints"]

  verbs: ["get", "list", "watch", "create", "update", "patch"]

创建ServiceAccount

cat serviceaccount.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: rbd-provisioner

kubectl apply -f rbac/

rbd-provisioner

存储类

 PVC

14、遇到的问题

1、

 

解决：ceph-common 客户端版问题

2、