kubeadm k8s配置 ceph rbd存储 类型storageClass

k8s的storageclass,实现pvc的动态创建,绑定
首先安装ceph参考https://blog.csdn.net/qq_42006894/article/details/88424199

1、Install with RBAC roles
参考：https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/deploy

 #cat rbd-provisioner.yaml
 kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns","coredns"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
subjects:
  - kind: ServiceAccount
    name: rbd-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: rbd-provisioner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rbd-provisioner
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
  
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rbd-provisioner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbd-provisioner
subjects:
- kind: ServiceAccount
  name: rbd-provisioner
  namespace: default

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rbd-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: rbd-provisioner
    spec:
      containers:
      - name: rbd-provisioner
        image: quay.io/external_storage/rbd-provisioner:latest
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/rbd
      serviceAccount: rbd-provisioner

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbd-provisioner

kubectl apply -f rbd-provisioner.yaml
2、创建storageClass
创建secret 和 pool

###创建admin secret 
ceph auth get client.admin 2>&1 |grep "key = " |awk '{print  $3'} |xargs echo -n > /tmp/key
kubectl create secret generic ceph-admin-secret --from-file=/tmp/key --namespace=default  --type=kubernetes.io/rbd
###创建ceph osd pool
ceph osd pool create kube 8 8
ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube'
###创建 用户secret 
ceph auth get-key client.kube > /tmp/key
kubectl create secret generic ceph-secret --from-file=/tmp/key --namespace=default   --type=kubernetes.io/rbd

创建storageClass

#cat rbd-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: rbd
provisioner: ceph.com/rbd
parameters:
  monitors: 172.16.16.28:6789
  pool: kube
  adminId: admin
  adminSecretNamespace: default
  adminSecretName: ceph-admin-secret
  userId: kube
  userSecretNamespace: default
  userSecretName: ceph-secret
  imageFormat: "2"
  imageFeatures: layering

kubectl apply -f rbd-sc.yaml
3、创建pvc

#cat mypvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: claim1
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi

创建pvc
kubectl apply -f mypvc.yaml
查看，如果是bound 状态说明成功

#kubectl get pvc
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
claim1   Bound    pvc-ff3e450b-4629-11e9-9740-080027a073ff   1Gi        RWO            rbd            51m

如果出错，定位错误方法

kubectl describe pvc/claim1   
kubectl get pods -o wide 
NAME                               READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
mysql-wayne-8744474d7-kz29c        1/1     Running   0          5d19h   10.244.6.85   node2    <none>           <none>
rbd-provisioner-67b4857bcd-8vsb8   1/1     Running   0          63m     10.244.3.7    node04   <none>           <none>
#查看到 rbd-provisioner 在 node04   上，然后再node04上查看容器log
docker logs -f 6ed00b76cb55         # 6ed00b76cb55   是容器的id （docker ps 查看）

4,遇到的坑
error retrieving resource lock default/ceph.com-rbd: endpoints “ceph.com-rbd” is forbidden: User “system:serviceaccount:default:rbd-provisioner” cannot get endpoints in the namespace “default”
在 rbd-provisioner.yaml 中 clusterrole最后添加

- apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

其他参考https://blog.csdn.net/qq_34857250/article/details/82562514

5、
参考：https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd