kubernetes的ceph RBD volume(1)：使用Ceph RBD作为后端Volume

最新推荐文章于 2022-10-12 22:00:43 发布

Junming_Zhao

最新推荐文章于 2022-10-12 22:00:43 发布

阅读量1.8k

点赞数 1

分类专栏： ceph kubernetes 文章标签： kubernetes ceph

本文链接：https://blog.csdn.net/junming_zhao/article/details/72625448

版权

ceph 同时被 2 个专栏收录

7 篇文章 0 订阅

订阅专栏

kubernetes

6 篇文章 0 订阅

订阅专栏

Kubernetes使用Ceph RBD作为后端Volume。 Kubernetes的官方源码的examples/volumes/rbd目录下，就有一个使用cephrbd作为kubernetes pod volume的例子，我们可以参考

1. 当ceph集群安装完成以后，我们就要创建相应的rbd块用于kubernetes存储。创建块设备之前，需要先创建存储池

ceph osd pool create kube 256 256 #后面两个256分别为pg-num和pgp-num

ceph osd pool ls detail

pool 4 'kube' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 2410 flags hashpspool stripe_width 0
removed_snaps [1~3]

2. 在kube存储池创建一个映像文件，就叫vol50，该映像文件的大小为50GB：

rbd create kube/vol50 --size 50000

rbd -p kube info vol50
rbd image 'vol50':
   size 51200 MB in 12800 objects
   order 22 (4096 kB objects)
   block_name_prefix: rb.0.754829.238e1f29
   format: 1

3. 创建用户client.kube 用admin用户

ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read class-write object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring

通常我们在ceph install时在ceph.conf中使用默认的安全验证协议 cephx – The Ceph authentication protocol 了。

4. 生成secret

得到key(base64)

grep key /etc/ceph/ceph.client.kube.keyring |awk '{printf "%s", $NF}'|base64

QVFCK0l4RlpqK0xDTkJBQTRKYVBPTWx6WkFIVVhLK2ozM2lQdUE9PQo=

写secret.yaml：

[root@testnew rbd]# cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
type: "kubernetes.io/rbd"
data:
key: QVFCK0l4RlpqK0xDTkJBQTRKYVBPTWx6WkFIVVhLK2ozM2lQdUE9PQo=

copy /etc/ceph/ceph.client.kube.keyring和ceph.conf到kubernetes的所有节点

kubectl create -f ceph-secret.yaml

[root@testnew ~]# kubectl get secret
NAME TYPE DATA AGE
ceph-secret kubernetes.io/rbd 1 13d

5. 格式化一个空image

格式化一个空image那样对其进行格式化了，这里格成ext4文件系统（格式化这一步可以不需要）

rbd map kube/vol50

rbd info kube/vol50

mkfs.ext4 /dev/rbd0

rbd unmap /dev/rbd0

6. 创建pod with RBD

[root@testnew kube]# cat frontend-rbd-controller.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: frontendrbd1
labels:
    name: frontendrbd1
spec:
replicas: 1
selector:
    name: frontendrbd1
template:
    metadata:
      labels:
        name: frontendrbd1
    spec:
      containers:
      - name: frontendrbd1
        image: kubeguide/guestbook-php-frontend
        env:
        - name: GET_HOSTS_FROM
          value: env
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /mnt/rbd
          name: rbdpb
      volumes:
      - name: rbdpb
        rbd:
          monitors:
          - 10.0.200.11:6789
          - 10.0.200.13:6789
          - 10.0.200.14:6789
          pool: kube
          image: vol50
          user: kube
          secretRef:
              name: ceph-secret
          fsType: ext4
          readOnly: false

kubectl create -f frontend-rbd-controller.yaml

[root@testnew ~]# kubectl get rc
NAME            DESIRED   CURRENT   READY     AGE
frontendrbd1    1         1         1         13d

[root@testnew ~]# kubectl get pods
NAME                  READY     STATUS    RESTARTS   AGE
frontendrbd1-h9z78    1/1       Running   1          13d

7. 验证volume在container里。

[root@testnew ~]# kubectl exec frontendrbd1-h9z78 -it bash
root@frontendrbd1-h9z78:/var/www/html# df -k
Filesystem                                                                                       1K-blocks     Used Available Use% Mounted on
/dev/mapper/docker-253:1-530097-861967a5b3b1a5f40b4880db1921a52af2656a10bf5ce9d1727c40845a4aa9c2 10474496   623084   9851412   6% /
tmpfs                                                                                              4087712        0   4087712   0% /dev
tmpfs                                                                                              4087712        0   4087712   0% /sys/fs/cgroup
/dev/rbd0                                                                                         51474912 16840936 31996152 35% /mnt/rbd
/dev/vda1                                                                                         19593296 15144980   3577460 81% /etc/hosts
shm                                                                                                  65536        0     65536   0% /dev/shm