腾讯云自建k8s对接文件存储CFS

环境

服务器：轻量应用服务器，使用私有网络VPC，需关联云联网实现内网互联

k8s：通过sealyun一键安装1.19.16版本k8s单主集群

csi插件：kubernetes-csi-tencentcloud

部署步骤

创建文件系统及挂载点

• 创建vpc，注意不要与轻量应用服务器自动生成的vpc冲突，否则云联网无法互联
• 创建文件系统及挂载点，记录下文件存储的客户端IP：172.16.0.10

验证网络通信

• 申请云联网关联，将轻量应用服务实例关联云联网；
• 将文件系统的 VPC 实例关联至云联网
• 在轻量应用服务器上telnet文件存储客户端的nfs端口，验证网络通信

telnet 172.16.0.10 111
telnet 172.16.0.10 2049 

挂载测试

mkdir /localfolder
sudo mount -t nfs -o vers=4.0,noresvport 172.16.0.10:/ /localfolder

安装CSI插件

准备API密钥

创建子用户，并授权文件存储全读写访问权限，记录API密钥

SecretId AKID4xaw2rRknZ4IFvKSJtCEzUifLksjKfkY 
SecretKey xxxxxxxxxxxxxxxxxxxxxxx

安装CSI-CFS插件

插件Github地址：kubernetes-csi-tencentcloud/cfs

• rbac

kubectl apply -f  deploy/cfs/kubernetes/csi-cfs-rbac.yaml

• 根据k8s版本安装相应版本的插件，将API密钥填充至csi-provisioner-cfsplugin.yaml文件中

kubectl apply -f  deploy/cfs/kubernetes/csi-cfs-csidriver-old.yaml
kubectl apply -f  deploy/cfs/kubernetes/csi-nodeplugin-cfsplugin-new.yaml
kubectl apply -f  deploy/cfs/kubernetes/csi-provisioner-cfsplugin-new.yaml

静态挂载测试

• 创建web目录

mkdir -p /localfolder/web

• static-allinone

vim static-allinone.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: web-static-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  csi:
    driver: com.tencent.cloud.csi.cfs
    volumeHandle: web-static-pv
    volumeAttributes: 
      host: 172.16.0.10
      path: /web       # 如果不是根目录的话，需提前创建
  storageClassName: ""
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: web-static-pvc
spec:
  storageClassName: ""
  volumeName: web-static-pv
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx 
spec:
  containers:
  - image: ccr.ccs.tencentyun.com/qcloud/nginx:1.9
    imagePullPolicy: Always
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    volumeMounts:
      - mountPath: /usr/share/nginx/html
        name: data-vol
  volumes:
  - name: data-vol
    persistentVolumeClaim:
      claimName: web-static-pvc

• 测试

echo "hello cfs" > /localfolder/web/index.html

动态挂载

vim dynamic-provison-allinone.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cfsauto
parameters:
  vpcid: vpc-klr8fwg5
  subnetid: subnet-63c5x24c
  resourcetags: ""
provisioner: com.tencent.cloud.csi.cfs
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-cfsplugin
spec:
  storageClassName: cfsauto
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: ccr.ccs.tencentyun.com/qcloud/nginx:1.9
    imagePullPolicy: Always
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
    volumeMounts:
      - mountPath: /usr/share/nginx/html
        name: data-cfsplugin
  volumes:
  - name: data-cfsplugin
    persistentVolumeClaim:
      claimName: data-cfsplugin

kubectl apply -f dynamic-provison-allinone.yaml

• 会自动创建新的cfs，好像这个sc不能直接绑定已有cfs的subpath（待后续研究）