如何使用阿里的csi-lugins的nas部分
文章目录
github地址,gitee也有很多拷贝
kubernetes-sigs/alibaba-cloud-csi-driver
说明: CSI插件需要您的集群版本大于1.14,且初始化集群的时候选择使用CSI插件。或者自行配置节点kubelet参数,确保enable-controller-attach-detach
为true。意思是由AD Controller来发起attach/detach请求,而不是kubelet发起。
1.创建serviceAccount
kubectl create -f xxx/alibaba-cloud-csi-driver/deploy/rbac.yaml
建议还不了解工作原理的可以先用下面的配置部署
# This YAML file contains all RBAC objects that are necessary to run external
# CSI provisioner.
#
# In production, each CSI driver deployment has to be customized:
# - to avoid conflicts, use non-default namespace and different names
# for non-namespaced entities like the ClusterRole
# - decide whether the deployment replicates the external CSI
# provisioner, in which case leadership election must be enabled;
# this influences the RBAC setup, see below
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-admin
# replace with the same namespace name with plugin
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: alicloud-csi-plugin
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["persistentvolumeclaims/status"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["create", "get", "list", "watch", "update", "delete"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["create", "list", "watch", "delete", "get", "update"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "list", "watch", "delete", "update"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents/status"]
verbs: ["update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots/status"]
verbs: ["update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: alicloud-csi-plugin
namespace: kube-system
subjects:
- kind: ServiceAccount
name: csi-admin
namespace: kube-system
roleRef:
kind: ClusterRole
name: alicloud-csi-plugin
apiGroup: rbac.authorization.k8s.io
2.容器部署csi-plugin
kubectl create -f xxx/alibaba-cloud-csi-driver/deploy/nas/nas-plugin.yaml
nas-plugin.yaml
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: nasplugin.csi.alibabacloud.com
spec:
attachRequired: false
podInfoOnMount: true
---
# This YAML defines all API objects to create RBAC roles for csi node plugin.
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-plugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-plugin
template:
metadata:
labels:
app: csi-plugin
spec:
tolerations:
- operator: Exists
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
nodeSelector:
beta.kubernetes.io/os: linux
serviceAccount: csi-admin
priorityClassName: system-node-critical
hostNetwork: true
hostPID: true
containers:
- name: nas-driver-registrar
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v1.2.0
imagePullPolicy: Always
args:
- "--v=5"
- "--csi-address=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet/
- name: registration-dir
mountPath: /registration
- name: csi-plugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.16.9.43-f36bb540-aliyun
imagePullPolicy: "Always"
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=nas"
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-plugins/driverplugin.csi.alibabacloud.com-replace/csi.sock
- name: MAX_VOLUMES_PERNODE
value: "15"
- name: SERVICE_TYPE
value: "plugin"
livenessProbe:
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
ports:
- name: healthz
containerPort: 11260
protocol: TCP
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet/
mountPropagation: "Bidirectional"
- name: etc
mountPath: /host/etc
- name: host-log
mountPath: /var/log/
- name: container-dir
mountPath: /var/lib/container
mountPropagation: "Bidirectional"
- name: host-dev
mountPath: /dev
mountPropagation: "HostToContainer"
volumes:
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: DirectoryOrCreate
- name: container-dir
hostPath:
path: /var/lib/container
type: DirectoryOrCreate
- name: kubelet-dir
hostPath:
path: /var/lib/kubelet
type: Directory
- name: host-dev
hostPath:
path: /dev
- name: host-log
hostPath:
path: /var/log/
- name: etc
hostPath:
path: /etc
updateStrategy:
rollingUpdate:
maxUnavailable: 10%
type: RollingUpdate
nas-provisioner.yaml
kubectl create -f xxx/alibaba-cloud-csi-driver/deploy/nas/nas-provisioner.yaml
kind: Service
apiVersion: v1
metadata:
name: csi-provisioner
namespace: kube-system
labels:
app: csi-provisioner
spec:
selector:
app: csi-provisioner
ports:
- name: dummy
port: 12345
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: csi-provisioner
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-provisioner
replicas: 2
template:
metadata:
labels:
app: csi-provisioner
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
tolerations:
- effect: NoSchedule
operator: Exists
key: node-role.kubernetes.io/master
- effect: NoSchedule
operator: Exists
key: node.cloudprovider.kubernetes.io/uninitialized
priorityClassName: system-node-critical
serviceAccount: csi-admin
hostNetwork: true
containers:
- name: external-nas-provisioner
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v1.4.0-aliyun
args:
- "--provisioner=nasplugin.csi.alibabacloud.com"
- "--csi-address=$(ADDRESS)"
- "--volume-name-prefix=nas"
- "--timeout=150s"
- "--enable-leader-election=true"
- "--leader-election-type=leases"
- "--retry-interval-start=500ms"
- "--v=5"
env:
- name: ADDRESS
value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
imagePullPolicy: "Always"
volumeMounts:
- name: nas-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
- name: csi-provisioner
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.16.9.43-f36bb540-aliyun
imagePullPolicy: "Always"
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=nas"
env:
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-provisioner/driverplugin.csi.alibabacloud.com-replace/csi.sock
- name: MAX_VOLUMES_PERNODE
value: "15"
- name: SERVICE_TYPE
value: "provisioner"
livenessProbe:
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
ports:
- name: healthz
containerPort: 11270
protocol: TCP
volumeMounts:
- name: host-dev
mountPath: /dev
mountPropagation: "HostToContainer"
- name: host-log
mountPath: /var/log/
- name: etc
mountPath: /host/etc
- name: nas-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
resources:
limits:
cpu: 1000m
memory: 1000Mi
requests:
cpu: 100m
memory: 100Mi
volumes:
- name: nas-provisioner-dir
emptyDir: {}
- name: host-log
hostPath:
path: /var/log/
- name: host-dev
hostPath:
path: /dev
- name: etc
hostPath:
path: /etc
3.测试
demo.yaml
kubectl create -f demo.yaml
下面的 # * 此处做替换!!!
server: “xxxxxxxxx.cn-shenzhen.nas.aliyuncs.com:/k8s/”到阿里云的NAS控制台创建一个文件系统
注意要和主机在同一个专有网,可以跨可用区
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: alicloud-nas-sp
namespace: default
annotations:
storageclass.kubernetes.io/is-default-class: 'true'
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
volumeAs: subpath # 基于已存在的NAS,并在其/k8s/子路径attach;可选参数:filesystem:创建一个文件系统和一个挂载点
# * 此处做替换!!!
server: "xxxxxxxxx.cn-shenzhen.nas.aliyuncs.com:/k8s/"
archiveOnDelete: "true"
provisioner: nasplugin.csi.alibabacloud.com
reclaimPolicy: Delete
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nas-pvc
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: alicloud-nas-sp
---
apiVersion: v1
kind: Pod
metadata:
name: pod-demo
namespace: default
spec:
containers:
- name: nginx-nas-pv
image: nginx
volumeMounts:
- name: nas-pvc
mountPath: /data
volumes:
- name: nas-pvc
persistentVolumeClaim:
claimName: nas-pvc
4.正确启用的效果展示