k8s之Statefulset(以部署nginx分析)

k8s之Statefulset小案例

实验环境

k8s环境参考： k8s-v1.20.10 二进制部署指导文档

Statefulset简介

​ StatefulSet是为了解决有状态服务的问题（对应Deployments和ReplicaSets是为无状态服务）而设计，其应用场景包括

• 在 k8s 中，ReplicaSet 和 Deployment 主要是用于处理无状态的服务，无状态服务的需求往往非常简单并且轻量，每一个无状态节点存储的数据在重启之后就会被删除。但是如果我们需要保留，那该怎么办呢？所以为了满足有状态的服务这一特殊需求，StatefulSet 就是 Kubernetes 为了运行有状态服务引入的资源，例如 MySQL 等
• 稳定的持久化存储，即Pod重新调度后还是能访问到相同的持久化数据，基于PVC来实现
• 稳定的网络标志，即Pod重新调度后其PodName和HostName不变，基于Headless Service（即没有Cluster IP的Service）来实现
• 有序部署，有序扩展，即Pod是有顺序的，在部署或者扩展的时候要依据定义的顺序依次进行（即从0到N-1，在下一个Pod运行之前的所有Pod必须都是Running和Ready状态），基于init containers来实现
• 有序收缩，有序删除（即从N-1到0），删除 Pod 不会删除其 pvc(后续重新部署还会使用之前的volume),手动删除 pvc 将自动释放 pv
• 通过Headless Service生成可解析的DNS记录
• 通过volumeClaimTemplates创建pvc和对应的pv绑定
• 和Deployment相同的是，StatefulSet管理了基于相同容器定义的一组 Pod。但和 Deployment不同的是，StatefulSet 为它们的每个 Pod 维护了一个固定的ID。这些 Pod 是基于相同的声明来创建的，但是不能相互替换：无论怎么调度，每个 Pod 都有一个永久不变的ID。StatefulSet和其他控制器使用相同的工作模式。你在StatefulSet对象中定义你期望的状态，然后StatefulSet的控制器就会通过各种更新来达到那种你想要的状态。
• StatefulSet中每个Pod的DNS格式为statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local

网络信息

• StatefulSet 中 Pod 的 hostname 格式为 $(StatefulSetname)-$(Pod 序号)。上面的例子将要创建三个 Pod，其名称分别为： web-0，web-1，web-2。
• StatefulSet 可以使用 Headless Service 来控制其 Pod 所在的域。该域（domain）的格式为 $(servicename).$(namespace).svc.cluster.local，其中 “cluster.local” 是集群的域。
• StatefulSet 中每一个 Pod 将被分配一个 dnsName，格式为： $(podName).$(所在域名)

字段名	组合一	组合二	组合三
集群域 Cluster Domain	cluster.local	cluster.local	kube.local
Service name	default/nginx	foo/nginx	foo/nginx
StatefulSet name	default/web	foo/web	foo/web
StatefulSet Domain	nginx.default.svc.cluster.local	nginx.foo.svc.cluster.local	nginx.foo.svc.kube.local
Pod DNS	web-{0…N-1}.nginx.default.svc.cluster.local	web-{0…N-1}.nginx.foo.svc.cluster.local	web-{0…N-1}.nginx.foo.svc.kube.local
Pod name	web-{0…N-1}	web-{0…N-1}	web-{0…N-1}

Headless service

Headless service不分配clusterIP，headless service可以通过解析service的DNS,返回所有Pod的dns和ip地址 (statefulSet部署的Pod才有DNS)，普通的service，只能通过解析service的DNS返回service的ClusterIP

​ 在使用Deployment时，创建的Pod名称是没有顺序的，是随机字符串，在用statefulset管理pod时要求pod名称必须是有序的 ，每一个pod不能被随意取代，pod重建后pod名称还是一样的。因为pod IP是变化的，所以要用Pod名称来识别。pod名称是pod唯一性的标识符，必须持久稳定有效。这时候要用到无头服务，它可以给每个Pod一个唯一的名称

# headless service会为service分配一个域名
	<service name>.$<namespace name>.svc.cluster.local
	
# K8s中资源的全局FQDN格式
	Service_NAME.NameSpace_NAME.Domain.LTD.
	Domain.LTD.=svc.cluster.local  #这是默认k8s集群的域名

# FQDN 全称 Fully Qualified Domain Name，即全限定域名：同时带有主机名和域名的名称，
	FQDN = Hostname + DomainNam
	主机名是 test，域名是 baidu.com，FQDN= test.baidu.com

# StatefulSet会为关联的Pod保持一个不变的Pod Name
	statefulset中Pod的名字格式为$(StatefulSet name)-$(pod序号)

# StatefulSet会为关联的Pod分配一个dnsName
	<Pod Name>.$<service name>.$<namespace name>.svc.cluster.local

部署nfs-provisioner

[root@k8s-master-1 storageClass]# cat deployment.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      nodeName: k8s-master-1   #设置在master节点运行
      tolerations:             #设置容忍master节点污点
      - key: node-role.kubernetes.io/master
        operator: Equal
        value: "true"
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/jiayu-kubernetes/nfs-subdir-external-provisioner:v4.0.0
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.0.11
            - name: NFS_PATH
              value: /data/k8s-nfs/nfs-provisioner
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.0.11  # NFS SERVER_IP
            path: /data/k8s-nfs/nfs-provisioner          
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nginx-nfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"  # 是否设置为默认的storageclass
provisioner: k8s/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
allowVolumeExpansion: true
parameters:
  archiveOnDelete: "false" # 设置为"false"时删除PVC不会保留数据,"true"则保留数据

部署Statefulset

[root@k8s-master-1 statefulset]# cat nginx.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  clusterIP: None
  selector:
    app: nginx
  ports:
  - name: web
    port: 80
    protocol: TCP
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  podManagementPolicy: OrderedReady  #pod名-> 0-N,删除N->0
  replicas: 3
  revisionHistoryLimit: 10
  serviceName: nginx
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:  #name没写，会默认生成的
      labels:  
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - name: web #填vcp名字
          mountPath: /usr/share/nginx/test
  volumeClaimTemplates:
  - metadata:
      name: web
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: nginx-nfs
      volumeMode: Filesystem
      resources:
        requests:
          storage: 512M

测试分析

# 查看svc
[root@k8s-master-1 statefulset]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP   3d10h
nginx        ClusterIP   None         <none>        80/TCP    2m48s

# 查看pods
[root@k8s-master-1 statefulset]# kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-67d8f9ffff-zpqzb   1/1     Running   0          36m
web-0                                     1/1     Running   0          2m39s
web-1                                     1/1     Running   0          2m34s
web-2                                     1/1     Running   0          2m29s

# 查看pvc
[root@k8s-master-1 statefulset]# kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
web-web-0   Bound    pvc-e6e8b347-aec4-4b18-839b-1363a8d5ade1   512M       RWO            nginx-nfs      3m39s
web-web-1   Bound    pvc-24f1590e-1904-4780-b761-b5433adb7816   512M       RWO            nginx-nfs      3m34s
web-web-2   Bound    pvc-6bc0b88e-63ba-404c-8195-cfc5f1372b25   512M       RWO            nginx-nfs      3m29s

# 查看pv
[root@k8s-master-1 statefulset]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-24f1590e-1904-4780-b761-b5433adb7816   512M       RWO            Delete           Bound    default/web-web-1   nginx-nfs               3m57s
pvc-6bc0b88e-63ba-404c-8195-cfc5f1372b25   512M       RWO            Delete           Bound    default/web-web-2   nginx-nfs               3m52s
pvc-e6e8b347-aec4-4b18-839b-1363a8d5ade1   512M       RWO            Delete           Bound    default/web-web-0   nginx-nfs               4m2s

# 测试headless dns解析，解析出来的并不是service ip，而是后端的POD IP
[root@k8s-master-1 ~]# kubectl run -it --rm busybox --image=busybox:1.28 -- sh
If you don't see a command prompt, try pressing enter.
/ # nslookup nginx.default.svc.cluster.local
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx.default.svc.cluster.local
Address 1: 10.70.2.47 web-0.nginx.default.svc.cluster.local
Address 2: 10.70.2.49 web-2.nginx.default.svc.cluster.local
Address 3: 10.70.2.48 web-1.nginx.default.svc.cluster.local

# 测试POD dns解析
/ # nslookup web-0.nginx.default.svc.cluster.local
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx.default.svc.cluster.local
Address 1: 10.70.2.47 web-0.nginx.default.svc.cluster.local

# 删除statefulset
[root@k8s-master-1 statefulset]# kubectl delete -f nginx.yaml 
service "nginx" deleted
statefulset.apps "web" deleted

# 查看pv和pvc，从claim可以看出，后续如果重新部署statefulset还是会使用同样的pv
[root@k8s-master-1 statefulset]# kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
web-web-0   Bound    pvc-e6e8b347-aec4-4b18-839b-1363a8d5ade1   512M       RWO            nginx-nfs      9m1s
web-web-1   Bound    pvc-24f1590e-1904-4780-b761-b5433adb7816   512M       RWO            nginx-nfs      8m56s
web-web-2   Bound    pvc-6bc0b88e-63ba-404c-8195-cfc5f1372b25   512M       RWO            nginx-nfs      8m51s
[root@k8s-master-1 statefulset]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-24f1590e-1904-4780-b761-b5433adb7816   512M       RWO            Delete           Bound    default/web-web-1   nginx-nfs               8m59s
pvc-6bc0b88e-63ba-404c-8195-cfc5f1372b25   512M       RWO            Delete           Bound    default/web-web-2   nginx-nfs               8m54s
pvc-e6e8b347-aec4-4b18-839b-1363a8d5ade1   512M       RWO            Delete           Bound    default/web-web-0   nginx-nfs               9m4s