最近学习k8s遇到很多问题,建了一个qq群:153144292,交流devops、k8s、docker等
一、statefulset
从前面的学习我们知道使用Deployment创建的pod是无状态的,当挂载了Volume之后,如
果该pod挂了,Replication Controller会再启动一个pod来保证可用性,但是由于pod是无状
态的,pod挂了就会和之前的Volume的关系断开,新创建的Pod无法找到之前的Pod。但是对于
用户而言,他们对底层的Pod挂了是没有感知的,但是当Pod挂了之后就无法再使用之前挂载
的存储卷。
CoreOS:Operator
为了解决这一问题,就引入了StatefulSet用于保留Pod的状态信息。
StatefulSet是为了解决有状态服务的问题(对应Deployments和ReplicaSets是为无状态服务而设计),
其应用场景包括:
statefulset 有状态副本集
cattle,pet
PetSet-->StatefulSet
1、稳定且需要有唯一的标识符
2、稳定且持久的存储
3、有序、平滑优雅的扩展
4、有序、平滑的删除和终止
5、有序的滚动更新
1、稳定的持久化存储,即Pod重新调度后还是能访问到相同的持久化数据,基于PVC来实现
2、稳定的网络标志,即Pod重新调度后其PodName和HostName不变,基于Headless Service(即没有Cluster IP的Service)来实现
3、有序部署,有序扩展,即Pod是有顺序的,在部署或者扩展的时候要依据定义的顺序依次依次进行(即从0到N-1,在下一个Pod运行之前所有之前的Pod必须都是Running和Ready状态),基于init containers来实现
4、有序收缩,有序删除(即从N-1到0)
5、有序的滚动更新
StatefulSet由以下几个部分组成:
Headless Service(无头服务)用于为Pod资源标识符生成可解析的DNS记录。每个pod唯一的名称。
volumeClaimTemplates (存储卷申请模板)基于静态或动态PV供给方式为Pod资源提供专有的固定存储。
StatefulSet,用于管控Pod资源。
二、为什么要有headless??
在deployment中,每一个pod是没有名称,是随机字符串,是无序的。而statefulset中是要求有序
的,每一个pod的名称必须是固定的。当节点挂了,重建之后的标识符是不变的,每一个节点的节点名称
是不能改变的。pod名称是作为pod识别的唯一标识符,必须保证其标识符的稳定并且唯一。
为了实现标识符的稳定,这时候就需要一个headless service 解析直达到pod,还需要给pod配置一
个唯一的名称。
内部dns解析:pod_name.service_name.ns_name.svc.cluster.local
三、为什么要 有volumeClainTemplate??
大部分有状态副本集都会用到持久存储,比如分布式系统来说,由于数据是不一样的,每个节点都需
要自己专用的存储节点。而在deployment中pod模板中创建的存储卷是一个共享的存储卷,多个pod使用同
一个存储卷,而statefulset定义中的每一个pod都不能使用同一个存储卷,由此基于pod模板创建pod是不
适应的,这就需要引入volumeClainTemplate,当在使用statefulset创建pod时,会自动生成一个PVC,从
而请求绑定一个PV,从而有自己专用的存储卷。
四、statefulSet使用演示
在创建StatefulSet之前需要准备的东西,值得注意的是创建顺序非常关键,创建顺序如下:
1、Volume
2、Persistent Volume
3、Persistent Volume Claim
4、Service
5、StatefulSet
Volume可以有很多种类型,比如nfs、glusterfs等
(1)查看statefulset的定义
[root@master ~]# kubectl explain statefulset
KIND: StatefulSet
VERSION: apps/v1
DESCRIPTION:
StatefulSet represents a set of pods with consistent identities. Identities
are defined as: - Network: A single stable DNS and hostname. - Storage: As
many VolumeClaims as requested. The StatefulSet guarantees that a given
network identity will always map to the same storage identity.
FIELDS:
apiVersion <string>
kind <string>
metadata <Object>
spec <Object>
status <Object>
[root@master ~]# kubectl explain statefulset.spec
KIND: StatefulSet
VERSION: apps/v1
RESOURCE: spec <Object>
DESCRIPTION:
Spec defines the desired identities of pods in this set.
A StatefulSetSpec is the specification of a StatefulSet.
FIELDS:
podManagementPolicy <string> #Pod管理策略
replicas <integer> #副本数量
revisionHistoryLimit <integer> #历史版本限制
selector <Object> -required- #选择器,必选项
serviceName <string> -required-#服务名称,必选项
template <Object> -required- #模板,必选项
updateStrategy <Object> #更新策略
volumeClaimTemplates <[]Object> #存储卷申请模板,列表对象形式
(2)清单定义StatefulSet
一个完整的StatefulSet控制器由:Headless Service、StatefulSet和volumeClaimTemplate组成。
如下资源清单中的定义:
apiVersion: v1
kind: Service
metadata:
name: myapp-svc
labels:
app: myapp-svc
spec:
ports:
- port: 80
name: web
clusterIP: None 这里无头
selector:
app: myapp-pod
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: myapp
spec:
serviceName: myapp-svc
replicas: 3
selector:
matchLabels:
app: myapp-pod
template:
metadata:
labels:
app: myapp-pod
spec:
containers:
- name: myapp
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: myappdata
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: myappdata
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
解析上例:由于StatefulSet资源依赖于一个实现存在的Headless类型的Service资源,所以
需要先定义一个名为myapp-svc的Headless Service资源,用于为关联到每个Pod资源创建DNS
资源记录。接着定义了一个名为myapp的StatefulSet资源,它通过Pod模板创建了3个Pod资源
副本,并基于volumeClaimTemplates向前面创建的PV进行了请求大小为2Gi的专用存储卷。
(3)删除前期的操作
删除pv、pvc和之前创建的一些pod
(4)修改pv的大小为2Gi
[root@master manifests]# cd volumes/
[root@master volumes]# ls
pod-hostpath--vol.yaml pod-nfs-vol.yaml pod-vol-demo.yaml pod-vol-pvc.yaml pv-demo.yaml vol-gitrepo-pod.yaml
[root@master volumes]# kubectl apply -f pv-demo.yaml
persistentvolume/pv001 created
persistentvolume/pv002 created
persistentvolume/pv003 created
persistentvolume/pv004 created
persistentvolume/pv005 created
[root@master volumes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 7s
pv002 2Gi RWO Retain Available 7s
pv003 2Gi RWO,RWX Retain Available 7s
pv004 4Gi RWO,RWX Retain Available 7s
pv005 5Gi RWO,RWX Retain Available 7s
(5)创建statefulset
[root@master volumes]# kubectl apply -f stateful-demo.yaml
service/myapp-svc created
statefulset.apps/myapp created
[root@master volumes]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 18d
myapp-svc ClusterIP None <none> 80/TCP 10s
[root@master volumes]# kubectl get sts
NAME READY AGE
myapp 3/3 28s
[root@master volumes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myappdata-myapp-0 Bound pv002 2Gi RWO 35s
myappdata-myapp-1 Bound pv003 2Gi RWO,RWX 32s
myappdata-myapp-2 Bound pv004 4Gi RWO,RWX 29s
[root@master volumes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 2m32s
pv002 2Gi RWO Retain Bound default/myappdata-myapp-0 2m32s
pv003 2Gi RWO,RWX Retain Bound default/myappdata-myapp-1 2m32s
pv004 4Gi RWO,RWX Retain Bound default/myappdata-myapp-2 2m32s
pv005 5Gi RWO,RWX Retain Available 2m32s
[root@master volumes]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 64s
myapp-1 1/1 Running 0 61s
myapp-2 1/1 Running 0 58s
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 7m24s
当删除的时候是从myapp-2开始进行删除的,关闭是逆向关闭
[root@master volumes]# kubectl delete -f stateful-demo.yaml
service "myapp-svc" deleted
statefulset.apps "myapp" deleted
[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 95s
myapp-1 1/1 Running 0 92s
myapp-2 1/1 Running 0 89s
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 7m55s
myapp-2 1/1 Terminating 0 93s
myapp-0 1/1 Terminating 0 99s
myapp-1 1/1 Terminating 0 96s
myapp-1 0/1 Terminating 0 97s
myapp-0 0/1 Terminating 0 100s
myapp-2 0/1 Terminating 0 94s
myapp-0 0/1 Terminating 0 101s
myapp-0 0/1 Terminating 0 101s
myapp-2 0/1 Terminating 0 95s
myapp-2 0/1 Terminating 0 95s
myapp-1 0/1 Terminating 0 109s
myapp-1 0/1 Terminating 0 109s
此时PVC依旧存在的,再重新创建pod时,依旧会重新去绑定原来的pvc
[root@master volumes]# kubectl apply -f stateful-demo.yaml
service/myapp-svc created
statefulset.apps/myapp created
[root@master volumes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myappdata-myapp-0 Bound pv002 2Gi RWO 2m42s
myappdata-myapp-1 Bound pv003 2Gi RWO,RWX 2m39s
myappdata-myapp-2 Bound pv004 4Gi RWO,RWX 2m36s
同时这里查看刚才那个动态log,发现是从0开始创建
[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 95s
myapp-1 1/1 Running 0 92s
myapp-2 1/1 Running 0 89s
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 7m55s
myapp-2 1/1 Terminating 0 93s
myapp-0 1/1 Terminating 0 99s
myapp-1 1/1 Terminating 0 96s
myapp-1 0/1 Terminating 0 97s
myapp-0 0/1 Terminating 0 100s
myapp-2 0/1 Terminating 0 94s
myapp-0 0/1 Terminating 0 101s
myapp-0 0/1 Terminating 0 101s
myapp-2 0/1 Terminating 0 95s
myapp-2 0/1 Terminating 0 95s
myapp-1 0/1 Terminating 0 109s
myapp-1 0/1 Terminating 0 109s
myapp-0 0/1 Pending 0 0s
myapp-0 0/1 Pending 0 0s
myapp-0 0/1 ContainerCreating 0 0s
myapp-0 1/1 Running 0 4s
myapp-1 0/1 Pending 0 0s
myapp-1 0/1 Pending 0 0s
myapp-1 0/1 ContainerCreating 0 0s
myapp-1 1/1 Running 0 3s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 ContainerCreating 0 0s
myapp-2 1/1 Running 0 5s的
五、滚动更新、扩展伸缩、版本升级、修改更新策略
1、滚动更新
RollingUpdate 更新策略在 StatefulSet 中实现 Pod 的自动滚动更新。
当StatefulSet的 .spec.updateStrategy.type 设置为 RollingUpdate 时,默认为:RollingUpdate。
StatefulSet 控制器将在 StatefulSet 中删除并重新创建每个 Pod。 它将以与 Pod 终止相同的顺序
进行(从最大的序数到最小的序数),每次更新一个 Pod。 在更新其前身之前,它将等待正在更新的
Pod 状态变成正在运行并就绪。如下操作的滚动更新是有2-0的顺序更新。
vim stateful-demo.yaml 修改image版本为v2
kubectl apply -f stateful-demo.yaml
kubectl get pods -w #查看滚动更新的过程
在创建的每一个Pod中,每一个pod自己的名称都是可以被解析的,如下:
[root@master volumes]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-0 1/1 Running 0 2m21s 10.244.1.35 node01 <none> <none>
myapp-1 1/1 Running 0 2m18s 10.244.2.50 node02 <none> <none>
myapp-2 1/1 Running 0 2m14s 10.244.1.36 node01 <none> <none>
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d 10.244.1.13 node01 <none> <none>
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d 10.244.2.15 node02 <none> <none>
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d 10.244.1.14 node01 <none> <none>
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 12m 10.244.2.47 node02 <none> <none>
[root@master mainfests]# kubectl exec -it myapp-0 -- /bin/sh
#nslookup myapp-0.myapp-svc.default.svc.cluster.local
Address 1: 10.244.1.35 myapp-0.myapp-svc.default.svc.cluster.local
#nslookup myapp-1.myapp-svc.default.svc.cluster.local
Address 1: 10.244.2.50 myapp-0.myapp-svc.default.svc.cluster.local
#nslookup myapp-2.myapp-svc.default.svc.cluster.local
Address 1: 10.244.1.36 myapp-0.myapp-svc.default.svc.cluster.local
从上面的解析,我们可以看到在容器当中可以通过对Pod的名称进行解析到ip。其解析的域名格式如下:
pod_name.service_name.ns_name.svc.cluster.local
eg: myapp-0.myapp.default.svc.cluster.local
2、扩展伸缩
[root@master volumes]# kubectl scale sts myapp --replicas=4
statefulset.apps/myapp scaled
[root@master volumes]# kubectl get pod
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 6m3s
myapp-1 1/1 Running 0 6m
myapp-2 1/1 Running 0 5m56s
myapp-3 1/1 Running 0 36s 这里
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 16m
[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 95s
myapp-1 1/1 Running 0 92s
myapp-2 1/1 Running 0 89s
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 7m55s
myapp-2 1/1 Terminating 0 93s
myapp-1 0/1 ContainerCreating 0 0s
myapp-1 1/1 Running 0 4s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 ContainerCreating 0 0s
myapp-2 1/1 Running 0 3s
myapp-3 0/1 Pending 0 0s
myapp-3 0/1 Pending 0 0s
myapp-3 0/1 Pending 0 0s
myapp-3 0/1 ContainerCreating 0 0s
myapp-3 1/1 Running 0 4s
[root@master volumes]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 12m
pv002 2Gi RWO Retain Bound default/myappdata-myapp-0 12m
pv003 2Gi RWO,RWX Retain Bound default/myappdata-myapp-1 12m
pv004 4Gi RWO,RWX Retain Bound default/myappdata-myapp-2 12m
pv005 5Gi RWO,RWX Retain Bound default/myappdata-myapp-3 12m
[root@master volumes]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myappdata-myapp-0 Bound pv002 2Gi RWO 11m
myappdata-myapp-1 Bound pv003 2Gi RWO,RWX 11m
myappdata-myapp-2 Bound pv004 4Gi RWO,RWX 11m
myappdata-myapp-3 Bound pv005 5Gi RWO,RWX 106s
^C[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 7m35s
myapp-1 1/1 Running 0 7m32s
myapp-2 1/1 Running 0 7m28s
myapp-3 1/1 Running 0 2m8s
nginx-7849c4bbcd-dscjr 1/1 Running 0 16d
nginx-7849c4bbcd-vdd45 1/1 Running 0 16d
nginx-7849c4bbcd-wrvks 1/1 Running 0 16d
nginx-deploy-84cbfc56b6-scrnt 1/1 Running 0 17m
myapp-3 1/1 Terminating 0 2m12s
myapp-3 0/1 Terminating 0 2m13s
myapp-3 0/1 Terminating 0 2m17s
myapp-3 0/1 Terminating 0 2m17s
myapp-2 1/1 Terminating 0 7m37s
myapp-2 0/1 Terminating 0 7m38s
myapp-2 0/1 Terminating 0 7m46s
myapp-2 0/1 Terminating 0 7m46s
3、更新策略和版本升级
修改更新策略,以partition方式进行更新,更新值为2,只有myapp编号大于等于2的才会进行更新。类似于金丝雀部署方式。
[root@master volumes]# kubectl patch sts myapp -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
statefulset.apps/myapp patched
[root@master volumes]# kubectl get sts myapp
NAME READY AGE
myapp 2/2 9m21s
[root@master volumes]# kubectl describe sts myapp
Name: myapp
Namespace: default
CreationTimestamp: Mon, 18 Mar 2019 11:02:39 -0400
Selector: app=myapp-pod
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"myapp","namespace":"default"},"spec":{"replicas":3,"sele...
Replicas: 824639088784 desired | 2 total
Update Strategy: RollingUpdate
Partition: 824639089500
Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=myapp-pod
Containers:
myapp:
Image: nginx
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html from myappdata (rw)
Volumes: <none>
Volume Claims:
Name: myappdata
StorageClass:
Labels: <none>
Annotations: <none>
Capacity: 2Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 9m49s statefulset-controller create Pod myapp-0 in StatefulSet myapp successful
Warning FailedCreate 9m46s (x7 over 9m46s) statefulset-controller create Pod myapp-1 in StatefulSet myapp failed error: The POST operation against Pod could not be completed at this time, please try again.
Normal SuccessfulCreate 9m46s statefulset-controller create Pod myapp-1 in StatefulSet myapp successful
Normal SuccessfulCreate 9m42s statefulset-controller create Pod myapp-2 in StatefulSet myapp successful
Normal SuccessfulCreate 4m22s statefulset-controller create Claim myappdata-myapp-3 Pod myapp-3 in StatefulSet myapp success
Normal SuccessfulCreate 4m22s statefulset-controller create Pod myapp-3 in StatefulSet myapp successful
Normal SuccessfulDelete 2m10s statefulset-controller delete Pod myapp-3 in StatefulSet myapp successful
Normal SuccessfulDelete 2m5s statefulset-controller delete Pod myapp-2 in StatefulSet myapp successful
版本升级,将image的版本升级为v3,升级后对比myapp-2和myapp-1的image版本是不同的。这样就实现了金丝雀发布的效果。
kubectl set image sts/myapp myapp=ikubernetes/myapp:v3
kubectl get sts -o wide
发现版本有2台更新2台未更新
kubectl patch sts myapp -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'