kubeadm部署statefulset
在Deployment中,与之对应的服务是service,而在StatefulSet中与之对应的headless service,headless service,即无头服务,
与service的区别就是它没有Cluster IP,解析它的名称时将返回该Headless Service对应的全部Pod的Endpoint列表。
headless service:
在用Deployment时,每一个Pod名称是没有顺序的,是随机字符串,因此是Pod名称是无序的,
但是在statefulset中要求必须是有序 ,每一个pod不能被随意取代,pod重建后pod名称还是一样的。
而pod IP是变化的,所以是以Pod名称来识别。pod名称是pod唯一性的标识符,必须持久稳定有效。
这时候要用到无头服务,它可以给每个Pod一个唯一的名称 。
volumeClaimTemplate:
对于有状态的副本集都会用到持久存储,对于分布式系统来讲,它的最大特点是数据是不一样的,
所以各个节点不能使用同一存储卷,每个节点有自已的专用存储,但是如果在Deployment中的Pod template里定义的存储卷,
是所有副本集共用一个存储卷,数据是相同的,因为是基于模板来的 ,而statefulset中每个Pod都要自已的专有存储卷,
所以statefulset的存储卷就不能再用Pod模板来创建了,于是statefulSet使用volumeClaimTemplate,称为卷申请模板,
它会为每个Pod生成不同的pvc,并绑定pv, 从而实现各pod有专用存储。这就是为什么要用volumeClaimTemplate的原因。
#创建pv
[root@master01 statefulset]# cat /root/pv/pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv01 #pv名称
spec:
capacity: #存储能力,一个pv对象都要指定一个存储能力,目前仅支持存储空间的设置
storage: 2Gi #存储空间
accessModes:
- ReadWriteOnce #访问模式
persistentVolumeReclaimPolicy: Retain #保留策略
nfs: #服务模式 (nfs、ceph、hostpath等)
path: /data/volumes/pv01 #共享数据目录挂载点
server: yunwei.host.com #nfs服务器地址
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv02
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
nfs:
path: /data/volumes/pv02
server: yunwei.host.com
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv03
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
nfs:
path: /data/volumes/pv03
server: yunwei.host.com
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv04
labels:
pvname: pv04
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
nfs:
path: /data/volumes/pv04
server: yunwei.host.com
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv05
spec:
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
nfs:
path: /data/volumes/pv05
server: yunwei.host.com
---
############################################################
#创建pv
[root@master01 pv]# kubectl apply -f pv.yaml
persistentvolume/pv01 created
persistentvolume/pv02 created
persistentvolume/pv03 created
persistentvolume/pv04 created
persistentvolume/pv05 created
[root@master01 pv]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv01 2Gi RWO Retain Available 2s
pv02 2Gi RWO Retain Available 2s
pv03 2Gi RWO Retain Available 2s
pv04 3Gi RWO Retain Available 2s
pv05 3Gi RWO Retain Available 2s
#################################################################
[root@master01 statefulset]# cat statefulset.yaml
apiVersion: v1
kind: Service
metadata:
name: myapp-svc #创建Headless Service名为:myapp-svc
namespace: test
labels:
app: myapp-svc
spec:
ports:
- port: 80
name:
clusterIP: None #定义 Headless Service,必须设置:clusterIP 为 None
selector:
app: myapp
---
apiVersion: apps/v1
kind: StatefulSet #定义创建的类型为:StatefulSet
metadata:
name: myapp
namespace: test #我定义创建StatefulSet类型在test名称空间
spec:
selector:
matchLabels: #通过标签选择器选择pod
app: myapp
serviceName: myapp-svc #声明它属于哪个Headless Service
replicas: 3 #创建3个pod副本
template:
metadata:
labels:
app: myapp #定义pod的标签键值。
spec:
terminationGracePeriodSeconds: 10
containers:
- name: myapp
image: harbor.od.com/kubeadm/myapp:v2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: cport
volumeMounts:
- name: myappdata
mountPath: /usr/share/nginx/html
volumeClaimTemplates: #定义pvc模板,自动会创建pvc,pvc会绑定对应的pv。
- metadata:
name: myappdata
spec:
accessModes: [ "ReadWriteOnce" ] #pvc选择pv的访问模式是只能单个挂载
resources:
requests:
storage: 2Gi #pv的容量不嫩低于2G
#创建statefulset
[root@master01 statefulset]# kubectl apply -f statefulset.yaml
service/myapp-svc created
statefulset.apps/myapp created
[root@master01 statefulset]# kubectl get sts -n test
NAME READY AGE
myapp 3/3 51s
#可以查看是否创建成功或者失败的原因
[root@master01 statefulset]# kubectl describe sts myapp -n test
#查看svc,
[root@master01 statefulset]# kubectl get svc -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
myapp-svc ClusterIP None <none> 80/TCP 2m10s
#查看pod,pod的名字是由podname跟一个有序的数字组成。
[root@master01 statefulset]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-0 1/1 Running 0 4m19s 10.244.2.42 node01 <none> <none>
myapp-1 1/1 Running 0 4m17s 10.244.2.43 node01 <none> <none>
myapp-2 1/1 Running 0 4m15s 10.244.2.44 node01 <none> <none>
#查看pvc绑定了哪一个pv,
[root@master01 statefulset]# kubectl describe pods myapp-0 -n test
。。。。。。。。。。。。。。。
。。。。。。。。。。。。。。。
Volumes:
myappdata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: myappdata-myapp-0 #myapp-0的pod绑定了myappdata-myapp-0这个pvc
ReadOnly: false
default-token-q2q77:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-q2q77
Optional: false
QoS Class: BestEffort
#pvc的名字是由volumeClaimTemplates定义的name跟statefulset定义的podname加有序数字组成,
[root@master01 statefulset]# kubectl get pvc -n test -o wide
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
myappdata-myapp-0 Bound pv01 2Gi RWO 6m33s Filesystem
myappdata-myapp-1 Bound pv02 2Gi RWO 6m31s Filesystem
myappdata-myapp-2 Bound pv03 2Gi RWO 6m29s Filesystem
#如果删除statefulset的pod,也是有序的一个一个删除,
[root@master01 statefulset]# kubectl delete -f statefulset.yaml
service "myapp-svc" deleted
statefulset.apps "myapp" deleted
#有序的删除,
[root@master01 pv]# kubectl get pods -n test -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 11m
myapp-1 1/1 Running 0 11m
myapp-2 1/1 Running 0 11m
myapp-0 1/1 Terminating 0 11m
myapp-2 1/1 Terminating 0 11m
myapp-1 1/1 Terminating 0 11m
#那创建statefulset
[root@master01 statefulset]# kubectl apply -f statefulset.yaml
service/myapp-svc created
statefulset.apps/myapp created
[root@master01 pv]# kubectl get pods -n test -w
NAME READY STATUS RESTARTS AGE
myapp-0 0/1 Pending 0 0s
myapp-0 0/1 Pending 0 0s
myapp-0 0/1 ContainerCreating 0 0s
myapp-0 1/1 Running 0 3s
myapp-1 0/1 Pending 0 0s
myapp-1 0/1 Pending 0 0s
myapp-1 0/1 ContainerCreating 0 0s
myapp-1 1/1 Running 0 2s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 ContainerCreating 0 0s
myapp-2 1/1 Running 0 2s
#那现在再检查myapp-0绑定pvc
[root@master01 statefulset]# kubectl describe pods myapp-0 -n test
。。。。。。。。。。。。。。。。。。。
。。。。。。。。。。。。。。。。。。。
Volumes:
myappdata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: myappdata-myapp-0 #myapp-0的pod还是绑定了myappdata-myapp-0这个pvc
ReadOnly: false
default-token-q2q77:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-q2q77
Optional: false
#测试,由于pvc通过pv来创建给pod使用,默认pv上面是没有数据的,所以要创建一个文件,以myapp-0的pod为例。
上面查看myapp-0绑定在myappdata-myapp-0这个pvc上面
[root@master01 statefulset]# kubectl get pvc -n test|grep myappdata-myapp-0
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myappdata-myapp-0 Bound pv01 2Gi RWO 25m
#就在pv01上面创建文件,由于当前使用的nfs挂载方式,默认文件被覆盖了,
[root@yunwei pv01]# cd /data/volumes/pv01
[root@yunwei pv01]# echo myapp-0 > index.html
[root@yunwei pv01]# cat index.html
myapp-0
[root@master01 statefulset]# kubectl get pods -n test -o wide |grep myapp-0
myapp-0 1/1 Running 0 8m45s 10.244.2.48 node01 <none> <none>
[root@master01 statefulset]# curl 10.244.2.48/index.html
myapp-0
#进入容器查看容器的hostname
[root@master01 statefulset]# kubectl exec myapp-0 -n test -- sh -c 'hostname'
myapp-0
[root@master01 statefulset]# kubectl exec myapp-1 -n test -- sh -c 'hostname'
myapp-1
[root@master01 statefulset]# kubectl exec myapp-2 -n test -- sh -c 'hostname'
myapp-2
总结:
Statefulset名称为myapp 三个Pod副本: myapp-0,myapp-1,myapp-2,
volumeClaimTemplates名称为:myappdata,那么自动创建出来的PVC名称为: myappdata-myapp-[0-2],为每个Pod创建一个PVC。
匹配Pod name(网络标识)的模式为:
1. (statefulset名称)−(序号),比如上面的示例:myapp-0,myapp-1,myapp-2
2. StatefulSet为每个Pod副本创建了一个DNS域名,这个域名的格式为: $(podname).(headless server name),
也就意味着服务间是通过Pod域名来通信而非Pod IP,因为当Pod所在Node发生故障时,Pod会被飘移到其它Node上,
Pod IP会发生变化,但是Pod域名不会有变化。
3. StatefulSet使用Headless服务来控制Pod的域名,这个域名的FQDN为:(pod_name).(servicename).(namespace).svc.cluster.local,
其中,“cluster.local”指的是集群的域名。
4. 根据volumeClaimTemplates,为每个Pod创建一个pvc,pvc的命名规则匹配模式:
(volumeClaimTemplates.name)-(pod_name),比如上面的volumeMounts.name=myappdata, Pod name=myapp-[0-2],
因此创建出来的PVC是:myappdata-myapp-[0-2]。删除Pod不会删除其pvc,手动删除pvc将自动释放pv。
#进入容器进行域名解析,
[root@master01 statefulset]# kubectl exec -it myapp-0 -n test -- /bin/sh
/ #
/ # nslookup myapp-0.myapp-svc.test.svc.cluster.local.
nslookup: can't resolve '(null)': Name does not resolve
Name: myapp-0.myapp-svc.test.svc.cluster.local.
Address 1: 10.244.2.48 myapp-0.myapp-svc.test.svc.cluster.local
/ #
/ # nslookup myapp-1.myapp-svc.test.svc.cluster.local.
nslookup: can't resolve '(null)': Name does not resolve
Name: myapp-1.myapp-svc.test.svc.cluster.local.
Address 1: 10.244.2.49 myapp-1.myapp-svc.test.svc.cluster.local
/ #
/ # nslookup myapp-2.myapp-svc.test.svc.cluster.local.
nslookup: can't resolve '(null)': Name does not resolve
Name: myapp-2.myapp-svc.test.svc.cluster.local.
Address 1: 10.244.2.50 myapp-2.myapp-svc.test.svc.cluster.local
/ #
#statefulset可以通过pvc做持久化存储,保证pod重新被调度后还是能访问到相同的持久化数据,在删除pod时,pvc不会被删除。
[root@master02 ~]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-0 1/1 Running 0 109m 10.244.2.48 node01 <none> <none>
myapp-1 1/1 Running 0 109m 10.244.2.49 node01 <none> <none>
myapp-2 1/1 Running 0 109m 10.244.2.50 node01 <none> <none>
[root@master02 ~]#
[root@master02 ~]# curl 10.244.2.48
myapp-0
#删除myapp-0的pod为例。
[root@master02 ~]# kubectl delete pods myapp-0 -n test
pod "myapp-0" deleted
[root@master02 ~]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-0 1/1 Running 0 16s 10.244.2.51 node01 <none> <none>
myapp-1 1/1 Running 0 110m 10.244.2.49 node01 <none> <none>
myapp-2 1/1 Running 0 110m 10.244.2.50 node01 <none> <none>
#访问新的myapp-0的pod
[root@master02 ~]# curl 10.244.2.51
myapp-0