Kubernetes实录系列记录文档完整目录参考: Kubernetes实录-目录
相关记录链接地址 :
Kubernetes平台上应用的自动化横着扩展(水平自动伸缩)是通过HPA(Horizontal Pod Autoscaler)来实现的,基于观测CPU使用率(v1版本,v2beta版本也支持memory或者其他自定义性能属性),当业务负载上升超过HPA设定值,创建新的POD保障业务对资源的需求,当负载下载后可以通过销毁POD是否资源来提高利用率。
当前Kubernetes支持的HPA功能API资源版本信息
kubectl version
v1.18.5
kubectl api-versions |grep autoscaling
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
本记录是对kubernetes HPA功能的验证,参考kubernetes官方文档,使用的是官方文档提供的镜像php-apache进行测试。
HPA控制器的工作流程(V1版本)
1. 创建HPA资源对象,关联对应资源例如Deployment,设定目标CPU使用率阈值,最大,最小replica数量。
前提:pod一定要设置资源限制,参数request,HPA才会工作。
2. HPA控制器每隔15秒钟(可以通过设置controller manager的–horizontal-pod-autoscaler-sync-period参数设定,默认15s)通过观测metrics值获取资源使用信息
3. HPA控制器将获取资源使用信息与HPA设定值进行对比,计算出需要调整的副本数量
4. 根据计算结果调整副本数量,使得单个POD的CPU使用率尽量逼近期望值,但不能照顾设定的最大,最小值。
5. 以上2,3,4周期循环
- HPA控制器观测资源使用率并作出决策是有周期的,执行是需要时间的,在执行自动伸缩过程中metrics不是静止不变的,可能降低或者升高,如果执行太频繁可能导致资源的使用快速抖动,因此控制器每次决策后的一段时间内不再进行新的决策。对于扩容这个时间是3分钟,缩容则是5分钟,对应调整参数
--horizontal-pod-autoscaler-downscale-delay
--horizontal-pod-autoscaler-upscale-delay
- 自动伸缩不是一次到位的,而是逐渐逼近计算值,每次调整不超过当前副本数量的2倍或者1/2
一、Kubernetes环境
1.1 kubernetes环境信息
kubernetes的部署可以参考本系列文档,具体进入使用kubeadm配置HA模式kubernetes集群 查看。
主机名称 | ip地址 | 操作系统 | 角色 | 软件版本 | 备注 |
---|---|---|---|---|---|
ejucsmaster-shqs-1 | 10.99.12.201 | CentOS 7.8 | proxy, master | 1.18.x | |
ejucsmaster-shqs-2 | 10.99.12.202 | CentOS 7.8 | proxy, master | 1.18.x | |
ejucsmaster-shqs-3 | 10.99.12.203 | CentOS 7.8 | proxy, master | 1.18.x | |
ejucsnode-shqs-1 | 10.99.12.204 | CentOS 7.8 | worker | 1.18.x | |
ejucsnode-shqs-2 | 10.99.12.205 | CentOS 7.8 | worker | 1.18.x | |
ejucsnode-shqs-2 | 10.99.12.206 | CentOS 7.8 | worker | 1.18.x |
1.2 metrics server
kubernetes集群需要配置好metrics server,配置参考文档Kubernetes部署metrics-server
二、配置HPA实现应用横向扩展
2.1 配置启动deployment:php-apache
- 编辑deployment文件
# cat hpa-deployment.ymal
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
labels:
app: hpa-test
spec:
replicas: 1
selector:
matchLabels:
name: php-apache
app: hpa-test
template:
metadata:
labels:
name: php-apache
app: hpa-test
spec:
containers:
- name: php-apache
image: mirrorgooglecontainers/hpa-example:latest
ports:
- containerPort: 80
name: http
protocol: TCP
resources:
requests:
cpu: 0.005
memory: 64Mi
limits:
cpu: 0.05
memory: 128Mi
- 执行deployment
kubectl apply -f hpa-deployment.ymal
deployment.apps/php-apache created
2.2 配置service:php-apache-svc
- 编辑service文件
# cat hpa-svc.yaml
---
apiVersion: v1
kind: Service
metadata:
name: php-apache-svc
labels:
app: hpa-test
spec:
selector:
name: php-apache
app: hpa-test
ports:
- name: http
port: 80
protocol: TCP
- 执行service
kubectl apply -f hpa-svc.yaml
service/php-apache-svc created
2.3 配置hpa:php-apache-hpa
- 编辑hpa文件
# cat hpa-hpa.yaml
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: php-apache-hpa
labels:
app: hpa-test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
targetCPUUtilizationPercentage: 50
- 执行hpa
kubectl apply -f hpa-hpa.yaml
horizontalpodautoscaler.autoscaling/php-apache-hpa created
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/php-apache-hpa Deployment/php-apache <unknown>/50% 1 10 0 3s
#这里TARGETS显示<unknown>/50%,是因为观测的pod资源使用率尚未准备完成,等一会再执行就会显示正常的数据。
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/php-apache-hpa Deployment/php-apache 20%/50% 1 10 1 21s
# 如果一直是unkown,则可以出现问题了,可以通过如下方式查询问题
1. kubectl describe hpa php-apache-hpa
2. kubectl logs -f {metrics-server-pod} -n kube-system
三,压力测试,观测HPA效果
3.1 生成一个压测客户端,持续压力测试
kubectl run --generator=run-pod/v1 -i --tty load-generator --image=busybox /bin/sh
# while true; do wget -q -O- http://php-apache-svc.default.svc.cluster.local; done
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!
3.2 压测下,观测结果
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 800%/50% 1 10 1 27m
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 1000%/50% 1 10 2 27m
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 1000%/50% 1 10 4 27m
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 1000%/50% 1 10 8 27m
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 120%/50% 1 10 10 27m
kubectl get deployment php-apache
NAME READY UP-TO-DATE AVAILABLE AGE
php-apache 10/10 10 10 28m
kubectl describe hpa php-apache-hpa
Name: php-apache-hpa
Namespace: default
Labels: app=hpa-test
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"app":"hpa-test"},"name":"php-apach...
CreationTimestamp: Thu, 14 Feb 2019 01:01:01 -0500
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 20% (1m) / 50%
Min replicas: 1
Max replicas: 10
Deployment pods: 10 current / 10 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 10; reason: cpu resource utilization (percentage of request) above target
结论:随着压力测试进行,deployment下pod的CPU使用率增加,超过HPA设定的百分比50%,之后逐次翻倍扩容replicaset。达到上限停止扩容。根据replicaset设置的request QoS逐渐稳定资源的使用率。
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 72%/50% 1 10 10 32m
3.3 停止压测
while true; do wget -q -O- http://php-apache-svc.default.svc.cluster.local; done
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!wget: can't connect to remote host (10.104.63.73): Connection refused
OK!OK!OK!OK!OK!OK!........OK!OK!OK! ^C
/ # exit
/ # Session ended, resume using 'kubectl attach load-generator -c load-generator -i -t' command when the pod is running
CPU使用率恢复到最初值20%,controller会周期观测,逐次缩容到最小值。
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 20%/50% 1 10 10 36m
#等待几分钟之后(默认5分钟),原因:
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 20%/50% 1 10 4 41m
#再次等待几分钟后(默认5分钟)
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 20%/50% 1 10 2 46m
#再次等待几分钟后(默认5分钟),稳定在最小副本数量
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache-hpa Deployment/php-apache 20%/50% 1 10 1 53m
四、其他
以上测试验证了HPA功能,使用的API版本是autoscaling/v1。通过kubectl api-versions可以查看到存在3个版本。v1版本只支持CPU,v2beta2版本支持多metrics(CPU,memory)以及自定义metrics。基于autoscaling/v2beta2的hpa yaml文件写法
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache-hpa
labels:
app: hpa-test
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50