Kubernetes实录-第一篇-集群部署配置(19) Kubernetes配置HPA实现应用横向扩展（autoscale）

Kubernetes实录系列记录文档完整目录参考: Kubernetes实录-目录

相关记录链接地址 ：

• 第13篇：Kubernetes部署metrics-server
• 第14篇：Kubernetes配置HPA实现应用横向扩展（autoscale ）

Kubernetes平台上应用的自动化横着扩展（水平自动伸缩）是通过HPA（Horizontal Pod Autoscaler）来实现的，基于观测CPU使用率(v1版本，v2beta版本也支持memory或者其他自定义性能属性)，当业务负载上升超过HPA设定值，创建新的POD保障业务对资源的需求，当负载下载后可以通过销毁POD是否资源来提高利用率。

当前Kubernetes支持的HPA功能API资源版本信息
kubectl version
v1.18.5
kubectl api-versions |grep autoscaling
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2

本记录是对kubernetes HPA功能的验证，参考kubernetes官方文档，使用的是官方文档提供的镜像php-apache进行测试。

HPA控制器的工作流程（V1版本）

1. 创建HPA资源对象，关联对应资源例如Deployment，设定目标CPU使用率阈值，最大，最小replica数量。
前提：pod一定要设置资源限制，参数request，HPA才会工作。
2. HPA控制器每隔15秒钟（可以通过设置controller manager的–horizontal-pod-autoscaler-sync-period参数设定，默认15s）通过观测metrics值获取资源使用信息
3. HPA控制器将获取资源使用信息与HPA设定值进行对比，计算出需要调整的副本数量
4. 根据计算结果调整副本数量，使得单个POD的CPU使用率尽量逼近期望值，但不能照顾设定的最大，最小值。
5. 以上2,3,4周期循环

1. HPA控制器观测资源使用率并作出决策是有周期的，执行是需要时间的，在执行自动伸缩过程中metrics不是静止不变的，可能降低或者升高，如果执行太频繁可能导致资源的使用快速抖动，因此控制器每次决策后的一段时间内不再进行新的决策。对于扩容这个时间是3分钟，缩容则是5分钟，对应调整参数
   --horizontal-pod-autoscaler-downscale-delay
--horizontal-pod-autoscaler-upscale-delay
2. 自动伸缩不是一次到位的，而是逐渐逼近计算值，每次调整不超过当前副本数量的2倍或者1/2

一、Kubernetes环境

1.1 kubernetes环境信息

kubernetes的部署可以参考本系列文档，具体进入使用kubeadm配置HA模式kubernetes集群 查看。

主机名称	ip地址	操作系统	角色	软件版本	备注
ejucsmaster-shqs-1	10.99.12.201	CentOS 7.8	proxy, master	1.18.x
ejucsmaster-shqs-2	10.99.12.202	CentOS 7.8	proxy, master	1.18.x
ejucsmaster-shqs-3	10.99.12.203	CentOS 7.8	proxy, master	1.18.x
ejucsnode-shqs-1	10.99.12.204	CentOS 7.8	worker	1.18.x
ejucsnode-shqs-2	10.99.12.205	CentOS 7.8	worker	1.18.x
ejucsnode-shqs-2	10.99.12.206	CentOS 7.8	worker	1.18.x

1.2 metrics server

kubernetes集群需要配置好metrics server，配置参考文档Kubernetes部署metrics-server

二、配置HPA实现应用横向扩展

2.1 配置启动deployment：php-apache

• 编辑deployment文件

# cat hpa-deployment.ymal

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
  labels:
    app: hpa-test
spec:
  replicas: 1
  selector:
    matchLabels:
      name: php-apache
      app: hpa-test
  template:
    metadata:
      labels:
        name: php-apache
        app: hpa-test
    spec:
      containers:
      - name: php-apache
        image: mirrorgooglecontainers/hpa-example:latest
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 0.005
            memory: 64Mi
          limits:
            cpu: 0.05
            memory: 128Mi

• 执行deployment

kubectl apply -f hpa-deployment.ymal
    deployment.apps/php-apache created

2.2 配置service：php-apache-svc

• 编辑service文件

# cat hpa-svc.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: php-apache-svc
  labels:
    app: hpa-test
spec:
  selector:
    name: php-apache
    app: hpa-test
  ports:
  - name: http
    port: 80
    protocol: TCP

• 执行service

kubectl apply -f hpa-svc.yaml
    service/php-apache-svc created

2.3 配置hpa:php-apache-hpa

• 编辑hpa文件

# cat hpa-hpa.yaml

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache-hpa
  labels:
    app: hpa-test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50

• 执行hpa

kubectl apply -f hpa-hpa.yaml
    horizontalpodautoscaler.autoscaling/php-apache-hpa created

kubectl get hpa
    NAME                                                 REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
    horizontalpodautoscaler.autoscaling/php-apache-hpa   Deployment/php-apache   <unknown>/50%   1         10        0          3s

#这里TARGETS显示<unknown>/50%，是因为观测的pod资源使用率尚未准备完成，等一会再执行就会显示正常的数据。
kubectl get hpa
    NAME                                                 REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
    horizontalpodautoscaler.autoscaling/php-apache-hpa   Deployment/php-apache   20%/50%   1         10        1          21s


# 如果一直是unkown，则可以出现问题了，可以通过如下方式查询问题
1. kubectl describe hpa php-apache-hpa
2. kubectl logs -f {metrics-server-pod} -n kube-system

三，压力测试，观测HPA效果

3.1 生成一个压测客户端,持续压力测试

kubectl run --generator=run-pod/v1 -i --tty load-generator --image=busybox /bin/sh
# while true; do wget -q -O- http://php-apache-svc.default.svc.cluster.local; done
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!

3.2 压测下，观测结果

kubectl get hpa
    NAME             REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
    php-apache-hpa   Deployment/php-apache   800%/50%   1         10        1          27m

kubectl get hpa
    NAME             REFERENCE               TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
    php-apache-hpa   Deployment/php-apache   1000%/50%   1         10        2          27m

kubectl get hpa
    NAME             REFERENCE               TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
    php-apache-hpa   Deployment/php-apache   1000%/50%   1         10        4          27m

kubectl get hpa
    NAME             REFERENCE               TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
    php-apache-hpa   Deployment/php-apache   1000%/50%   1         10        8          27m


kubectl get hpa
    NAME             REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
    php-apache-hpa   Deployment/php-apache   120%/50%   1         10        10         27m

kubectl get deployment php-apache
    NAME         READY   UP-TO-DATE   AVAILABLE   AGE
    php-apache   10/10   10           10          28m



kubectl describe hpa php-apache-hpa
	Name:                                                  php-apache-hpa
	Namespace:                                             default
	Labels:                                                app=hpa-test
	Annotations:                                           kubectl.kubernetes.io/last-applied-configuration:
	                                                         {"apiVersion":"autoscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"app":"hpa-test"},"name":"php-apach...
	CreationTimestamp:                                     Thu, 14 Feb 2019 01:01:01 -0500
	Reference:                                             Deployment/php-apache
	Metrics:                                               ( current / target )
	  resource cpu on pods  (as a percentage of request):  20% (1m) / 50%
	Min replicas:                                          1
	Max replicas:                                          10
	Deployment pods:                                       10 current / 10 desired
	Conditions:
	  Type            Status  Reason               Message
	  ----            ------  ------               -------
	  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
	  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
	  ScalingLimited  True    TooManyReplicas      the desired replica count is more than the maximum replica count
	Events:
	  Type    Reason             Age   From                       Message
	  ----    ------             ----  ----                       -------
	  Normal  SuccessfulRescale  11m   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
	  Normal  SuccessfulRescale  11m   horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
	  Normal  SuccessfulRescale  11m   horizontal-pod-autoscaler  New size: 10; reason: cpu resource utilization (percentage of request) above target

结论：随着压力测试进行，deployment下pod的CPU使用率增加，超过HPA设定的百分比50%,之后逐次翻倍扩容replicaset。达到上限停止扩容。根据replicaset设置的request QoS逐渐稳定资源的使用率。

kubectl get hpa
    NAME             REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
    php-apache-hpa   Deployment/php-apache   72%/50%   1         10        10         32m

3.3 停止压测

while true; do wget -q -O- http://php-apache-svc.default.svc.cluster.local; done
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!wget: can't connect to remote host (10.104.63.73): Connection refused
OK!OK!OK!OK!OK!OK!........OK!OK!OK! ^C
/ # exit
/ # Session ended, resume using 'kubectl attach load-generator -c load-generator -i -t' command when the pod is running

CPU使用率恢复到最初值20%，controller会周期观测，逐次缩容到最小值。

kubectl get hpa
	NAME             REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
	php-apache-hpa   Deployment/php-apache   20%/50%   1         10        10         36m

#等待几分钟之后(默认5分钟)，原因:
kubectl get hpa
NAME             REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache-hpa   Deployment/php-apache   20%/50%   1         10        4          41m

#再次等待几分钟后(默认5分钟)
kubectl get hpa
NAME             REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache-hpa   Deployment/php-apache   20%/50%   1         10        2          46m

#再次等待几分钟后（默认5分钟），稳定在最小副本数量
kubectl get hpa
NAME             REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache-hpa   Deployment/php-apache   20%/50%   1         10        1          53m

四、其他

以上测试验证了HPA功能，使用的API版本是autoscaling/v1。通过kubectl api-versions可以查看到存在3个版本。v1版本只支持CPU，v2beta2版本支持多metrics(CPU，memory)以及自定义metrics。基于autoscaling/v2beta2的hpa yaml文件写法

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache-hpa
  labels:
    app: hpa-test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50