11-k8s结合Metrics-Server实现自动伸缩

k8s结合Metrics-Server实现自动伸缩

资源限制的介绍

requests.cpu是命名空间中所有容器的最大组合CPU请求（以毫秒为单位）。 在上面的示例中，您可以拥有50个具有10m请求的容器，5个具有100m请求的容器，甚至一个具有500m请求的容器。 只要命名空间中请求的总CPU和小于500m！

requests.memory是命名空间中所有容器的最大组合内存请求。 在上面的示例中，您可以拥有50个具有2MiB请求的容器，5个具有20MiB请求的容器，甚至是具有100MiB请求的单个容器。 只要命名空间中请求的总内存小于100MiB！

limits.cpu是命名空间中所有容器的最大组合CPU限制。 它就像requests.cpu，但是这里指的是限制。

limits.memory是命名空间中所有容器的最大组合内存限制。 它就像requests.memory，但是同样地这里指的是限制。

如果您使用的是生产和开发命名空间（与每个团队或服务的命名空间不同），则常见的模式是在生产命名空间上没有配额，在开发命名空间上则是没有严格的配额。 这使得生产能够在流量激增的情况下获取所需的所有资源。

Metrics-Server概念介绍

Metrics-Server是集群核心监控数据的聚合器，在k8s早期版本中，对资源的监控使用的是heapster的资源监控工具。但是从 Kubernetes 1.8 开始，Kubernetes 通过 Metrics API 获取资源使用指标，例如容器 CPU 和内存使用情况。这些度量指标可以由用户直接访问，例如通过使用kubectl top 命令，或者使用集群中的控制器，,因为k8s的api-server将所有的数据持久化到了etcd中，显然k8s本身不能处理这种频率的采集，而且这种监控数据变化快且都是临时数据，因此需要有一个组件单独处理他们

环境

注意：此环境为kubeadm搭建
如果是minikube搭建：需要开启Metrics-Server即可

minikube addons list
minikube addons enable metrics-server

[root@k8s-master ~]# docker -v
Docker version 19.03.5, build 633a0ea
[root@k8s-master ~]# kubelet --version
Kubernetes v1.18.0

一、修改配置安装Metrics-Server

1.检查 API Server 是否开启了 Aggregator Routing：查看 API Server 是否具有 --enable-aggregator-routing=true 选项。

如果没有则需要添加

[root@k8s-master ~]# ps -ef | grep apiserver
root      16619  16581  3 18:51 ?        00:02:06 kube-apiserver --advertise-address=192.168.10.101 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root      36741  35704  0 19:54 pts/0    00:00:00 grep --color=auto apiserver
#此处没有需要添加 

2.修改每个 API Server 的 kube-apiserver.yaml 配置开启 Aggregator Routing：修改 manifests 配置后 API Server 会自动重启生效。

[root@k8s-master ~]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
 21     - --enable-bootstrap-token-auth=true
 22     - --enable-aggregator-routing=true		#在21行下方添加此行内容
 23     - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
：wq			#保存退出

二、安装metrics-server(v0.3.6)

1、下载yaml文件

方法一：互联网下载：

[root@k8s-master ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
修改配置文件
[root@k8s-master ~]# vim components.yaml
 85       - name: metrics-server
 86         image: k8s.gcr.io/metrics-server-amd64:v0.3.6		
 #将86行换为修改为阿里云的镜像地址
 86		   image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
 #在90行下方添加
 88         args:
 89           - --cert-dir=/tmp
 90           - --secure-port=4443
 91           - /metrics-server					#添加内容
 92           - --kubelet-preferred-address-types=InternalIP	#添加内容
 93           - --kubelet-insecure-tls			#添加内容

方法二：手动编写（不需要修改）ps:因为我已经修改好了

[root@k8s-master ~]# vim components.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - /metrics-server
          - --kubelet-preferred-address-types=InternalIP
          - --kubelet-insecure-tls
        ports:
        - name: main-port
          containerPort: 4443
          protocol: TCP
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: "amd64"
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

2、安装

[root@k8s-master ~]# kubectl apply -f components.yaml
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

3、查看metrics-server服务状态

[root@k8s-master ~]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-59dd47f7d9-dhsjw      1/1     Running   0          34s

4、检查接口是否有异常

[root@k8s-master ~]# kubectl describe apiservice v1beta1.metrics.k8s.io
Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       <none>
Annotations:  API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2021-03-01T12:18:14Z
  Resource Version:    13106
  Self Link:           /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  UID:                 0f3e9d6f-8618-4914-a687-9d5853b9a940
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2021-03-01T12:18:18Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

5、执行以下命令，检查节点占用性能情况。

[root@k8s-master ~]#  kubectl top nodes		#查看node节点负载情况
NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master   93m          4%     1176Mi          68%
k8s-node01   30m          1%     848Mi           49%
k8s-node02   20m          1%     850Mi           49%
[root@k8s-master ~]# kubectl top pods		#使用次命令需要多等待一会，默认采集信息为每分钟采集一次
error: metrics not available yet
[root@k8s-master ~]# sleep 180 && kubectl top pods		#等待180秒进行查看，前提是需要有pod在运行
NAME            CPU(cores)   MEMORY(bytes)
rc-demo-chw5t   0m           4Mi
rc-demo-mmzvc   0m           3Mi
rc-demo-v289z   0m           3Mi
 # MEMORY(bytes) 内存信息   CPU(cores) cpu信息

二、生成测试pod

这里同时创建Service，是为了后续的模拟压力测试。

[root@k8s-master ~]# vim Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      run: php-apache
  replicas: 1
  template:
    metadata:
      labels:
        run: php-apache
    spec:
      containers:
        - name: php-apache
          image: "registry.cn-shenzhen.aliyuncs.com/cookcodeblog/hpa-example:latest"
          ports:
            - containerPort: 80
          resources:			#限制指标
            limits:
              cpu: 500m
            requests:
              cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  labels:
    run: php-apache
spec:
  ports:
    - port: 80
  selector:
    run: php-apache

值得注意的是：要监控的pod必须进行资源限制：限制指标

创建deployment和servcie：

[root@k8s-master ~]# kubectl create -f Deployment.yaml
deployment.apps/php-apache created
service/php-apache created

验证:

[root@k8s-master ~]# kubectl get deployment php-apache
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
php-apache   1/1     1            1           30s
[root@k8s-master ~]# kubectl get svc php-apache
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
php-apache   ClusterIP   10.102.178.170   <none>        80/TCP    36s
[root@k8s-master ~]# kubectl get pods -l run=php-apache -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE  NOMINATED NODE   READINESS GATES
php-apache-6b79cc8c8f-rtrv8   1/1     Running   0          55s   10.244.2.4   k8s-node02  <none>           <none>

三、创建HPA

[root@k8s-master ~]# kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
horizontalpodautoscaler.autoscaling/php-apache autoscaled
#默认创建的HPA名称和需要自动伸缩的对象名一致
# 可以通过--name来指定HPA名称
# 删除 hpa 
# kubectl delete horizontalpodautoscalers.autoscaling {Deploymeng名称}

为deployment php-apache 创建HPA，其中最小副本数为1，最大副本数为10，保持该deployment的所有Pod的平均CPU使用率不超过50%。

在本例中，deployment的pod的resources.request.cpu为200m （200 milli-cores vCPU)，所以HPA将保持所有Pod的平均CPU使用率不超过100m。

可以通过kubectl top pods查看pod的CPU使用情况。

查看HPA：

[root@k8s-master ~]# kubectl get hpa php-apache
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          82s
#此时已经可以监控php-apache的CPU信息

如果TARGETS列值格式为<acutal>/<expected>, 如果actual值总是为unkown，则表示无法从Metrics Server中获取指标值。说明Metrics Server有问题

#HPA默认每15秒从Metrics Server取一下指标来判断是否要自动伸缩。
The Horizontal Pod Autoscaler is implemented as a control loop, with a period controlled by the controller manager’s --horizontal-pod-autoscaler-sync-period flag (with a default value of 15 seconds).
#------------------------------------------------------------------------------------
#Metrics Server采集指标的默认间隔为60秒。
Default 60 seconds, can be changed using metrics-resolution flag. We are not recommending setting values below 15s, as this is the resolution of metrics calculated within Kubelet.

四、模拟增加负载

打开一个新的终端，创建一个临时的pod load-generator，并在该pod中向php-apache服务发起不间断循环请求，模拟增加php-apache的负载（CPU使用率）。

[root@k8s-master ~]# kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
If you don't see a command prompt, try pressing enter.
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!
#使用busybox 死循环每0.01秒请求一次网站页面

模拟压力测试几分钟后，观察HPA：

[root@k8s-master ~]# while sleep 5; do kubectl get hpa php-apache; done
NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   250%/50%   1         10        4          12m
#使用循环语句每5秒查看一次HPA的信息
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   57%/50%   1         10        5          14m
#经过一段时间可以看出
#------------------------------------参数解释-----------------------------------
#TARGETS:平均负载57%，设置的平均负载大于50%就会增加pod数量
#MINPODS ：最少pod的副本数量为1
#MAXPODS ：最多pod的副本数量为10
#REPLICAS：当前的pod副本数量为5
#AGE	：运行时间
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   49%/50%   1         10        7          19m
#又经过一段时间当前的pod副本数量为7台
#此时的TARGETS不大于50就不会在增加pod副本数量了

查看当前pod

[root@k8s-master ~]# kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
load-generator                1/1     Running   0          9m43s
php-apache-6b79cc8c8f-6gcsf   1/1     Running   0          9m7s
php-apache-6b79cc8c8f-jkz7q   1/1     Running   0          9m7s
php-apache-6b79cc8c8f-mzkwc   1/1     Running   0          7m6s
php-apache-6b79cc8c8f-n2hkk   1/1     Running   0          7m6s
php-apache-6b79cc8c8f-q6qg5   1/1     Running   0          9m7s
php-apache-6b79cc8c8f-qjbtl   1/1     Running   0          8m52s
php-apache-6b79cc8c8f-rtrv8   1/1     Running   0          23m

也就是HPA通过自动扩容到7个副本，来分摊了负载，使得所有Pod的平均CPU使用率保持（近似）在目标值。

可以通过kubectl describe hpa php-apache查看HPA自动伸缩的事件。

[root@k8s-master ~]# kubectl describe hpa php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 01 Mar 2021 20:34:46 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  44% (88m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       7 current / 7 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  11m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  11m    horizontal-pod-autoscaler  New size: 5; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  9m36s  horizontal-pod-autoscaler  New size: 7; reason: cpu resource utilization (percentage of request) above target
  #信息为当前cpu使用百分比大于设置阈值，触发自动伸缩，伸缩为7台

五、模拟减少负载

在运行load-generator的Terminal，按下Ctrl + C来终止进程

OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!^C
pod "load-generator" deleted
pod default/load-generator terminated (Error)
[root@k8s-master ~]#

继续观察HPA

php-apache   Deployment/php-apache   0%/50%    1         10        7          27m
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        7          27m
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        7          28m
#此时cpu平均负载为0但是要过了伸缩冷却时间5分钟后才能进行弹性伸缩

#如果观察HPA没有scale down，需要再等待一段时间。Kuberntes为了保证缩容时业务不中断，和防止频繁伸缩导致系统抖动，scaledown一次后需要等待一段时间才能再次scaledown，也叫伸缩冷却(cooldown)。默认伸缩冷却时间为5分钟。

--horizontal-pod-autoscaler-downscale-stabilization: The value for this option is a duration that specifies how long the autoscaler has to wait before another downscale operation can be performed after the current one has completed. The default value is 5 minutes (5m0s).

等待伸缩冷却时间为5分钟过后进行查看

[root@k8s-master ~]# kubectl get hpa php-apache
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          34m
#因为平均负载小于阈值，此时已经由原来的7台缩至1台，

通过kubectl describe hpa php-apache查看HPA自动伸缩的事件，可以看到“horizontal-pod-autoscaler New size: 1; reason: All metrics below target”的事件。

[root@k8s-master ~]# kubectl describe hpa php-apache
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 01 Mar 2021 20:34:46 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count fromcpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  23m    horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  23m    horizontal-pod-autoscaler  New size: 5; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  21m    horizontal-pod-autoscaler  New size: 7; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  3m23s  horizontal-pod-autoscaler  New size: 1; reason: All metrics below targe
  #日志中显示因为所有指标都低于阈值，触发伸缩，由7台伸缩至一台