解决kubernetes中hpa没生效问题

前言

今天我正在专心致志的敲go代码的bug，突然同事问到我们现在的服务数量太多了，同一个服务的实例pod数可能达到了十几个，能不能放在一个pod中，跑多个container的形式运行。当时我想了一下，就回答了：放在一个pod里边，这些container管理起来可能比较复杂一些，pod挂掉，所有的container都卒了，粒度太大。然后正准备继续写bug的时候，突然想到同事这么问的目的可能是性能优化，自然而然的就想到hpa了，联想到istio和knative的hpa问题还没解决，卒，好吧，来解决问题吧。

问题

这是个什么问题尼？简单来说就是项目中用到了istio和knative，但是这框架的hpa没生效，有同学可能不知道hpa是啥? hpa就是Horizontal Pod Autoscaler的缩写，水平pod自动扩容器。也就是可以根据cpu和内存指标等，自动扩缩容。但问题是这个hpa没有生效，所以导致了我们在做压力测试找系统瓶颈的时候，需要手动调整deployment的replica值到一个合适的大小，既不能占用太多资源，又不能是性能瓶颈，所以找到合适的值还是颇费了一番功夫的。当时其实就遇到这个问题了，简单看了一眼，好像是少了一个服务，但是这个服务没有，所以hpa没生效，但是不是很好解决，并且当时还有别的事情在做，所以就没有时间做这个，现在刚好可以看看。

分析

首先习惯性的看一下hpa的情况。

[root@master knativetest]# kubectl get hpa -A
NAMESPACE          NAME                       REFERENCE                             TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
istio-system       cluster-local-gateway      Deployment/cluster-local-gateway      <unknown>/80%    1         5         1          33d
istio-system       istio-ingressgateway       Deployment/istio-ingressgateway       <unknown>/80%    4         5         4          33d
istio-system       istiod                     Deployment/istiod                     <unknown>/80%    1         5         1          33d
knative-serving    activator                  Deployment/activator                  <unknown>/100%   1         20        1          33d
tekton-pipelines   tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   <unknown>/100%   1         5         1          7d21h

可以观察到所有的TARGETS都是<unknown>，但是后边有%80、%100的阀值。初步可以判断是没有拿到当前值。
查看一下hpa的详情。

[root@master knativetest]# kubectl describe hpa/istio-ingressgateway -n istio-system 
Name:                                                  istio-ingressgateway
Namespace:                                             istio-system
Labels:                                                app=istio-ingressgateway
                                                       install.operator.istio.io/owning-resource=unknown
                                                       install.operator.istio.io/owning-resource-namespace=istio-system
                                                       istio=ingressgateway
                                                       istio.io/rev=default
                                                       operator.istio.io/component=IngressGateways
                                                       operator.istio.io/managed=Reconcile
                                                       operator.istio.io/version=1.8.0
                                                       release=istio
Annotations:                                           <none>
CreationTimestamp:                                     Fri, 20 Nov 2020 17:13:57 +0800
Reference:                                             Deployment/istio-ingressgateway
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 1%
Min replicas:                                          1
Max replicas:                                          5
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                     From                       Message
  ----     ------                   ----                    ----                       -------
  Warning  FailedGetResourceMetric  19m (x1734 over 5d18h)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
  Warning  FailedGetResourceMetric  4m37s (x13 over 16m)    horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

报错信息是不能够从metrcis api中拿到服务的指标。印证了是当前指标值没拿到的问题。之前做过基于promethes和cadivsor的直觉告诉我，应该是这样，没错。
于是拿着报错信息找了大佬Google帮忙了一下，0.00012s检索出来1000条记录，找了最上边的stackoverflow的文章喵了一喵，哦，还真的是需要部署一个服务，还顺藤摸瓜找到了GitHub地址，心里暗暗窃喜，原来这个简单，部署一个新的服务，这个服务会提供每个hpa所需要的当前值，那么这个服务是如何知道当前值的尼，很简单，这个服务会从每个pod的kubelet中拿到pod所使用的资源量。

metrics-server链接
metrics-server yaml链接

直接跑起来：

[root@master knativetest]# kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

想着问题应该解决了，然后接着查看

kubectl get hpa -A

还是一如既往的，额好吧，看看刚才一大堆的create，相应的资源有没有建立起来吧，果然。

[root@master knativetest]# kubectl get pods -A | grep metrics
kube-system        metrics-server-866b7d5b74-gkg75                0/1     CrashLoopBackOff   27         3h4m

服务没起来，看看到底为啥没起来呗。

[root@master knativetest]# kubectl logs pods/metrics-server-866b7d5b74-gkg75   -n kube-system
E1209 06:18:59.170461       1 server.go:132] unable to fully scrape metrics: unable to fully scrape metrics from node docker-desktop: unable to fetch metrics from node docker-desktop: Get "https://192.168.65.3:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 192.168.65.3 because it doesn't contain any IP SANs
I1209 06:18:59.249064       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1209 06:18:59.249155       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1209 06:18:59.249433       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1209 06:18:59.249486       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1209 06:18:59.250473       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1209 06:18:59.250499       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1209 06:18:59.252315       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1209 06:18:59.253795       1 secure_serving.go:197] Serving securely on [::]:4443
I1209 06:18:59.254096       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1209 06:18:59.350031       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I1209 06:18:59.351561       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I1209 06:18:59.352764       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I1209 06:19:27.547596       1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1209 06:19:27.548154       1 requestheader_controller.go:183] Shutting down RequestHeaderAuthRequestController
I1209 06:19:27.548216       1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1209 06:19:27.548312       1 dynamic_serving_content.go:145] Shutting down serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1209 06:19:27.548356       1 tlsconfig.go:255] Shutting down DynamicServingCertificateController
I1209 06:19:27.548519       1 secure_serving.go:241] Stopped listening on [::]:4443

又看见一个报错，关键字:x509: cannot validate certificate for 192.168.65.3 because it doesn’t contain any IP SANs 心中一惊，直觉告诉我，x509一般都是tls问题，岂不是https不能用，要改为http方式？
接着请Google老大哥帮忙，果然stackoverflow出大神：

- --kubelet-insecure-tls

伪yaml：

spec:
  template:
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --kubelet-insecure-tls
        image: naison/metrics-server:v0.4.1

的确是印证了自己的直觉，metrics的启动参数加配置，跳过tls，加了之后重新apply 一下yaml文件，果然就好了。

ps：一顿docker和scp操作，将gcr.io的镜像，转存到私有仓。为我的docker私有仓库打个广告: )

[root@master knativetest]# kubectl get pods -A | grep metrics
kube-system        metrics-server-866b7d5b74-gkg75                1/1     running   	0         3h4m

再看看hpa的状态。

naison@P_CAIWFENG-MB0 knativetest % kubectl get hpa -A
NAMESPACE          NAME                       REFERENCE                             TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
istio-system       cluster-local-gateway      Deployment/cluster-local-gateway      12%/80%          1         5         1          33d
istio-system       istio-ingressgateway       Deployment/istio-ingressgateway       11%/80%          4         5         4          33d
istio-system       istiod                     Deployment/istiod                     0%/80%           1         5         1          33d
knative-serving    activator                  Deployment/activator                  <unknown>/100%   1         20        1          33d
tekton-pipelines   tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   3%/100%          1         5         1          7d21h

但是为啥还有一个hpa没生效，又看了下hpa的官方文档，找到一句话：
Please note that if some of the Pod’s containers do not have the relevant resource request set, CPU utilization for the Pod will not be defined and the autoscaler will not take any action for that metric. See the algorithm details section below for more information about how the autoscaling algorithm works.

大意就是如果不设置container的request request值，那么hpa是无法生效的。
所以看了一言activator的deployment的yaml，果然没有，于是立马加上：

[root@master knativetest]# kubectl patch deployment activator  -p='{"spec":{"template":{"spec":{"containers":[{"name":"activator","resources":{"requests":{"cpu":"200m"}}}]}}}}' -n knative-serving
deployment.apps/activator patched

加了之后，立马查看结果，还是没变。
于是又喵了一眼hpa的文档。

The Horizontal Pod Autoscaler is implemented as a control loop, with a period controlled by the controller manager’s --horizontal-pod-autoscaler-sync-period flag (with a default value of 15 seconds).

哦，每15s一次，如果当前所有pod的指标平均值大于request的资源值，就需要扩容，计算公式为：

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

还没理解透彻嘛意思，十五秒过了，立马查看hpa的状态。果然好了(这里是个坑)

[root@master knativetest]# kubectl get hpa -A
NAMESPACE          NAME                       REFERENCE                             TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
istio-system       cluster-local-gateway      Deployment/cluster-local-gateway      16%/1000%   1         20        1          34d
istio-system       istio-ingressgateway       Deployment/istio-ingressgateway       13%/1000%   4         20        4          34d
istio-system       istiod                     Deployment/istiod                     0%/80%      1         20        1          34d
knative-serving    activator                  Deployment/activator                  53%/100%    1         20        1          34d
tekton-pipelines   tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   3%/100%     1         5         1          8d

果然，当我再次暗暗窃喜的时候，想着一切顺利解决的时候，想着进行一下测试吧，果然一测就出问题。

序章

出了什么问题尼？
使用ab压力测试云函数的url地址，发现扩容好多pod啊。

[root@master knativetest]# kubectl get pods -n istio-system
NAME                                     READY   STATUS    RESTARTS   AGE
cluster-local-gateway-5d7c8c9d69-fv7l2   1/1     Running   8          18d
istio-ingressgateway-844bf55fb4-d6wht    1/1     Running   0          5m58s
istio-ingressgateway-844bf55fb4-khrtc    1/1     Running   0          5m58s
istio-ingressgateway-844bf55fb4-nfchw    1/1     Running   0          5m58s
istio-ingressgateway-844bf55fb4-nfjg5    1/1     Running   0          3h7m
istio-ingressgateway-844bf55fb4-twcwc    1/1     Running   0          4m58s
istiod-cfcffb65d-j4cmj                   1/1     Running   8          18d

过了一会儿，istio-ingressgateway和cluster-local-gateway就扩容到18个了，但是压力才测试了一会儿，并且之前测试出来的最优性能时，istio-ingressgateway和cluster-local-gateway也只有四个。
到底是什么原因扩容了这么多尼？又难倒我了。
在大佬的提醒下，看看每个扩容出来的pod cpu占用是多少。
看了一下，占用不高，但是占用不高为啥还会继续扩容尼。
仔细看了一个hpa.yaml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: istio-ingressgateway
  namespace: istio-system
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: istio-ingressgateway
  targetCPUUtilizationPercentage: 80

这里的 targetCPUUtilizationPercentage: 80，是什么含义尼？是不是理解的pode的cpu使用阀值80尼？会想起了计算公式：

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

翻译一下：

需要的pod数 = 向下取整[当前pod数 * ( pod当前使用值 / pod request值)]

注意这里是pod，request值也就是resource的request值。
然后立马看看deployment的container中的resources标签值，100m
恍然大悟，原来是分母太小了，导致pod只用了一点儿资源，就扩容了。所以立即更改了

targetCPUUtilizationPercentage: 80

这里的数值，1000，

[root@master knativetest]# kubectl get hpa -A
NAMESPACE          NAME                       REFERENCE                             TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
istio-system       cluster-local-gateway      Deployment/cluster-local-gateway      16%/1000%   1         20        1          34d
istio-system       istio-ingressgateway       Deployment/istio-ingressgateway       13%/1000%   1         20        1          34d
istio-system       istiod                     Deployment/istiod                     0%/80%      1         20        1          34d
knative-serving    activator                  Deployment/activator                  53%/100%    1         20        1          34d
tekton-pipelines   tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   3%/100%     1         5         1          8d

这样再次压测的时候，就没有很快的扩容了。至此，问题算是完美解决了（其实这里的值1000，是否合理，有待验证）。

后记

整体来说，还是很好玩的，不过一次次的失败，一次次的分析问题，然后解决问题，还是很有成就感的。
还发现一个有意思的点，那就是安装了metrics-server组件后，命令：

[root@master knativetest]# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
master   401m         10%    1971Mi          72%       
[root@master knativetest]# kubectl top pods
NAME                                                CPU(cores)   MEMORY(bytes)   
helloworld-go-yhdwv-1-deployment-8684c55d8b-sg7zv   9m           36Mi

也可以正常使用了，赞👍

完