Kubernetes学习（七）补充：基于自定义指标进行扩缩容

资源指标只包含CPU、内存，一般来说也够了。但如果想根据自定义指标:如请求qps/5xx错误数来实现HPA，就需要使用自定义指标了，目前比较成熟的实现是 Prometheus Custom Metrics。自定义指标由Prometheus来提供，再利用k8s-prometheus-adpater聚合到apiserver，实现和核心指标（metric-server)同样的效果。

1、部署Prometheus

自行部署Prometheus，需要采集Pod指标

2、部署 Custom Metrics Adapter

prometheus采集到的metrics并不能直接给k8s用，因为两者数据格式不兼容，还需要另外一个组件(k8s-prometheus-adpater)，将prometheus的metrics 数据格式转换成k8s API接口能识别的格式，转换以后，因为是自定义API，所以还需要用Kubernetes aggregator在主APIServer中注册，以便直接通过/apis/来访问。

GitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using PrometheusGitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using PrometheusGitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using Prometheus

该 PrometheusAdapter 有一个稳定的Helm Charts，我们直接使用。镜像可以替换成国内镜像。

验证适配器是否注册到APIServer：

[root@master chart]# kubectl get apiservices |grep custom 
v1beta1.custom.metrics.k8s.io          kube-system/prometheus-adapter   True        3m56s
[root@master chart]#

3、部署测试应用

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: metrics-app
  name: metrics-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: metrics-app
  template:
    metadata:
      labels:
        app: metrics-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "80"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - image: art.local:8081/docker-local/lizhenliang/metrics-app
        name: metrics-app
        ports:
        - name: web
          containerPort: 80
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-app
  labels:
    app: metrics-app
spec:
  type: NodePort
  ports:
  - name: web
    port: 80
    targetPort: 80
    nodePort: 30099
  selector:
    app: metrics-app

 该metrics-app暴露了一个Prometheus指标接口，提供2个指标：

http_requests_total
http_requests_per_second

4、创建自定义HPA策略

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: metrics-app-hpa-custom
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: metrics-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: 800m

[root@master k8s]# kubectl get hpa
NAME                     REFERENCE                TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa-custom   Deployment/metrics-app   <unknown>/800m   1         10        2          27s
[root@master k8s]#

此时适配器还不知道要什么指标（http_requests_per_second），HPA也就获取不到Pod提供指标。

5、配置适配器收集特定的指标

编辑PrometheusAdapter创建的prometheus-adapter Config Map：

新增查询规则

- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

重新部署PrometheusAdapter后，再次查看hpa，此时能够获取到自定义指标

[root@master log]# kubectl get hpa
NAME                     REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa-custom   Deployment/metrics-app   499m/800m   1         10        2          76m
[root@master log]#

6、压测，观察自定义指标扩缩容

[root@localhost ~]# ab -n 100000 -c 100 http://179.220.56.232:30099/metrics

[root@master ~]# kubectl get hpa
NAME                     REFERENCE                TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa-custom   Deployment/metrics-app   163846m/800m   1         10        10         5h46m
[root@master ~]# kubectl describe hpa metrics-app-hpa-custom 
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                  metrics-app-hpa-custom
Namespace:                             default
Labels:                                <none>
Annotations:                           <none>
CreationTimestamp:                     Mon, 20 Mar 2023 10:01:59 +0800
Reference:                             Deployment/metrics-app
Metrics:                               ( current / target )
  "http_requests_per_second" on pods:  499m / 800m
Min replicas:                          1
Max replicas:                          10
Deployment pods:                       3 current / 3 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from pods metric http_requests_per_second
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  24m   horizontal-pod-autoscaler  New size: 4; reason: pods metric http_requests_per_second above target
  Normal  SuccessfulRescale  24m   horizontal-pod-autoscaler  New size: 8; reason: pods metric http_requests_per_second above target
  Normal  SuccessfulRescale  24m   horizontal-pod-autoscaler  New size: 10; reason: pods metric http_requests_per_second above target
  Normal  SuccessfulRescale  17m   horizontal-pod-autoscaler  New size: 7; reason: All metrics below target
  Normal  SuccessfulRescale  12m   horizontal-pod-autoscaler  New size: 5; reason: All metrics below target
  Normal  SuccessfulRescale  7m1s  horizontal-pod-autoscaler  New size: 4; reason: All metrics below target
  Normal  SuccessfulRescale  2m1s  horizontal-pod-autoscaler  New size: 3; reason: All metrics below targe