kubernetes基于k8s-prometheus-adapter实现 pod 的hpa(自动伸缩)

最新推荐文章于 2024-08-10 07:39:39 发布

qinzhao168

最新推荐文章于 2024-08-10 07:39:39 发布

阅读量7.1k

点赞数 1

分类专栏： kubetnetes 文章标签： kubernetes 自动伸缩pod k8s-

本文链接：https://blog.csdn.net/qq_21816375/article/details/82662897

版权

kubetnetes 专栏收录该内容

106 篇文章 8 订阅

订阅专栏

系统

[root@k8s-build ~]# uname -a 
Linux k8s-build 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

kubernetes的版本

[root@master-47-35 custom-metrics-api]# kubectl version 
Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

获取代码k8s-prometheus-adapter

yum install git -y 
cd /root/gopath/src/
git clone https://github.com/DirectXMan12/k8s-prometheus-adapter.git

切换分支到v0.2.1

 [root@k8s-build k8s-prometheus-adapter]# git checkout v0.2.1
Note: checking out 'v0.2.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD 目前位于 3fc2dbd... Explicitly build on tags in Travis
[root@k8s-build k8s-prometheus-adapter]# git status 
# 头指针分离于 v0.2.1
无文件要提交，干净的工作区

构建k8s-prometheus-adapter镜像

[root@k8s-build k8s-prometheus-adapter]# make docker-build
cp deploy/Dockerfile /tmp/tmp.FbNVQfpHCg
cd /tmp/tmp.FbNVQfpHCg && sed -i "s|BASEIMAGE|busybox|g" Dockerfile
docker run -it -v /tmp/tmp.FbNVQfpHCg:/build -v /root/gopath/src/k8s-prometheus-adapter:/go/src/github.com/directxman12/k8s-prometheus-adapter -e GOARCH=amd64 golang:1.10 /bin/bash -c "\
    CGO_ENABLED=0 go build -tags netgo -o /build/adapter github.com/directxman12/k8s-prometheus-adapter/cmd/adapter"
Unable to find image 'golang:1.10' locally
Trying to pull repository docker.io/library/golang ... 
1.10: Pulling from docker.io/library/golang
05d1a5232b46: Pull complete 
5cee356eda6b: Pull complete 
89d3385f0fd3: Pull complete 
80ae6b477848: Pull complete 
94ebfeaaddf3: Pull complete 
e132030a369d: Pull complete 
c67c5750c788: Pull complete 
Digest: sha256:322a64a16553516761f8e71eb94136140031e6a7fcfe794e579de14a7da21ee7
Status: Downloaded newer image for docker.io/golang:1.10
docker build -t directxman12/k8s-prometheus-adapter-amd64:latest /tmp/tmp.FbNVQfpHCg
Sending build context to Docker daemon 55.61 MB
Step 1/4 : FROM busybox
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
8c5a7da1afbc: Already exists 
Digest: sha256:032ddd66f10483436e8a252e69fdfd20d0164e9953585c10d378183a0924db34
Status: Downloaded newer image for docker.io/busybox:latest
 ---> e1ddd7948a1c
Step 2/4 : COPY adapter /
 ---> a39120e24ddb
Removing intermediate container 6358600308cb
Step 3/4 : USER 1001:1001
 ---> Running in 9755e39bed35
 ---> 718d846e413b
Removing intermediate container 9755e39bed35
Step 4/4 : ENTRYPOINT /adapter
 ---> Running in 68554cb44316
 ---> cb309c2eb2f4
Removing intermediate container 68554cb44316
Successfully built cb309c2eb2f4
rm -rf /tmp/tmp.FbNVQfpHCg

查看k8s-prometheus-adapter镜像

[root@k8s-build k8s-prometheus-adapter]# docker images | grep k8s-prometheus-adapter-amd64
directxman12/k8s-prometheus-adapter-amd64     latest                         cb309c2eb2f4        About a minute ago       56.8 MB

修改镜像tag,指定到私有的镜像仓库

[root@k8s-build k8s-prometheus-adapter]# docker tag cb309c2eb2f4 harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1
[root@k8s-build k8s-prometheus-adapter]# docker push harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1
The push refers to a repository [harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64]
d4821deae84a: Pushed 
f9d9e4e6e2f0: Pushed 
v0.2.1: digest: sha256:32ead17195963566630d9a8286773ddff92f9ccc126cf9a6e2cdcf686d163aef size: 739

生成名字叫cm-adapter-serving-certs的secret,值为serving.crt和 serving.key，实际上就是证书

证书生成请参考requestheader-authentication

这里有生成证书的脚本,供参考，这个证书生成脚本需要gcc golang环境，安装gcc go

yum install go gcc -y

脚本内容

[root@k8s-build certs]# cat gencerts.sh 
#!/usr/bin/env bash

# Detect if we are on mac or should use GNU base64 options
case $(uname) in
        Darwin)
            b64_opts='-b=0'
            ;; 
        *)
            b64_opts='--wrap=0'
esac

go get -v -u github.com/cloudflare/cfssl/cmd/...
export PATH=$GOPATH/bin:$PATH
export PURPOSE=metrics
openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout ${PURPOSE}-ca.key -out ${PURPOSE}-ca.crt -subj "/CN=ca"
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "${PURPOSE}-ca-config.json"

export SERVICE_NAME=custom-metrics-apiserver
export ALT_NAMES='"custom-metrics-apiserver.custom-metrics","custom-metrics-apiserver.custom-metrics.svc"'
echo "{\"CN\":\"${SERVICE_NAME}\", \"hosts\": [${ALT_NAMES}], \"key\": {\"algo\": \"rsa\",\"size\": 2048}}" | \
        cfssl gencert -ca=metrics-ca.crt -ca-key=metrics-ca.key -config=metrics-ca-config.json - | cfssljson -bare apiserver

cat <<-EOF > cm-adapter-serving-certs.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cm-adapter-serving-certs
  namespace: custom-metrics
data:
  serving.crt: $(base64 ${b64_opts} < apiserver.pem)
  serving.key: $(base64 ${b64_opts} < apiserver-key.pem)
EOF

生成证书

[root@k8s-build certs]# ./gencerts.sh 
github.com/cloudflare/cfssl (download)
Generating a 2048 bit RSA private key
.+++
......+++
writing new private key to 'metrics-ca.key'
-----
2018/08/25 20:36:58 [INFO] generate received request
2018/08/25 20:36:58 [INFO] received CSR
2018/08/25 20:36:58 [INFO] generating key: rsa-2048
2018/08/25 20:36:59 [INFO] encoded CSR
2018/08/25 20:36:59 [INFO] signed certificate with serial number 573231977150840198182111747492918974864906787988

查看脚本生成的内容

[root@k8s-build certs]# ls -l 
总用量 32
-rw-r--r-- 1 root root 1066 8月  25 20:36 apiserver.csr
-rw------- 1 root root 1679 8月  25 20:36 apiserver-key.pem
-rw-r--r-- 1 root root 1277 8月  25 20:36 apiserver.pem
-rw-r--r-- 1 root root 4053 8月  25 20:36 cm-adapter-serving-certs.yaml
-rwxr-xr-x 1 root root 1198 8月  25 20:36 gencerts.sh
-rw-r--r-- 1 root root   94 8月  25 20:36 metrics-ca-config.json
-rw-r--r-- 1 root root 1074 8月  25 20:36 metrics-ca.crt
-rw-r--r-- 1 root root 1704 8月  25 20:36 metrics-ca.key

查看证书内容

[root@master-47-35 custom-metrics]# cat cm-adapter-serving-certs.yaml
apiVersion: v1
data:
  serving.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHRENDQWdDZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFmTVIwd0d3WURWUVFEREJSc2IyTmgKYkdodmMzUkFNVFV4TXpjME5qZzBNREFlRncweE56RXlNakF3TlRFME1EQmFGdzB4T0RFeU1qQXdOVEUwTURCYQpNQjh4SFRBYkJnTlZCQU1NRkd4dlkyRnNhRzl6ZEVBeE5URXpOelEyT0RRd01JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW9mZE4zd1RqVldlZFo2YlBRN3ZrMTNPeUkwMEVwOGcvYzlTTCtneWMKQ3RHK3hYSkhrVlN4eW1zVlhsNnF4T2tWTWkrL2JqRFVqc1VsSnY5a2JjZkhHemYwYmxiQWdocVhvQ20ycjkvRwp0TVNlczlTL0NER1FNQUhPbkxsemdjVkxVQ1kyUVVXdS9HZDVwQU96dkJSUVBWczNBNUNZcDMzS0grTFVTTExOCnVNMjcvSkVnclUwczdBQ29ZaDk3KzQrSU1qb1pmbHRiK0JOQzRaU3BvemhQZ0EwQWhXRXl0SXkrRDdVWWdIR2cKNm8ybWJpTkV1QUNrQmk5TmFITm8zdGdieVN3OGQzYU9vd3JQQ3VlNEV0UTNIZGlNMHB5WUtuSUhsTFhyNjA1Sgpra25ENWs0OWFoaktQUFJhTlpjNlZpZlEvTzNiekVnNUwvVld6K2lneFJrSyt3SURBUUFCbzE4d1hUQU9CZ05WCkhROEJBZjhFQkFNQ0FxUXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBbEJnTlZIUkVFSGpBY2dnbHNiMk5oYkdodmMzU0NDV3h2WTJGc2FHOXpkSWNFZndBQUFUQU5CZ2txaGtpRwo5dzBCQVFzRkFBT0NBUUVBWERVV3ozVW0zVFg4VFJCVmQ1VVF5Z1ZhKzloMlRtVWJMU0JpdlBURDVRTWFBZUJzCkRYQU0vbTZrWXZIcFlTTi9MOWhraWd1QzhTazdlVFZqeWZabjlsYmNoSVV1NytoYys1UThJTDZVb0VaQ2cxVHQKNDNlYzBhZlc1YUU2bm5lejVhMWs3UDFhY0xhL1ZFUzVWM25wUmg2YUl4a0xVVHNMaUFsdGRGcURpeTF4aVlORgorL2pWMGFHRXhWWWJMbFVQMkIxaTlQODZneGFjNkJRTCtSQTNZZjFISWl5ZkliemMrbjV4Z2U1T0NIbC9XVjV1CkIzTHdZdkh1TGdpTDNZbyswN0gycWZ3MlNIUmZxdlJHdlpmS2tnWmcrTVQrZElYR3ZTTE0ybUhhdElxQnNXZEwKS1VFZnorREdYbzBIam1LTmhuWThRM0kwN1IxY29ta05KUFMxL3c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==  
  serving.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBb2ZkTjN3VGpWV2VkWjZiUFE3dmsxM095STAwRXA4Zy9jOVNMK2d5Y0N0Ryt4WEpICmtWU3h5bXNWWGw2cXhPa1ZNaSsvYmpEVWpzVWxKdjlrYmNmSEd6ZjBibGJBZ2hxWG9DbTJyOS9HdE1TZXM5Uy8KQ0RHUU1BSE9uTGx6Z2NWTFVDWTJRVVd1L0dkNXBBT3p2QlJRUFZzM0E1Q1lwMzNLSCtMVVNMTE51TTI3L0pFZwpyVTBzN0FDb1loOTcrNCtJTWpvWmZsdGIrQk5DNFpTcG96aFBnQTBBaFdFeXRJeStEN1VZZ0hHZzZvMm1iaU5FCnVBQ2tCaTlOYUhObzN0Z2J5U3c4ZDNhT293clBDdWU0RXRRM0hkaU0wcHlZS25JSGxMWHI2MDVKa2tuRDVrNDkKYWhqS1BQUmFOWmM2VmlmUS9PM2J6RWc1TC9WV3oraWd4UmtLK3dJREFRQUJBb0lCQURUci82L05neEtXNXFIRgpxWmV5UDFYKzRjZ1RGR0JXV0JWYTc5bm1ZTjVRSm91Umliejg5WkwwWm1rTmViTVpqZWwySEpDK3Q5Vm1xWHNTCm5CZ2EzUDRSekhaNU5qSE15cHAweHRLMHhkS3ZKRVFTVWRFMWtLMS9DYW5BRnZQaUtYZW81TFZ6TlhnWHYzRk8KSEY1cUxNVy9sanc5OVFOVllNUUdjK0xGT2Q4V0pRRm9KZVh3bFBKdC9vK3NWUCtHK3dtYmp3VUNtc3ZqTFc5NAoxbS9XeWtwVlMrMCt3QVRJTGJCeTJqUUVtK3hwZk1IT0xhU01Hem9oUnEySHFvY1d2NE10OWI4bmFPNURyQ3dCCmZIQnIxd3VtL3duWGVESUFQay8yVVYxSllzdDQveVFjSENUS1V0Qzh3WFd6V3RabEE3dFNmM1huNUM0MXRhakoKaTIxMzNuRUNnWUVBd1BUOVRUS3R4NzZHYVovZE5WWStLY1ExYnplQlA5NlI0SFp0M1d4NllUbEJZZ2oxQWg4dQpZVGJCYmhYdDExcTQ0b0h4eXk0eThCVC9xWU55dWhIUGJjMExCYW4zamtwRVZEZ3VaempMSHc3YUUwTlVkNFNDCmFzd0FjTGlnMEdiTU1RMVJnZndHdVpOc3ZXZ2wrVXZWdVB3SVRKUzBUYkdQeVVGTnpZSmlsUjBDZ1lFQTF1STMKaFB1VjBoOGUyZEVHT3g5L0FXb1hCU3lhajNVZ2xTV2VnNTNBOFJ1OU9POTl5OGxHSXZxUDdsVXpub01xZzlwMwpEdXF3TFdCSnQwYU11SG1yZVF6RzhiME9ieG1UTmwreTdUaTIwOU9nUTBaZ0ZhSFhtbng2UStaWlovNFd0UXlMCmxzUUxPMDNqcjUwb3dQV3FJSnRmdEVXckZsYXBCd01IdFBxY0hQY0NnWUJVT1J0dlhOV1p3RUh2VE1aRGg3ZHUKaDZSRFBWai9SZWtxcnNiWnhZWE5kaTVoMmtNSGdCREFnaVFBSnpZL3IrdlBncVRGNlE1MjJOT1hiRGJoblBDTApudHBXeUkrejNocmhIL3l4TllzRHJYbm11cGJkN0VyNHpyN0tkT3hNeXZJSjN5UmF0bVZCRDJhQmJqcFZESkxuClFhdGxzZXJabHMyS0xTVUd0UWpPeVFLQmdDTUVpbmVwYUNqdDJXRTFwd1pMZzY3L1V2RU02OE1sRzBJeTl6b0wKb3oybmY1YWZzS2hkRElqMnZDM2l4SjY0VVYxYlpMdVJrMVJSdExHaFZaMWFkbzhyZmVBM0p2c0Q5L0dNZHVRaApzZW1qT3lLN2ZUMEpPdEY2SjVLWkgwakxKM2dwcXpYT2pPWUZNMjJTOXVrN0g1Wmd1NS90R294eFl1bzVuQy9xCkJ6YlJBb0dBVHUwc2Rnb2dxL2RHdDFIVjNmZnhyVGhVazBMdVZxM0x4Ni9mN0tXWGN4b21qUWlQc0RZUTVkU3IKb21Kb2wvVUxWUnRQbGJtcmZKN1VpcHBlNXh1eUFNQUVocldIeG15SlR2TWZQSks3UGs4eHgrcUlSUGd0YklobwpWcjN5ZFoyYnRka2h4OXo5dGY4WXJ5RkxqNnNIWTY4V1Z2T3hETkcwYTZPYUhWWFJJMUE9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  name: cm-adapter-serving-certs
  namespace: custom-metrics

创建k8s-prometheus-adapter configmap,可以使用这个方式

kubectl create  secret generic  cm-adapter-serving-certs --from-file=serving.crt=apiserver.crt --from-file=serving.key=apiserver.key -n custom-metrics

也可以使用这种方式

kubectl create -f cm-adapter-serving-certs.yaml

创建custom-metrics namespace

kubectl create namespace custom-metrics

部署

修改镜像custom-metrics-apiserver-deployment.yaml的镜像

cd /opt/ssl/custom-metrics/k8s-prometheus-adapter/deploy/manifests

修改
    custom-metrics-apiserver-deployment.yaml

   image: directxman12/k8s-prometheus-adapter-amd64
    to 
   image: harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1

   - --prometheus-url=http://prometheus.prom.svc:9090/
   to  
   - --prometheus-url=http://prometheus-monitor.kube-system:9090/

  - --config=/default-config.yaml 
  to 
  - --config=/etc/adapter/config.yaml
添加这个
- --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

修改之后

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: custom-metrics-apiserver
  name: custom-metrics-apiserver
  namespace: custom-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-metrics-apiserver
  template:
    metadata:
      labels:
        app: custom-metrics-apiserver
      name: custom-metrics-apiserver
    spec:
      serviceAccountName: custom-metrics-apiserver
      containers:
      - name: custom-metrics-apiserver
        image: directxman12/k8s-prometheus-adapter-amd64
        args:
        - /adapter
        - --secure-port=6443
        - --tls-cert-file=/var/run/serving-cert/serving.crt
        - --tls-private-key-file=/var/run/serving-cert/serving.key
        - --logtostderr=true
        - --prometheus-url=http://prometheus-monitor.kube-system:9090/
        - --metrics-relist-interval=1m
        - --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        - --v=10
        - --config=/etc/adapter/config.yaml
        ports:
        - containerPort: 6443
        volumeMounts:
        - mountPath: /var/run/serving-cert
          name: volume-serving-cert
          readOnly: true
        - mountPath: /etc/adapter/
          name: config
          readOnly: true
        - mountPath: /tmp
          name: tmp-vol
      volumes:
      - name: volume-serving-cert
        secret:
          secretName: cm-adapter-serving-certs
      - name: config
        configMap:
          name: adapter-config
      - name: tmp-vol
        emptyDir: {}

添加kube-apiserver的启动参数

         --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \
         --requestheader-allowed-names=aggregator \
         --requestheader-extra-headers-prefix=X-Remote-Extra- \
         --requestheader-group-headers=X-Remote-Group \
         --requestheader-username-headers=X-Remote-User \
         --proxy-client-cert-file=/etc/kubernetes/ssl/aggregator-proxy.pem \
         --proxy-client-key-file=/etc/kubernetes/ssl/aggregator-proxy-key.pem \
         --enable-aggregator-routing=true \

添加kube-controller-manager的启动参数

 "--horizontal-pod-autoscaler-use-rest-clients",
 "--horizontal-pod-autoscaler-sync-period=60s",

cd /opt/ssl/custom-metrics/k8s-prometheus-adapter/deploy/manifests
kubectl create -f .

查看部署结果

[root@master-47-35 custom-metrics]# kubectl get all -n custom-metrics 
NAME                                           READY     STATUS    RESTARTS   AGE
pod/custom-metrics-apiserver-5d8dd96b5-8s9rq   1/1       Running   0          16h

NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/custom-metrics-apiserver   ClusterIP   10.254.40.72   <none>        443/TCP   16h

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/custom-metrics-apiserver   1         1         1            1           16h

NAME                                                 DESIRED   CURRENT   READY     AGE
replicaset.apps/custom-metrics-apiserver-5d8dd96b5   1         1         1         16h

查看api-versions

[root@master-47-35 k8s-prometheus-adapter]# kubectl api-versions 
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom.metrics.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

确认这个custom.metrics.k8s.io/v1beta1 api存在

root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_writes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_writes_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/start_time_seconds","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_cpu_shares","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_reservation_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_swap","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_schedstat_run","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/network_tcp_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_io_current","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_cpu_quota","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_writes_merged","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_failures","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_user","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_cfs_throttled_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_schedstat_run_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_load_average_10s","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_cpu_period","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_read","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_schedstat_run_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_rss","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_cache","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads_merged","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_sector_reads","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_swap_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_schedstat_run","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/last_seen","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_max_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_working_set_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_schedstat_runqueue","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_system","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_cfs_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_sector_writes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/network_udp_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_cfs_throttled","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_inodes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_io_time","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_write","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/memory_failcnt","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/tasks_state","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_schedstat_runqueue","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_io_time_weighted","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_inodes_free","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]}]}

接下来就是验证了

获取CPU cpu_usage

[root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/cpu_usage"
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/cpu_usage"},"items":[{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-qg2gj","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-t4xf2","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-w26gb","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"}]}

获取内存 memory_usage_bytes
[root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/memory_usage_bytes"
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/memory_usage_bytes"},"items":[{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-qg2gj","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"3547136"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-t4xf2","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"4435968"},{"describedObject":{"kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-w26gb","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"6172672"}]}

查看v1beta1.custom.metrics.k8s.io是否存在

[root@master-47-35 ~]# kubectl get apiservice v1beta1.custom.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: 2018-09-12T08:57:08Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-09-12T10:34:21Z
  finalizers:
  - foregroundDeletion
  name: v1beta1.custom.metrics.k8s.io
  resourceVersion: "3189364"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
  uid: d470a606-b669-11e8-aec0-5254e98192ae
spec:
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2018-09-12T10:14:40Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

基于内存

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2018-09-13T02:23:13Z","reason":"SucceededGetScale","message":"the
      HPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2018-09-13T02:24:13Z","reason":"ScalingDisabled","message":"scaling
      is disabled since the replica count of the target is zero"},{"type":"ScalingLimited","status":"True","lastTransitionTime":"2018-09-13T02:03:10Z","reason":"TooManyReplicas","message":"the
      desired replica count is more than the maximum replica count"}]'
    autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Pods","pods":{"metricName":"memory_usage_bytes","targetAverageValue":"1M"}}]'
  creationTimestamp: 2018-09-13T02:01:10Z
  name: example2
  namespace: demo
  resourceVersion: "3299536"
  selfLink: /apis/autoscaling/v1/namespaces/demo/horizontalpodautoscalers/example2
  uid: e2be2af9-b6f8-11e8-bf72-5254e98192ae
spec:
  maxReplicas: 2
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: example2
status:
  currentReplicas: 0
  desiredReplicas: 0
  lastScaleTime: 2018-09-13T02:17:13Z

基于cpu cpu_usage的使用

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2018-09-13T02:32:34Z","reason":"ReadyForNewScale","message":"the
      last scale time was sufficiently old as to warrant a new scale"},{"type":"ScalingActive","status":"True","lastTransitionTime":"2018-09-13T02:23:33Z","reason":"ValidMetricFound","message":"the
      HPA was able to successfully calculate a replica count from pods metric cpu_usage"},{"type":"ScalingLimited","status":"True","lastTransitionTime":"2018-09-13T02:23:33Z","reason":"TooManyReplicas","message":"the
      desired replica count is more than the maximum replica count"}]'
    autoscaling.alpha.kubernetes.io/current-metrics: '[{"type":"Pods","pods":{"metricName":"cpu_usage","currentAverageValue":"322m"}}]'
    autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Pods","pods":{"metricName":"cpu_usage","targetAverageValue":"30m"}}]'
  creationTimestamp: 2018-09-13T02:21:33Z
  name: example
  namespace: demo
  resourceVersion: "3301437"
  selfLink: /apis/autoscaling/v1/namespaces/demo/horizontalpodautoscalers/example
  uid: bbc859b6-b6fb-11e8-bf72-5254e98192ae
spec:
  maxReplicas: 4
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: example
status:
  currentReplicas: 4
  desiredReplicas: 4
  lastScaleTime: 2018-09-13T02:26:34Z