K8S 资源指标监控-部署metrics-server

K8S资源指标获取工具：metrics-server
自定义指标的监控工具：prometheus,k8s-prometheus-adapter

prometheus：prometheus能够收集各种维度的资源指标，比如CPU利用率，网络连接的数量，网络报文的收发速率，包括进程的新建及回收速率等等，能够监控许许多多的指标，而这些指标K8S早期是不支持的，所以需要把prometheus能采集到的各种指标整合进k8s里，能让K8S根据这些指标来判断是否需要根据这些指标来进行pod的伸缩。

prometheus既作为监控系统来使用，也作为某些特殊的资源指标的提供者来使用。但是这些指标不是标准的K8S内建指标，称之为自定义指标，但是prometheus要想将监控采集到的数据作为指标来展示，则需要一个插件，这个插件叫k8s-prometheus-adapter，这些指标判断pod是否需要伸缩的基本标准，例如根据cpu的利用率、内存使用量去进行伸缩。

随着prometheus和k8s-prometheus-adapter的引入，新一代的k8s架构也就形成了。

K8S新一代架构

核心指标流水线：由kubelet、metrics-server以及由API server提供的api组成；CPU累积使用率、内存的实时使用率、pod的资源占用率及容器的磁盘占用率；

监控流水线：用于从系统收集各种指标数据并提供给终端用户、存储系统以及HPA，包含核心指标以及其他许多非核心指标。非核心指标本身不能被K8S所解析。所以需要k8s-prometheus-adapter将prometheus采集到的数据转化为k8s能理解的格式，为k8s所使用。

核心指标监控

之前使用的是heapster，但是1.12后就废弃了，之后使用的替代者是metrics-server；metrics-server是由用户开发的一个api server，用于服务资源指标，而不是服务pod，deploy的。metrics-server本身不是k8s的组成部分，是托管运行在k8s上的一个pod，那么如果想要用户在k8s上无缝的使用metrics-server提供的api服务，因此在新一代的架构中需要这样去组合它们。如图，使用一个聚合器去聚合k8s的api server与metrics-server，然后由群组/apis/metrics.k8s.io/v1beta1来获取。

之后如果用户还有其他的api server都可以整合进aggregator，由aggregator来提供服务，如图。

查看k8s默认的api-version，可以看到是没有metrics.k8s.io这个组的

当你部署好metrics-server后再查看api-versions就可以看到metrics.k8s.io这个组了。

部署metrics-server
进到kubernetes项目下的cluster下的addons，找到对应的项目下载下来

[root@master bcia]# mkdir metrics-server -p 
[root@master bcia]# cd metrics-server/
[root@master metrics-server]# for file in auth-delegator.yaml auth-reader.yaml metrics-apiservice.yaml metrics-server-deployment.yaml metrics-server-service.yaml resource-reader.yaml ; do wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/metrics-server/$file;done    
//一次性下载所有文件
--2019-11-02 10:18:10--  https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/metrics-server/auth-delegator.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.228.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.228.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 398 [text/plain]
Saving to: ‘auth-delegator.yaml’

100%[==========================================================================>] 398         --.-K/s   in 0s
...省略...

[root@master metrics-server]# ls
auth-delegator.yaml  metrics-apiservice.yaml         metrics-server-service.yaml
auth-reader.yaml     metrics-server-deployment.yaml  resource-reader.yaml
[root@master metrics-server]# kubectl apply -f .          //一次性运行所有文件
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
configmap/metrics-server-config created
deployment.apps/metrics-server-v0.3.6 created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

运行后发现报错，一次性删除所有，修改几处地方，如图
1、metrics-server-deployment.yaml
metrics-server的command中加上 - --kubelet-insecure-tls 表示不验证客户端的证书，注释掉端口10255，注释后会使用10250，通过https通信

addon-resizer的command中写上具体的cpu、memory、extra-memory的值，注释掉minClusterSize={{ metrics_server_min_cluster_size }}


2、resource-reader.yaml
加上nodes/stats，如图

修改后的metrics-server-deployment.yaml和resource-reader.yaml文件内容放在了本文的最后。

测试是否可使用

//查看pods是否正常运行
[root@master metrics-server]# kubectl get pods -n kube-system
NAME                                     READY   STATUS    RESTARTS   AGE
coredns-8686dcc4fd-bzgss                 1/1     Running   0          9d
coredns-8686dcc4fd-xgd49                 1/1     Running   0          9d
etcd-master                              1/1     Running   0          9d
kube-apiserver-master                    1/1     Running   0          9d
kube-controller-manager-master           1/1     Running   0          9d
kube-flannel-ds-amd64-52d6n              1/1     Running   0          9d
kube-flannel-ds-amd64-k8qxt              1/1     Running   0          8d
kube-flannel-ds-amd64-lnss4              1/1     Running   0          9d
kube-proxy-4s5mf                         1/1     Running   0          8d
kube-proxy-b6szk                         1/1     Running   0          9d
kube-proxy-wsnfz                         1/1     Running   0          9d
kube-scheduler-master                    1/1     Running   0          9d
kubernetes-dashboard-76f6bf8c57-rncvn    1/1     Running   0          8d
metrics-server-v0.3.6-677d79858c-75vk7   2/2     Running   0          18m
tiller-deploy-57c977bff7-tcnrf           1/1     Running   0          7d20h

查看api-versions，会看到多出了metrics.k8s.io/v1beta1

查看node及pod监控指标

//查看node及pod监控指标
[root@master metrics-server]# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master   145m         3%     1801Mi          11%
node2    697m         17%    12176Mi         77%
node3    838m         20%    12217Mi         77%
[root@master metrics-server]# kubectl top pods
NAME                                            CPU(cores)   MEMORY(bytes)
account-deploy-6d86f9df74-khv4v                 5m           444Mi
admin-deploy-55dcf4bc4d-srw8m                   2m           317Mi
backend-deploy-6f7bdd9bf4-w4sqc                 4m           497Mi
crm-deploy-7879694578-cngzp                     4m           421Mi
device-deploy-77768bf87c-ct5nc                  5m           434Mi
elassandra-0                                    168m         4879Mi
gateway-deploy-68c988676d-wnqsz                 4m           379Mi
jhipster-alerter-74fc8984c4-27bx8               1m           46Mi
jhipster-console-85556468d-kjfg6                3m           119Mi
jhipster-curator-67b58477b9-5f8br               1m           11Mi
jhipster-logstash-74878f8b49-mpn62              59m          860Mi
jhipster-zipkin-5b5ff7bdbc-bsxhk                1m           1571Mi
order-deploy-c4c846c54-2gxkp                    5m           440Mi
pos-registry-76bbd6c689-q5w2b                   442m         474Mi
recv-deploy-5dd686c947-v4qqh                    5m           424Mi
store-deploy-54c994c9b6-82b8z                   6m           493Mi
task-deploy-64c9984d88-fqxqq                    6m           461Mi
wiggly-cat-redis-ha-sentinel-655f7b5f9d-bbrz6   4m           4Mi
wiggly-cat-redis-ha-sentinel-655f7b5f9d-bj4bq   4m           5Mi
wiggly-cat-redis-ha-sentinel-655f7b5f9d-f9pdd   4m           5Mi
wiggly-cat-redis-ha-server-b58c8d788-6xlwk      3m           11Mi
wiggly-cat-redis-ha-server-b58c8d788-r949h      3m           8Mi
wiggly-cat-redis-ha-server-b58c8d788-w2gtb      3m           22Mi

至此，metrics-server部署结束。下一篇写Prometheus

apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metrics-server-config
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server-v0.3.3
  namespace: kube-system
  labels:
    k8s-app: metrics-server
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v0.3.3
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
      version: v0.3.3
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
        version: v0.3.3
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: gcr.azk8s.cn/google-containers/metrics-server-amd64:v0.3.3
        command:
        - /metrics-server
        - --metric-resolution=30s
      #  - --kubeconfig=/key/kubeconfig
        # These are needed for GKE, which doesn't support secure communication yet.
        # Remove these lines for non-GKE clusters, and when GKE supports token-based auth.
        #- --kubelet-port=10255
        # - --deprecated-kubelet-completely-insecure=true
        #- --source=kubernetes.summary_api:https://kubernetes.default.svc?kubeletHttps=true&kubeletPort=10250&useServiceAccount=true&insecure=true
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
        ports:
        - containerPort: 443
          name: https
          protocol: TCP
      - name: metrics-server-nanny
        image: gcr.azk8s.cn/google-containers/addon-resizer:1.8.4
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 5m
            memory: 50Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        volumeMounts:
        - name: metrics-server-config-volume
          mountPath: /etc/config
        command:
          - /pod_nanny
          - --config-dir=/etc/config
          - --cpu=80m
          - --extra-cpu=0.5m
          - --memory=80Mi
          - --extra-memory=8Mi
          - --threshold=5
          - --deployment=metrics-server-v0.3.3
          - --container=metrics-server
          - --poll-period=300000
          - --estimator=exponential
          # Specifies the smallest cluster (defined in number of nodes)
          # resources will be scaled to.
         # - --minClusterSize={{ metrics_server_min_cluster_size }}
      volumes:
        - name: metrics-server-config-volume
          configMap:
            name: metrics-server-config
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - get
  - list
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system