Prometheus集群外监控k8s

k8s集群安装：kube-state-metrics

本文前提已在集群外安装好了Prometheus，alertmanager，grafana。已搭建完毕K8S集群。

本文只写了kube-state-metrics 和Prometheus的内容。

下载地址：

https://github.com/kubernetes/kube-state-metrics

注意和集群的版本是否匹配

k8s集群1.23用的是v2.4.2

#git clone https://github.com/kubernetes/kube-state-metrics

#cd ./kube-state-metrics/examples/standard

修改service.yaml，我svc这里使用了LoadBalance，主要是测试外网。你可以使用ClusterIP

按顺序apply这里的所有的yaml文件

#kubectl apply -f .

如果deployment无法下载镜像的话，image修改为：bitnami/kube-state-metrics

测试访问

#curl http://svc:8080/healthz

返回OK说明正常

kube-state-metrics主要是采集数据用的

配置Prometheus获取数据

参考地址：https://github.com/prometheus/prometheus/blob/release-2.34/documentation/examples/prometheus-kubernetes.yml

token获取方式：

创建RBAC，请参考kuberneters其他资料，流程为创建role,rolebinding,sa,不再赘述

获取token的命令为 describe

#kubectl describe secret XXXX-token

复制到 /usr/local/prometheus/k8s-token.conf

prometheus.yaml添加如下内容：

主要是role: endpoints这一个部分的内容

- job_name: kubernetes-nodes-cadvisor
    metrics_path: /metrics
    scheme: https
    kubernetes_sd_configs:
    - role: node
      api_server: https://172.21.114.169:6443
      bearer_token_file: /usr/local/prometheus/k8s-token.conf
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /usr/local/prometheus/k8s-token.conf
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.*)
    - action: replace
      regex: (.*)
      source_labels: ["__address__"]
      target_label: __address__
      replacement: 172.21.114.169:6443
    - action: replace
      source_labels: [__meta_kubernetes_node_name]
      target_label: __metrics_path__
      regex: (.*)
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
#可以不加这个，只是为了测试

- job_name: kubernetes-service-endpoints
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://172.21.114.169:6443
      bearer_token_file: /usr/local/prometheus/k8s-token.conf
      tls_config:
        insecure_skip_verify: true
    bearer_token_file: /usr/local/prometheus/k8s-token.conf
    tls_config:
      insecure_skip_verify: true
    relabel_configs:
    - action: keep
      regex: true
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scrape
    - action: replace
      regex: (https?)
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_scheme
      target_label: __scheme__
    - action: replace
      regex: (.+)
      source_labels:
      - __meta_kubernetes_service_annotation_prometheus_io_path
      target_label: __metrics_path__
    - action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      source_labels:
      - __address__
      - __meta_kubernetes_service_annotation_prometheus_io_port
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - action: replace
      source_labels:
      - __meta_kubernetes_namespace
      target_label: kubernetes_namespace
    - action: replace
      source_labels:
      - __meta_kubernetes_service_name
      target_label: kubernetes_service_name

访问Prometheus就可以查看数据了：

kube_pod_container_info