k8s集群安装:kube-state-metrics
本文前提已在集群外安装好了Prometheus,alertmanager,grafana。已搭建完毕K8S集群。
本文只写了kube-state-metrics 和Prometheus的内容。
下载地址:
https://github.com/kubernetes/kube-state-metrics
注意和集群的版本是否匹配
k8s集群1.23用的是v2.4.2
#git clone https://github.com/kubernetes/kube-state-metrics
#cd ./kube-state-metrics/examples/standard
修改service.yaml,我svc这里使用了LoadBalance,主要是测试外网。你可以使用ClusterIP
按顺序apply这里的所有的yaml文件
#kubectl apply -f .
如果deployment无法下载镜像的话,image修改为:bitnami/kube-state-metrics
测试访问
#curl http://svc:8080/healthz
返回OK说明正常
kube-state-metrics主要是采集数据用的
配置Prometheus获取数据
token获取方式:
创建RBAC,请参考kuberneters其他资料,流程为创建role,rolebinding,sa,不再赘述
获取token的命令为 describe
#kubectl describe secret XXXX-token
复制到 /usr/local/prometheus/k8s-token.conf
prometheus.yaml添加如下内容:
主要是role: endpoints这一个部分的内容
- job_name: kubernetes-nodes-cadvisor
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:
- role: node
api_server: https://172.21.114.169:6443
bearer_token_file: /usr/local/prometheus/k8s-token.conf
tls_config:
insecure_skip_verify: true
bearer_token_file: /usr/local/prometheus/k8s-token.conf
tls_config:
insecure_skip_verify: true
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.*)
- action: replace
regex: (.*)
source_labels: ["__address__"]
target_label: __address__
replacement: 172.21.114.169:6443
- action: replace
source_labels: [__meta_kubernetes_node_name]
target_label: __metrics_path__
regex: (.*)
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
#可以不加这个,只是为了测试
- job_name: kubernetes-service-endpoints
kubernetes_sd_configs:
- role: endpoints
api_server: https://172.21.114.169:6443
bearer_token_file: /usr/local/prometheus/k8s-token.conf
tls_config:
insecure_skip_verify: true
bearer_token_file: /usr/local/prometheus/k8s-token.conf
tls_config:
insecure_skip_verify: true
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_service_name
target_label: kubernetes_service_name
访问Prometheus就可以查看数据了:
kube_pod_container_info