Prometheus基于k8s的自动发现配置监控

k8s配置Prometheus监控时，可以通过servicemonitor的方式增加job，以此来增加监控项，但这种方式进行监控配置，只能手工一个一个的增加，如果k8s集群规模较大的情况下，这种方式会很麻烦。

一种方式是采用consul注册中心的方式进行自动发现。

另外一种方式是基于kubernetes_sd_configs的自动发现的方式配置增加监控项，本文主要讲解此种配置方式。

基于consul的自动发现

在安装consul后，可以通过指定consul读取特定配置文件的方式发现并加载监控项

 但这种方式和Prometheus基于文件的动态发现没有本质区别，甚至还增加了系统的复杂度，并不可取。

可以通过调用consul的API的方式，让程序自动向consul进行注册，在Prometheus中配置consul的相关项，让其自动增加监控Target

#Prometheus主配置文件增加如下内容
#以便可以从consul中自动获取监控信息
   - job_name: 'consul-prometheus'
     consul_sd_configs:
       - server: '10.0.12.8:8500'
         services: []

# 注册服务
curl -X PUT -d '{"id": "consul-redis","name": "redis","address": "10.0.12.8","port": 6379,"tags": ["service"],"checks": [{"http": "http://10.0.12.8:6379/","interval": "5s"}]}' http://10.0.12.8:8500/v1/agent/service/register
 
# 查询指定节点以及指定的服务信息
[root@iZ2zejaz33icbod2k4cvy6Z ~]# curl http://10.0.12.8:8500/v1/catalog/service/consul-redis
 
#删除指定服务 redis为要删除服务的id
curl -X PUT  http://10.0.12.8:8500/v1/agent/service/deregister/consul-redis

基于kubernetes_sd_configs的自动发现

准备Prometheus的自动发现的配置文件并加载

[root@VM-12-8-centos kube-prom]# cat prometheus-additional.yaml 
- job_name: 'blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - http://10.1.226.250:6000
      - http://10.1.38.97:3000/healthz/ready
      - http://10.1.116.84:5000
      - http://10.1.215.125:7000/healthz/ready
      - http://10.1.111.235:8000/healthz/ready
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:9115
- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: service
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod
    action: replace
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: pod

运行生成secret文件

[root@VM-12-8-centos kube-prom]# kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run -oyaml > additional-scrape-configs.yaml

应用，配置进入Prometheus中

[root@VM-12-8-centos kube-prom]# kubectl apply -f additional-scrape-configs.yaml -n monitoring 
secret/additional-scrape-configs configured

运行curl -XPOST http://10.0.12.8:30090/-/reload热加载一下，就可以在dashboard中看到增加的配置了

修改prometheus-k8s 的 ClusterRole权限 

 Prometheus 绑定了一个名为 prometheus-k8s 的 ServiceAccount 对象，而这个对象绑定的是一个名为 prometheus-k8s 的 ClusterRole，这个角色没有对 Service 或者 Pod 的 list 权限，所以需要进行修改

[root@VM-12-8-centos manifests]# kubectl edit clusterrole prometheus-k8s -n monitoring -o yaml  
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: "2022-11-13T14:21:08Z"
  name: prometheus-k8s
  resourceVersion: "16164985"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/prometheus-k8s
  uid: 7a404fac-9462-486a-a109-65a1ef98e423
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

 pod配置自动发现

pod要自动发现，必须在annotations:增加prometheus.io/scrape: "true"

新建一个pod

[root@VM-12-8-centos k8s]# cat PODforheadlesssvr.yml
apiVersion: v1
kind: Pod
metadata:
  name: ex-podforheadlesssvr
  annotations:
    prometheus.io/scrape: "true"
spec:
  containers:
  - name: testcontainer
    image: docker.io/appropriate/curl
    imagePullPolicy: IfNotPresent
    command: ['sh', '-c']
    args: ['echo "test pod for headless service";sleep 96000']
[root@VM-12-8-centos k8s]# kubectl apply -f ex6_1_4PODforheadlesssvr.yml
pod/ex-podforheadlesssvr created
[root@VM-12-8-centos k8s]# kubectl get po
NAME                                     READY   STATUS      RESTARTS   AGE
ex-podforheadlesssvr                     1/1     Running     0          3s

过一会检查dashboard，已经在界面上了

 状态为down，因为这个pod对应的镜像并没有相关的metrics接口，我们主要是用来进行自动发现测试的

在服务发现界面

在target labels部分

如上操作，就可以基于k8s自动发现 在Prometheus中增加监控项了