kubernetes-解决prometheus从错误的端口获取指标

版本：prometheus     2.2.1 

1. 增加kubernetes pod 自动发现配置：

    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

在需要监控的pod的deployment 中指定了prometheus.io/port ：

      annotations:
        prometheus.io/port: '8481'
        prometheus.io/scrape: 'true'

发布pod后，仍然看到Prometheus试图从其他端口获取指标：

上图看到从80端口查找指标,并不是我们指定的端口：8481，而是一个错误的endpoint:

endpoints:
http://172.30.59.17:80/metrics

参考job_name: kubernetes-service-endpoints的写法，更改检索端口的正则：

        regex: (.+):(?:\d+);(\d+)

改为：

        regex: ([^:]+)(?::\d+)?;(\d+)

kubernetes-pods完整配置如下：

    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

再次在prometheus的target中查看，已经正确了：

目前最新版的prometheus helm chart 已经增加上kubernetes-pods自动发现的配置了，详细请点击查看。