prometheus监控之黑盒(blackbox)监控

最新推荐文章于 2025-04-07 15:52:16 发布

运维开发小白丶

最新推荐文章于 2025-04-07 15:52:16 发布

阅读量1k

点赞数 6

分类专栏： prometheus监控 consul 文章标签： prometheus consul

本文链接：https://blog.csdn.net/liulunan_lln/article/details/144453682

版权

prometheus监控同时被 2 个专栏收录

19 篇文章

订阅专栏

consul

4 篇文章

订阅专栏

1.简单介绍

blackbox-exporter项目地址：https://github.com/prometheus/blackbox_exporter

blackbox-exporter是Prometheus官方提供的一个黑盒监控解决方案，blackbox-exporter无须安装在被监控的目标环境中，用户只需要将其安装在与Prometheus和被监控目标互通的环境中，通过HTTP、HTTPS(URL/API可用性检测)、DNS(域名解析)、TCP(端口存活检测)、ICMP(主机存活检测)等方式对网络进行探测监控，还可以探测SSL证书过期时间。

2.二进制安装

2.1 下载并解压

mkdir -p /root/exporter/blackbox_exporter && cd /root/exporter/blackbox_exporter 
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.23.0/blackbox_exporter-0.23.0.linux-amd64.tar.gz
tar zxvf blackbox_exporter-0.23.0.linux-amd64.tar.gz

2.2 创建systemd服务

执行vi /etc/systemd/system/blackbox_exporter.service并把下面内容复制到文件中

[Service]
ExecStart=/root/exporter/blackbox_exporter/blackbox_exporter --config.file=/root/exporter/blackbox_exporter/blackbox.yml --web.listen-address=:9115
Restart=on-failure

[Install]
WantedBy=multi-user.target

2.3 配置

# /root/exporter/blackbox_exporter/blackbox.yml
modules:
  http_2xx: # http检测模块,blockbox-exporter中所有的探针均是以module的信息进行配置
    prober: http
    http:
      preferred_ip_protocol: "ip4"
      tls_config:
        insecure_skip_verify: true
  http_post_2xx: # http post监测模块
    prober: http
    http:
      method: POST
  tcp_connect: # tcp检测模块
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

2.4 启动

systemctl daemon-reload 
systemctl start blackbox_exporter # 启动
systemctl status blackbox_exporter # 状态
systemctl enable blackbox_exporter # 开机自启动

2.5手动获取指标

执行curl [http://xx.xx.xx.xx:9115/probe?target=[target目标]&module=[模块名]&debug=true](http://10.17.12.10:9115/probe?target=https://monitor.asiainfo.com&module=http_2xx&debug=true)后效果如下：
请添加图片描述

3.监控

3.1HTTP监控

- job_name: "blackbox_http"
    metrics_path: /probe # 指定指标接口
    params: # 指定查询参数，在prometheus向target发送get请求获取指标数据时，会传递到url上
      module: [http_2xx]
    honor_labels: true
    consul_sd_configs:
      - server: 'xx.xx.xx.xx:8500' # 服务发现consul地址
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*blackbox-http.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      # 将标签__meta_consul_service_metadata_instance的值赋值给__param_target标签
      # 以__param开头的标签也会作为查询参数传递prometheus的get请求，作用和上面的params配置类似
      - source_labels: [__meta_consul_service_metadata_instance]
        target_label: __param_target
      # 将标签__param_target的值赋值给instance标签
      - source_labels: [__param_target]
        target_label: instance
      # 将标签__address__的值修改给balckbox-expoter的地址
      - target_label: __address__
        replacement: xx.xx.xx.xx:9115 # blackbox-exporter地址

3.2 TCP监控

- job_name: 'blackbox-tcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    honor_labels: true
    consul_sd_configs:
      - server: 'xx.xx.xx.xx:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*blackbox-tcp.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels: [__meta_consul_service_metadata_instance]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: xx.xx.xx.xx:9115

3.3 ICMP监控

- job_name: "blackbox_icmp"
    metrics_path: /probe
    params:
      module: [icmp]
    consul_sd_configs:
      - server: 'xx.xx.xx.xx:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*blackbox-icmp.*
        action: keep
      - regex: __meta_consul_service_metadata_(.+)
        action: labelmap
      - source_labels: [__meta_consul_service_metadata_instance]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: xx.xx.xx.xx:9115

3.4 python注册consul服务

注册服务模板请参考: https://blog.csdn.net/liulunan_lln/article/details/140875069?spm=1001.2014.3001.5502

# 模版请参考
def register_service_web_exporter_to_consul() -> None:
    print("register service exporter to consul.")
    host = ""
    exporter_id = f"web-exporter-{host}"
    params = consul_register_template.render(exporter_id=exporter_id,
                                             tags=['service', "web", 'exporter', 'blackbox-http'],
                                             exporter_address=host,
                                             exporter_port=80,
                                             labels={
                                                 "host": "xxx",
                                                 "port": 80,
                                                 "instance": "http|https://xx.xx.com",
                                             })
    print(f"Start register svc: {exporter_id}")
    resp = requests.put(f"https://xx.xx.com/v1/agent/service/register", # consul服务地址
                        json=json.loads(params),
                        verify=False)
    if not resp.ok:
        raise Exception(f"register svc {exporter_id} failed.")
    print(f"register svc {exporter_id} Success.")

4.指标说明

# DNS解析时间,单位 s
probe_dns_lookup_time_seconds 0.000199105
# 探测从开始到结束的时间,单位 s,请求这个页面响应时间
probe_duration_seconds 0.010889113
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HTTP 内容响应的长度
probe_http_content_length -1
# 按照阶段统计每阶段的时间
probe_http_duration_seconds{phase="connect"} 0.001083728    #连接时间
probe_http_duration_seconds{phase="processing"} 0.008365885 #处理请求的时间
probe_http_duration_seconds{phase="resolve"} 0.000199105    #响应时间
probe_http_duration_seconds{phase="tls"} 0                  #校验证书的时间
probe_http_duration_seconds{phase="transfer"} 0.000446424   #传输时间
# 重定向的次数
probe_http_redirects 0
# ssl 指示是否将 SSL 用于最终重定向
probe_http_ssl 0
# 返回的状态码
probe_http_status_code 200
# 未压缩的响应主体长度
probe_http_uncompressed_body_length 1766
# http协议的版本
probe_http_version 1.1
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
probe_ip_addr_hash 3.24030434e+09
# 使用的ip协议的版本号
probe_ip_protocol 4
probe_ssl_earliest_cert_expiry 1.749882884e+09
robe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
probe_ssl_last_chain_info{fingerprint_sha256="5ce3bbf06bd1608e04a64b1cd91e3fa69ed86cd9c55a1da52a8187140e0ece5b",issuer="CN=GlobalSign GCC R3 DV TLS CA 2020,O=GlobalSign nv-sa,C=BE",subject="CN=*.asiainfo.com",subjectalternative="*.asiainfo.com,asiainfo.com"} 1
# 是否探测成功
probe_success 1
# tls版本
probe_tls_version_info{version="TLS 1.2"} 1

5.Grafana模板

导入dashboard 
13659 HTTP状态监控
9965 SSL TCP HTTP综合监控图标
13230 SSL证书监控

6.prometheus告警规则

- alert: blackbox-default
    annotations:
      description: 域名证书7天后过期
      summary: 域名证书即将过期,VALUE = {{ $value }}
    expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7
    for: 30m
    labels:
      rule_type: blackbox
      severity: emergency