Prometheus（六）黑盒监控

黑盒监控（blackbox_exporter）
之前介绍的对exporter的使用可以称为“白盒监控”，既需要把对应的exporter程序安装到被监控的目标主机上，从而实现对主机资源及其状态的数据采集工作。
黑盒监控，blackbox_exporter无须安装在被监控的目标环境中，用户只需要将其安装在于promethenus和被监控目标互通的环境中，通过HTTP、HTTPS、DNS、TCP、ICMP等方式对网络进行探测监控，还可以探测SSL证书过期时间
blackbox_exporter下载地址：https://prometheus.io/download/

1）安装blackbox_exporter
#上传软件

[root@localhost opt] ll blackbox_exporter-0.16.0.linux-amd64.tar.gz 
-rw-r--r--. 1 root root 8314959 May 18 20:40 blackbox_exporter-0.16.0.linux-amd64.tar.gz
[root@localhost opt] tar -zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz 
[root@localhost opt] cp -r blackbox_exporter-0.16.0.linux-amd64 /usr/local/blackbox_exporter

2）添加blackbox_exporter为系统服务开机启动配置文件blackbox_exporter.service

[root@localhost ~] vi /usr/lib/systemd/system/blackbox_exporter.service

[Unit]
Description=blackbox_exporter
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/blackbox_exporter/blackbox_exporter \
    --config.file=/usr/local/blackbox_exporter/blackbox.yml \
    --web.listen-address=:9115
Restart=on-failure

[Install]
WantedBy=multi-user.target

[root@localhost ~] systemctl daemon-reload
[root@localhost ~] systemctl restart blackbox_exporter
[root@localhost ~] netstat -tlunp | grep blackbox_expo
tcp6       0      0 :::9115              :::*              LISTEN      11925/blackbox_expo

web浏览

icmp监控，监控主机存活状态
通过icmp 这个指标的采集，我们可以确认到对方的线路是否有问题。这个也是监控里面比较重要的一个环节。我们要了解全国各地到我们机房的线路有哪条有问题我们总结了两种方案：
全国各地各节点ping 和访问数据采集。这种类似听云运营商有提供这类服务，但是要花钱；
我现在用的方法就是：找各地测试ping 的节点，我们从机房主动ping 看是否到哪个线路有故障，下面我们开始。

prometheus 添加相关监控，Blackbox 使用默认配置启动即可

[root@localhost ~] vi /usr/local/prometheus/prometheus.yml

 - job_name: "icmp_ping"
    metrics_path: /probe
    params:
      module: [icmp]  # 使用icmp模块
    file_sd_configs:
    - refresh_interval: 10s
      files:
      - "ping/ping_status*.yml"  #具体的配置文件
    relabel_configs:
    - source_labels: [__address__]
      regex: (.*)(:80)?
      target_label: __param_target
      replacement: ${1}
    - source_labels: [__param_target]
      target_label: instance
    - source_labels: [__param_target]
      regex: (.*)
      target_label: ping
      replacement: ${1}
    - source_labels: []
      regex: .*
      target_label: __address__
      replacement: 127.0.0.1:9115

针对上面的配置文件（- “ping/ping_status*.yml” ）创建相关的过滤配置

[root@prometheus rj] cd /usr/local/prometheus/
[root@localhost prometheus] mkdir ping
[root@localhost prometheus] vi ping_status.yml 

- targets: ['220.181.38.150','14.215.177.39','180.101.49.12','14.215.177.39','180.101.49.11','14.215.177.38','14.215.177.38']
  labels:
    group: '一线城市-电信网络监控'
- targets: ['112.80.248.75','163.177.151.109','61.135.169.125','163.177.151.110','180.101.49.11','61.135.169.121','180.101.49.11']
  labels:
    group: '一线城市-联通网络监控'
- targets: ['183.232.231.172','36.152.44.95','182.61.200.6','36.152.44.96','220.181.38.149']
  labels:
    group: '一线城市-移动网络监控' 

[root@prometheus ping] systemctl restart prometheus

以下是其他的几种监控状态如下
监控主机端口存活状态

[root@localhost ~] vi /usr/local/prometheus/prometheus.yml

- job_name: 'prometheus_port_status'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets: ['172.19.155.133:8765']
        labels:
          instance: 'port_status'
          group: 'tcp'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.19.155.133:9115

监控网站状态
prometheus 添加相关监控，Blackbox 使用默认配置启动即可

[root@localhost ~]# cat /usr/local/prometheus/prometheus.yml

  - job_name: "blackbox"
    metrics_path: /probe
    params:
      module: [http_2xx]  #使用http模块
    file_sd_configs: 
    - refresh_interval: 1m
      files: 
      - "/qq/blackbox*.yml"
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9115
[root@localhost ~]# cat /qq/blackbox-dis.yml 
- targets:
  - https://www.zhibo8.cc
  - https://www.baidu.com