Prometheus + Grafana 实现钉钉告警

Prometheus + Grafana 实现钉钉报警

资源列表

操作系统	配置	主机名	IP
CentOS 7.9	2C2G	prometheus	192.168.10.192
CentOS 7.9	2C2G	grafana	192.168.10.193
CentOS 7.9	2C2G	node	192.168.10.194
CentOS 7.9	2C2G	alert	192.168.10.195

• 所需安装包

Prometheus相关安装包都在github进行托管，而访问github时好时坏，
有条件的或使用GitHub镜像网站可以自行下载；没条件的可以通过下面链接进行下载:
Prometheus+Grafana.zip
https://url83.ctfile.com/f/33603683-606759914-012c3c?p=1942 (访问密码: 1942)

物料包

1 基础环境

• 关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

• 关闭selinux

sed -i "s/^SELINUX=*/SELINUX=disabled/g" /etc/selinux/config
setenforce 0

• 修改主机名，添加hosts解析

cat << EOF >> /etc/hosts
192.168.10.192 prometheus
192.168.10.193 grafana
192.168.10.194 node
192.168.10.195 alert
EOF

# 修改主机名
hostnamectl set-hostname grafana && su
hostnamectl set-hostname node && su
hostnamectl set-hostname alert && su
hostnamectl set-hostname prometheus && su

• 时间同步
  这里很重要

systemctl restart chronyd
chronyc sources
# 检查时间
date

2 Prometheus节点部署prometheus

• 上传prometheus源码包，并解压

tar zxf prometheus-2.26.0.linux-amd64.tar.gz 
mv prometheus-2.26.0.linux-amd64 /usr/local/prometheus

• 创建用于运行Prometheus的组和用户

groupadd prometheus
useradd -g prometheus -s /sbin/nologin prometheus

• 创建Prometheus数据存储目录

mkdir -p /var/lib/prometheus
chown -R prometheus /var/lib/prometheus/

• 备份并修改配置文件

cp /usr/local/prometheus/prometheus.yml{,.bak}
# 修改配置文件
cat << EOF >> /usr/local/prometheus/prometheus.yml
  - job_name: 'node'
    scrape_interval: 10s
    static_configs:
    # 这里是node节点的IP
    - targets: ['192.168.10.194:9100']
      labels:
        instance: node
EOF

• 启动prometheus

/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml

3 grafana节点部署grafana

• 上传并安装官网下载的grafana的rpm安装包

wget https://dl.grafana.com/oss/release/grafana-8.5.2.x86_64.rpm
rpm -ivh grafana-8.5.2-1.x86_64.rpm 

• 根据回显提示，重载系统配置，启动grafana

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server.service
systemctl start grafana-server

• web登录配置模板

8919
9276

4 node节点部署node_exporter

• 上传并解压node_exporter安装包

tar zxf node_exporter-1.1.2.linux-amd64.tar.gz 
mv node_exporter-1.1.2.linux-amd64 node_exporter

• 创建存储exporter的目录

mkdir -p /usr/local/prometheus_exporter
mv node_exporter /usr/local/prometheus_exporter/

• 启动node_exporter

/usr/local/prometheus_exporter/node_exporter/node_exporter

5 alert节点部署Alertmanager

• 上传并解压安装包

tar zxvf alertmanager-0.21.0.linux-amd64.tar.gz
mv alertmanager-0.21.0.linux-amd64 /usr/local/alertmanager

• 备份并修改配置文件

cp /usr/local/alertmanager/alertmanager.yml{,.bak}
# 修改配置文件
cat << EOF > /usr/local/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 5s
  group_interval: 5s
  repeat_interval: 5m
  receiver: webhook
receivers:
- name: webhook
  webhook_configs:
  - url: http://192.168.10.195:8060/dingtalk/webhook1/send
    send_resolved: true
EOF

• 启动Alertmanager

/usr/local/alertmanager/alertmanager --config.file /usr/local/alertmanager/alertmanager.yml

6 prometheus节点对接Alertmanager，并配置报警规则

• 修改配置文件

[root@prometheus ~]# cat /usr/local/prometheus/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
# 取消alert节点的主机端口，注意yml文件的格式，主机名要对应，如果害怕主机名错误，可以直接写IP地址
       - alert:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# 添加16行内容，加载告警规则目录内，告警规则文件
  - "/data/prometheus/rules/*_rules.yml"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'node'
    scrape_interval: 10s
    static_configs:
    - targets: ['192.168.10.194:9100']
      labels:
        instance: node

• 创建告警规则存储目录

mkdir -p /data/prometheus/rules

• 创建磁盘分区告警规则

[root@prometheus ~]# cat /data/prometheus/rules/disk_rules.yml 
groups:
- name: general.rules
  rules:
  - alert: NodeFilesystemUsage
    expr: 100 - (node_filesystem_free_bytes{mountpoint="/",fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"} * 100) > 80
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "Instance {{ $labels.instance }} ：{{ $labels.mountpoint }} 分区使
用率过高" # 自定义摘要
      description: "{{ $labels.instance }} ： {{ $labels.job }} ：{{ $labels.mountpoint }} 这个分区使用大于百分之80% (当前值：{{ $value }})" # 自定义具体描述

• 创建节点宕机告警规则

[root@prometheus ~]# cat /data/prometheus/rules/up_rules.yml 
groups:
- name: general.rules
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      severity: error
    annotations:
      summary: "Instance {{ $labels.instance }} 停止工作" # 自定义摘要
      description: "{{ $labels.instance }} ： {{ $labels.job }} 已经停止五分钟以上了" # 自定义具体描述

• 重启prometheus

/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml

7 在钉钉内添加自定义机器人

• 自定义关键字

Firing

• 上传并解压Prometheus的钉钉告警插件

tar zxf prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz 

• 启动钉钉告警插件

cd prometheus-webhook-dingtalk-1.4.0.linux-amd64
# 这里的webhook1为钉钉机器人的webhook地址
./prometheus-webhook-dingtalk --ding.profile="webhook1=https://oapi.dingtalk.com/robot/send?access_token=***************************************"

• 等待钉钉报警

8 遇到的问题

• prometheus告警

level=error ts=2022-07-01T09:33:15.217Z caller=notifier.go:527 component=notifier alertmanager=http://alertmanager:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://alertmanager:9093/api/v2/alerts\": dial tcp: lookup alertmanager on 192.168.10.2:53: no such host"
level=error ts=2022-07-01T09:34:29.995Z caller=notifier.go:527 component=notifier alertmanager=http://alertmanager:9093/api/v2/alerts count=3 msg="Error sending alert" err="Post \"http://alertmanager:9093/api/v2/alerts\": context deadline exceeded"

• 解决办法

# 注意报警信息，一条是上下文有问题，一条是没有主机alertmanager:9093
检查配置文件，注意到
       - alertmanager:9093
而配置的主机名为alert，将主机名修改为alert节点或者直接写入alert节点的IP地址

• grafana没有数据Nodata

解决办法：一些人在基础环境部分，往往会忘记进行时间同步，或者时间同步不彻底，没有仔细检查时间同步结果，而导致的结果，再次进行时间同步，并检查时间同步结果