prometheus告警

最新推荐文章于 2024-03-20 18:55:53 发布

Jenvid.yang

最新推荐文章于 2024-03-20 18:55:53 发布

阅读量825

点赞数 27

文章标签： prometheus

本文链接：https://blog.csdn.net/a53715668/article/details/136331684

版权

alter 告警

下载解压安装

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar -zxvf alertmanager-0.26.0.linux-amd64.tar.gz -C /usr/local
mv alertmanager-0.26.0.linux-amd64 alertmanager

配置起启动服务

cat > /lib/systemd/system/alertmanager.service <<EOF
[Unit]
Description=mysqld_exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/exporter/mysqld_exporter --config.my-cnf=/usr/local/exporter/config.my-cnf --web.config.file=/usr/local/exporter/webconf.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动服务

systemctl daemon-reload
systemctl enable --now alertmanager
systemctl status alertmanager

prometheus配置

prometheus.yml（缺省配置打开即可）

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
        # 打开下面这个配置
           - alertmanager:9093
rule_files:
# 打开下面这个配置
    - "first_rules.yml"

将主机名增加到增加hosts文件，或者修改为对应的主机名或者IP地址

增加rules `./rules/alarm_rule.yml`

  - alert: 服务告警
    expr: up == 0
    for: 30s
    labels:
      severity: critical
    annotations:
      instance: "{{ $labels.instance }}"
      decriptions: "{{ $labels.job }} 服务已关闭 "

重载配置 curl -X POST http://localhost:9090/-/reload -u admin:123456

已经可以看到告警信息

报警状态State分3种：

inactive：没有异常。
pending：已触发阈值，但未满足告警持续时间（即 rule 中的 for 字段）。
firing：已触发阈值且满足条件并发送至 alertmanager。

配置邮箱通知

https://prometheus.io/docs/alerting/latest/configuration/

https://blog.51cto.com/u_14205795/4561320

https://blog.51cto.com/u_10874766/2523351

global:
  smtp_from: '790079600@qq.com'
  smtp_smarthost: 'smtp.qq.com:465'               
  smtp_auth_username: '790079600@qq.com'
  smtp_auth_password: 'xxx'
  smtp_require_tls: false
  smtp_hello: 'qq.com'
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 5m
  receiver: 'email'
receivers:
  - name: 'email'
    email_configs:
      - to: '790079600@qq.com'
        send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

配置第三方通知

webhook1

安装

wget https://github.com/Rainbowhhy/alertmanager-webhook/releases/download/v1.0/alertmanager-webhook-v1.0-linux-amd64.tar.gz
tar -zxvf alertmanager-webhook-v1.0-linux-amd64.tar.gz -C /usr/local/
mv alertmanager-webhook-v1.0-linux-amd64/ alertmanager-webhook

配置文件

cat alertmanager-webhook.yaml.orig
# 企业微信机器人key
# 使用企业微信时必须配置，不使用则留空
qywechatKey:

# 飞书机器人key
# 使用飞书时必须配置，不使用则留空
feishuKey:

# 钉钉机器人key
# 使用钉钉时必须配置，不使用则留
dingdingKey:

# Redis配置
redisServer: 127.0.0.1  # 必须配置
redisPort:  # 可选项，为空默认为6379
redisPassword: wecloudsre@2023 # redis未设置密码则留空，如果设置了密码登陆则必须配置

# 日志配置
logFileDir:   # 可选项，为空则为程序运行目录
logFilePath: alertmanager-webhook.log # 必须配置

# 服务监听配置
port: 9095 # 可选项，为空则默认为9095
host: 0.0.0.0 # 可选项，为空默认监听 127.0.0.1

启动

 ./alertmanager-webhook -c alertmanager-webhook.yaml

alertmanager 修改配置

global:
  smtp_from: '790079600@qq.com'
  smtp_smarthost: 'smtp.qq.com:465'               
  smtp_auth_username: '790079600@qq.com'
  smtp_auth_password: 'xxxxxx'
  smtp_require_tls: false
  smtp_hello: 'qq.com'
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 5m 
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:9095/qywechat'
        send_resolved: true
  - name: 'email'
    email_configs:
      - to: '790079600@qq.com'
        send_resolved: true
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

webhook2 - 强烈推荐，太好用了

https://github.com/feiyu563/PrometheusAlert?tab=readme-ov-file

安装

wget https://github.com/feiyu563/PrometheusAlert/releases/download/v4.9/linux.zip
unzip linux.zip
mv linux /usr/local/prometheusalert
cd /usr/local/prometheusalert
./PrometheusAlert

飞书和企微群告警

github上有详细的配置指导

修改app.conf配置

#是否开启微信告警通道,可同时开始多个通道0为关闭,1为开启
open-weixin=1
#默认企业微信机器人地址
wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=22b46592-2601-40c4-aa17-xxx
#是否开启飞书告警通道,可同时开始多个通道0为关闭,1为开启
open-feishu=1
#默认飞书机器人地址
fsurl=https://open.feishu.cn/open-apis/bot/hook/53e6cda3-b391-4f30-8d8f-xxx

修改配置alertmanager.yml

global:
  resolve_timeout: 5m
route:
  group_by: ['instance']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 5m 
  receiver: 'web.hook.prometheusalert'
receivers:
  - name: 'web.hook.prometheusalert'
    webhook_configs:
    - url: 'http://127.0.0.1:8080/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=【app.conf中的地址】'
      send_resolved: true
    - url: 'http://127.0.0.1:8080/prometheusalert?type=wx&tpl=prometheus-wx&wxurl=【app.conf中的地址】'
    - url: 'http://127.0.0.1:8080/prometheusalert?type=email&tpl=prometheus-email&email=【收件人的值】'

配置服务

cat > /lib/systemd/system/prometheusalert.service <<EOF
[Unit]
Description=prometheusalert

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheusalert/PrometheusAlert

[Install]
WantedBy=multi-user.target
EOF