Alertmanager机制

除了基本的告警通知能力外，Alertmanager还支持对告警进行去重分组抑制静默和路由等功能

分组机制
将相似告警合并为单个告警通知的机制，在系统因大面积故障而触发告警是，分组机制能避免用户被大量的告警噪声淹没，进而导致关键信息的隐没
抑制机制
系统中某个组件或服务故障而触发告警通知后，那些依赖于该组件或服务的其他组件或服务也会因此而触发告警，抑制是避免类似的级联告警的一种特性，从而让用户的经历集中于真正的故障所在
静默机制
在一个特定的时间窗口内，便接收到告警通知，Alertmanager也不会真正向用户发送告警行为；通常，在系统例行维护期间，需要激活告警系统的静默特性

部署Alertmanger

• 下载二进制包

[root@server1 ~]# wget
https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertman
ager-0.21.0.linux-amd64.tar.gz
[root@server1 ~]# sha256sum alertmanager-0.21.0.linux-amd64.tar.gz
9ccd863937436fd6bfe650e22521a7f2e6a727540988eef515dde208f9aef232
alertmanager-0.21.0.linux-amd64.tar.gz

• 解压缩

[root@server1 ~]# tar -zxvf alertmanager-0.21.0.linux-amd64.tar.gz -C /data/
[root@server1 ~]# cd /data/
[root@server1 data]# ln -sv alertmanager-0.21.0.linux-amd64 alertmanager
[root@server1 data]# chown -R root:root alertmanager-0.21.0.linux-amd64

• 查看配置文件
  参考文档：https://blog.csdn.net/aixiaoyang168/article/details/98474494
• 配置默认文件

[root@server1 alertmanager]# cat alertmanager.yml
global: # 全局配置模块
resolve_timeout: 5m # 用于设置处理超时时间，默认是5分钟
route: # 路由配置模块
group_by: ['alertname'] # 告警分组
group_wait: 10s # 10s内收到的同组告警在同一条告警通知中发送出去
group_interval: 10s # 同组之间发送告警通知的时间间隔
repeat_interval: 1h # 相同告警信息发送重复告警的周期
receiver: 'web.hook' # 使用的接收器名称
receivers: # 接收器
- name: 'web.hook' # 接收器名称
webhook_configs: # 设置webhook地址
- url: 'http://127.0.0.1:5001/'
inhibit_rules: # 告警抑制功能模块
- source_match:
severity: 'critical' # 当存在源标签告警触发时抑制含有目标标签的告警
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance'] # 保证该配置下标签内容相同才会被
抑制

• 自定义alertmanager使用qq邮箱报警的配置文件

[root@server1 alertmanager]# vim alertmanager.yml
global:
resolve_timeout: 5m
smtp_from: 'smqy0608@qq.com'
smtp_smarthost: 'smtp.qq.com:465'
smtp_auth_username: 'smqy0608@qq.com'
smtp_auth_password: '****************' #qq邮箱授权码
smtp_require_tls: false
smtp_hello: 'qq.com'
route:
group_by: ['alertname']
group_wait: 5s
group_interval: 5s
repeat_interval: 5m
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: '2404053206@qq.com'
send_resolved: true
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']

• 启动Alertmanager

[root@server1 alertmanager]# ./alertmanager

• 组合Prometheus与alertmanager

[root@server1 prometheus]# cat prometheus.yml
alerting:
alertmanagers:
- static_configs:
- targets:
- 192.168.175.30:9093
.......
scrape_configs:
- job_name: 'Alertmanager'
static_configs:
- targets: ['192.168.175.30:9093']

• 告警规则

[root@server1 prometheus]# mkdir -p /data/prometheus/rules/
[root@server1 prometheus]# vim prometheus.yml
rule_files:
- "/data/prometheus/rules/*.yml"
[root@server1 rules]# cd /data/prometheus/rules
[root@server1 rules]# vim rules.yml
groups:
- name: up
rules:
- alert: node
expr: up{job="node_service_discovery"} == 0
for: 1m
labels:
severity: critical
annotations:
description: "Node has been dwon for more than 1 minutes"
summary: "Node down"
[root@server1 prometheus]# ./promtool check rules rules/rules.yml
Checking rules/rules.yml
SUCCESS: 1 rules found
#重启promtetheus

vbfcljypjyzejdch