Alertmanager配置钉钉+邮件同时告警

1、配置钉钉告警及模板

1. 创建钉钉机器人获取webhook和加签密钥


2. 安装Webhook-dingtalk插件
wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v2.1.0/prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz

tar zxf prometheus-webhook-dingtalk-2.1.0.linux-amd64.tar.gz 
mv prometheus-webhook-dingtalk-2.1.0.linux-amd64 /usr/local/webhook-dingtalk


3. 配置webhook-dingtalk使用systemd管理
cp /usr/local/webhook-dingtalk/config.example.yml /usr/local/webhook-dingtalk/config.yml
vim /usr/lib/systemd/system/webhook.service

[Unit]
Description=Prometheus-Server
After=network.target

[Service]
ExecStart=/usr/local/webhook-dingtalk/prometheus-webhook-dingtalk --config.file=/usr/local/webhook-dingtalk/config.yml
User=root

[Install]
WantedBy=multi-user.target


4. 启动 && 开机自启
systemctl enable webhook.service --now
systemctl status webhook.service 
#验证端口
netstat -anput |grep 8060


5. 配置Webhook-dingtalk插件对接钉钉群
vim /usr/local/webhook-dingtalk/config.yml

templates:
  - /usr/local/webhook-dingtalk/template.tmpl   # 告警模板路径

targets:
  webhook1:
    url: https://oapi.dingtalk.com/robot/send?access_token=xxxxxxxxxxx   # webhook地址
    secret: xxxxxxxxxx # 加签秘钥


6. 添加钉钉报警模板
自定义告警模板配置目录
/usr/local/webhook-dingtalk/template.tmpl

7. 重启
systemctl restart webhook
systemctl status webhook

 2、修改alertmanager配置文件

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: alertmanager
  namespace: prometheus
data:
  alertmanager.yml: |-
    global:  
      resolve_timeout: 1m
      smtp_smarthost: 'smtp.qq.com:25'
      smtp_from: 'xxxxxx@qq.com'
      smtp_auth_username: 'xxxxxx@qq.com'
      smtp_auth_password: 'xxxxxxxxx'
      smtp_require_tls: false
    templates:
    - '/etc/alertmanager/*.tmpl'
    route:
      group_by: [alertname]
      group_wait: 10s
      group_interval: 15s
      repeat_interval: 1h
      receiver: default-receiver   # 告警发送到default-receiver接受者
      routes:
      - receiver: 'dingding.webhook1'
        continue: true
      - receiver: 'default-receiver'
    receivers:
    - name: 'default-receiver'     # 定义default-receiver接受者
      email_configs:
      - to: 'xxxxxx@qq.com'   # 告警发送邮箱地址
        send_resolved: true
        html: '{{ template "email.to.html" . }}'
    - name: 'dingding.webhook1'
      webhook_configs:
      - url: 'http://10.0.0.10:8060/dingtalk/webhook1/send'
        send_resolved: true

  email.tmpl: |-
    {{ define "email.from" }}xxx.com{{ end }}
    {{ define "email.to" }}xxx.com{{ end }}
    {{ define "email.to.html" }}
    {{- if gt (len .Alerts.Firing) 0 -}}
    {{ range .Alerts }}
    ========= 监控报警 =========<br>
    告警级别: {{ .Labels.severity }} <br>
    告警类型: {{ .Labels.alertname }} <br>
    告警主机: {{ .Labels.instance }} <br>
    告警主题: {{ .Annotations.summary }}  <br>
    告警详情: {{ .Annotations.description }} <br>
    触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} <br>
    ========= = end =  =========<br>
    {{ end }}{{ end -}}

    {{- if gt (len .Alerts.Resolved) 0 -}}
    {{ range .Alerts }}
    ========= 告警恢复 =========<br>
    告警级别: {{ .Labels.severity }} <br>
    告警类型: {{ .Labels.alertname }} <br>
    告警主机: {{ .Labels.instance }} <br>
    告警主题: {{ .Annotations.summary }} <br>
    告警详情: {{ .Annotations.description }} <br>
    触发时间: {{ .StartsAt.Format "2006-01-02 15:04:05" }} <br>
    恢复时间: {{ .EndsAt.Format "2006-01-02 15:04:05" }} <br>
    ========= = end =  =========<br>
    {{ end }}{{ end -}}

    {{- end }}

3、测试验证

钉钉成功收到告警

邮件成功收到告警