prometheus alertmanager 自定义webhook

目的：配置 alertmanager，使其能够以 webhook 的方式触发告警的推送

 一、编写webhook

解析告警内容并发送短信（此处以简单打印为例）

根据flask不同的启动方式，web打包成docker镜像有不同写法

写法一 

from flask import Flask, request
import json
app = Flask(__name__)

@app.route('/send', methods=['POST'])
def send():
    try:
        data = json.loads(request.data)
        alerts =  data['alerts']
        for i in alerts:
            print('SEND SMS: ' + str(i))
    except Exception as e:
        print(e)
    return 'ok'
if __name__ == '__main__':
#    app.run()
    app.run(host='0.0.0.0', port=15000,debug = True)

打包成镜像，编写 Dockerfile

FROM rockylinux:9
MAINTAINER lishanbin@petalmail.com
 
#将源改为国内的阿里源
RUN sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
    -i.bak \
    /etc/yum.repos.d/rocky-*.repo
 
RUN dnf makecache
 
RUN  dnf install -y yum
RUN yum -y install epel-release 
RUN yum  install python3-pip -y
RUN pip3 install flask
COPY app.py /app.py



EXPOSE 15000

CMD ["python3", "/app.py"]

打包/运行镜像：

 docker build -t registry.cn-shenzhen.aliyuncs.com/lishanbin/flask-alert:15000.0 .
 docker run -d --name flask -p 15000:15000 registry.cn-shenzhen.aliyuncs.com/lishanbin/flask-alert:15000

写法二：

from flask import Flask, request
import json
app = Flask(__name__)

@app.route('/send', methods=['POST'])
def send():
    try:
        data = json.loads(request.data)
        alerts =  data['alerts']
        for i in alerts:
            print('SEND SMS: ' + str(i))
    except Exception as e:
        print(e)
    return 'ok'

打包成镜像，编写 Dockerfile

#cat Dockerfile
FROM rockylinux:9
MAINTAINER lishanbin@petalmail.com
 
#将源改为国内的阿里源
RUN sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
    -i.bak \
    /etc/yum.repos.d/rocky-*.repo
 
RUN dnf makecache
 
RUN  dnf install -y yum
RUN yum -y install epel-release 
RUN yum  install python3-pip -y
RUN pip3 install flask
COPY app.py /app.py
COPY run.sh /run.sh
RUN chmod +x /run.sh

EXPOSE 5000

ENTRYPOINT ["/run.sh"]

cat run.sh 
#!/bin/bash
cd /
export FLASK_APP=app.py
flask run -h 0.0.0.0 -p 5000

打包/运行镜像：

 docker build -t registry.cn-shenzhen.aliyuncs.com/lishanbin/flask-alert:5000.0 .
 docker run -d --name flask -p 5000:5000 registry.cn-shenzhen.aliyuncs.com/lishanbin/flask-alert:5000

在k8s环境中部署flask-webhook镜像(可选)

前面可以直接使用python app.py 、flask run -h 0.0.0.0 -p 15000、docker直接启动程序了。

cat flask-webhook.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-alert-deployment
  namespace: monitor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-alert
  template:
    metadata:
      labels:
        app: flask-alert
    spec:
      containers:
        - name: flask-alert
          image: registry.cn-shenzhen.aliyuncs.com/lishanbin/flask-alert:1.0 
          imagePullPolicy: Always
          ports:
            - containerPort: 5000
              name: http
          volumeMounts:
            - name: localtime
              mountPath: /etc/localtime
      volumes:
        - name: localtime
          hostPath:
            path: /etc/localtime
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: flask-alert-service
  name: flask-alert-service
  namespace: monitor
spec:
  clusterIP: None
  ports:
  - name: http
    port: 5000
    targetPort: http
  selector:
    app: flask-alert

二、alertmanger调用webhook

    receivers:  ##定义劲爆接受者信息
    - name: 'alert-receiver'
      webhook_configs:
        #      - url: 'http://10.11.12.13:15000/send'
      - url: 'http://10.11.12.13:15000/send'

三、触发告警、查看webhook打印信息

我这里使用接口调用，实际告警也是可以的

shell脚本方式调用

cat  alerts_message.sh
#!/usr/bin/env bash
alerts_message='[
  {
    "labels": {
       "alertname": "磁盘已满",
       "dev": "sda1",
       "instance": "实例1",
       "msgtype": "testing"
     },
     "annotations": {
        "info": "程序员小王提示您：这个磁盘sda1已经满了，快处理！",
        "summary": "请检查实例示例1"
      }
  },
  {
    "labels": {
       "alertname": "磁盘已满",
       "dev": "sda2",
       "instance": "实例1",
       "msgtype": "testing"
     },
     "annotations": {
        "info": "程序员小王提示您：这个磁盘sda2已经满了，快处理！",
        "summary": "请检查实例示例1",
        "runbook": "以下链接http://test-url应该是可点击的"
      }
  }
]'
curl -XPOST -d"$alerts_message" http://10.11.12.13:31192/api/v1/alerts

运行：sh alerts_message.sh

postman方式调用（推荐）

 

json格式

[
  {
    "labels": {
       "alertname": "系统连续崩溃，已经出现雪崩状况！",
       "dev": "sda1",
       "instance": "实例1",
       "msgtype": "testing"
     },
     "annotations": {
        "info": "程序员小王提示您：这个系统雪崩了，快处理！",
        "summary": "请检查实例示例1"
      }
  },
  {
    "labels": {
       "alertname": "管理系统损坏",
       "dev": "sda2",
       "instance": "实例1",
       "msgtype": "testing"
     },
     "annotations": {
        "info": "程序员小王提示您：电子商务管理系统中订单，仓库模块已经雪崩，快处理！",
        "summary": "请检查实例示例1",
        "runbook": "以下链接http://192.168.5.128:9093/api/v2/alerts应该是可点击的"
      }
  }
]

四、打印告警信息（模拟短信发送）

有个问题：发现打包成docker镜像后，docker日志无法打印出json告警，请求是可以打印的。