prometheus + alertmanager 搭建告警通知

prometheus

下载prometheus-2.53.2

prometheus.yml文件修改

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 127.0.0.1:9093

rule_files:
  - "rules/rule-*.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090", "127.0.0.1:9104"]

其中127.0.0.1:9104是mysqld_exporter的metric地址

新建rules目录，并创建规则 rule-first.yml

groups:
  - name: InstanceDown_Rule
    rules:
      - alert: InstanceDown  # 告警名称
        expr: up == 0        # 告警条件
        for: 30s              # 告警触发前需要持续满足条件的时间
        labels:
          severity: critical # 告警的严重程度
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "Instance {{ $labels.instance }} has been down for more than 5 minutes."

启动Prometheus

prometheus --config.file=prometheus.yml --storage.tsdb.path=./data --web.enable-lifecycle

访问Prometheus

http://127.0.0.1:9090/

alertmanager

下载alertmanager-0.27.0

修改配置文件alertmanager.yml

route:
  group_by: ['instance']
  group_wait: 10s
  group_interval: 20s
  #repeat_interval: 1h
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/alert/hook'
        send_resolved: true
#inhibit_rules:
#  - source_match:
#      severity: 'critical'
#    target_match:
#      severity: 'warning'
#    equal: ['instance']

其中 http://127.0.0.1:5001/alert/hook 是接收告警的钩子接口

启动alertmanager

alertmanager --config.file=alertmanager.yml

访问alertmanager

http://127.0.0.1:9093/

grafana

下载grafana-11.2.1

启动grafana

grafana-server

访问grafana

http://127.0.0.1:3000

mysqld_exporter

下载 mysqld_exporter-0.15.1

在mysqld_exporter根目录创建.my.cnf文件

[client]
user=root
password=root

user和password分别是MySQL数据库的用户和密码；mysqld_exporter需要安装与mysql_server同一个服务器上。

启动mysqld_exporter

mysqld_exporter

访问mysqld_exporter

http://127.0.0.1:9104/

编写hook

@PostMapping("/alert/hook")
public Response<String> alertHook(@RequestBody Map<String, Object> alertDataMap) {
	//TODO 在这里实现告警处理，发微信、邮件、钉钉都可以
	System.out.println(JSON.toJSONString(alertDataMap));
	return Response.ok("success");
}

以上使用Java编写接口，根据实际可以使用任何一种语言编写。

可以关闭mysqld_exporter，模拟服务宕机。/alert/hook接口收到的数据如下：

{
	"receiver": "web\\.hook",
	"status": "firing",
	"alerts": [{
		"status": "firing",
		"labels": {
			"alertname": "InstanceDown",
			"instance": "127.0.0.1:9104",
			"job": "prometheus",
			"severity": "critical"
		},
		"annotations": {
			"description": "Instance 127.0.0.1:9104 has been down for more than 5 minutes.",
			"summary": "Instance 127.0.0.1:9104 down"
		},
		"startsAt": "2024-10-10T11:27:58.11Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://olive-my:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1",
		"fingerprint": "106b3a6075af7628"
	}],
	"groupLabels": {
		"instance": "127.0.0.1:9104"
	},
	"commonLabels": {
		"alertname": "InstanceDown",
		"instance": "127.0.0.1:9104",
		"job": "prometheus",
		"severity": "critical"
	},
	"commonAnnotations": {
		"description": "Instance 127.0.0.1:9104 has been down for more than 5 minutes.",
		"summary": "Instance 127.0.0.1:9104 down"
	},
	"externalURL": "http://olive-my:9093",
	"version": "4",
	"groupKey": "{}:{instance=\"127.0.0.1:9104\"}",
	"truncatedAlerts": 0
}