记:Altermanager告警信息无法发送问题

于 2024-02-22 23:47:50 发布
阅读量619 收藏 9
点赞数 7
文章标签： prometheus
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/phsycaler/article/details/136245355
版权
文章讲述了在将监控系统从本地部署到生产环境时，作者遇到普罗米修斯发送告警邮件失败的问题。经过排查，发现问题出在服务器时间不同步，导致Alertmanager将告警标记为已解决。通过调整时间并延长resolve_timeout设置，解决了告警接收问题。
摘要由CSDN通过智能技术生成

问题描述

之前在本地已经搭建好普罗米修斯+Alertmanager监控告警，并配置了邮箱告警，测试后没问题，能收到告警邮件。但是部署到生产环境上后，就收不到了。

生产环境服务器：

机房1和机房2是两个不同的网段，两个网络的应用程序要互通，需要开通策略，并且策略已开通

环境

IP

服务

机房1

172.19.57.28:9090

普罗米修斯

机房2

172.20.134.67:8025

alertmanager

这么部署理论上没有问题，普罗米修斯也能监控到alertmanager状态，说明开通的策略没问题

排查

  1. 使用curl手动调用添加告警信息api
curl -X POST -H "Content-Type: application/json" -d '[{"labels":{"alertname":"系统连续崩溃，已经出现雪崩状况！","dev":"sda1","instance":"实例1","msgtype":"testing"}}]' http://172.17.255.142:8025/api/v2/alerts

结果收到了告警信息邮件

说明alert自身没有问题，邮箱配置也没有问题，那么问题可能出在了普罗米修斯把告警信息发送给alert的环节中。

  1. 使用wireshark在alert服务器抓包

安装wireshark

yum -y install wireshark

抓取url包含/api/v2/alerts的数据包

tshark -i eth0 -Y "ip.dst == 172.20.134.67 and tcp.dstport == 8025 and http.request.uri contains \"/api/v2/alerts\"" -V

数据包输出的信息：

Frame 1130: 607 bytes on wire (4856 bits), 607 bytes captured (4856 bits) on interface 0
    Interface id: 0
    Encapsulation type: Ethernet (1)
    Arrival Time: Jan 23, 2024 11:40:36.058137187 CST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1705981236.058137187 seconds
    [Time delta from previous captured frame: 0.245109469 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 45.302038112 seconds]
    Frame Number: 1130
    Frame Length: 607 bytes (4856 bits)
    Capture Length: 607 bytes (4856 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:tcp:http:json]
Ethernet II, Src: c0:ff:a8:25:d2:06 (c0:ff:a8:25:d2:06), Dst: d0:0d:e5:3a:2f:59 (d0:0d:e5:3a:2f:59)
    Destination: d0:0d:e5:3a:2f:59 (d0:0d:e5:3a:2f:59)
        Address: d0:0d:e5:3a:2f:59 (d0:0d:e5:3a:2f:59)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: c0:ff:a8:25:d2:06 (c0:ff:a8:25:d2:06)
        Address: c0:ff:a8:25:d2:06 (c0:ff:a8:25:d2:06)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 172.18.1.130 (172.18.1.130), Dst: 172.20.134.67 (172.20.134.67)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 593
    Identification: 0xefeb (61419)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 56
    Protocol: TCP (6)
    Header checksum: 0x70cf [validation disabled]
        [Good: False]
        [Bad: False]
    Source: 172.18.1.130 (172.18.1.130)
    Destination: 172.20.134.67 (172.20.134.67)
Transmission Control Protocol, Src Port: 56798 (56798), Dst Port: ca-audit-da (8025), Seq: 1, Ack: 2, Len: 541
    Source port: 56798 (56798)
    Destination port: ca-audit-da (8025)
    [Stream index: 10]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 542    (relative sequence number)]
    Acknowledgment number: 2    (relative ack number)
    Header length: 32 bytes
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 229
    [Calculated window size: 229]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x2ef0 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        Timestamps: TSval 2167850496, TSecr 4192078536
            Kind: Timestamp (8)
            Length: 10
            Timestamp value: 2167850496
            Timestamp echo reply: 4192078536
    [SEQ/ACK analysis]
        [Bytes in flight: 541]
Hypertext Transfer Protocol
    POST /api/v2/alerts HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): POST /api/v2/alerts HTTP/1.1\r\n]
            [Message: POST /api/v2/alerts HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: POST
        Request URI: /api/v2/alerts
        Request Version: HTTP/1.1
    Host: 172.17.255.142:8025\r\n
    User-Agent: Prometheus/2.45.0\r\n
    Content-Length: 398\r\n
        [Content length: 398]
    Content-Type: application/json\r\n
    \r\n
    [Full request URI: http://172.17.255.142:8025/api/v2/alerts]
    [HTTP request 1/1]
JavaScript Object Notation: application/json
    Array
        Object
            Member Key: "annotations"
                Object
                    Member Key: "description"
                        String value: Mysql数据库宕机，请检查
                    Member Key: "summary"
                        String value: 您的 172.19.57.19:9105 的Mysql已停止运行！
            Member Key: "endsAt"
                String value: 2024-01-23T03:39:32.337Z
            Member Key: "startsAt"
                String value: 2024-01-23T02:49:17.337Z
            Member Key: "generatorURL"
                String value: http://zonghe31:9090/graph?g0.expr=mysql_up+%3D%3D+0&g0.tab=1
            Member Key: "labels"
                Object
                    Member Key: "alertname"
                        String value: Mysql status
                    Member Key: "instance"
                        String value: 172.19.57.19:9105
                    Member Key: "job"
                        String value: mysql8_node
                    Member Key: "severity"
                        String value: error


Frame 1507: 570 bytes on wire (4560 bits), 570 bytes captured (4560 bits) on interface 0
    Interface id: 0
    Encapsulation type: Ethernet (1)
    Arrival Time: Jan 23, 2024 11:40:54.797106494 CST
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1705981254.797106494 seconds
    [Time delta from previous captured frame: 0.132284603 seconds]
    [Time delta from previous displayed frame: 18.738969307 seconds]
    [Time since reference or first frame: 64.041007419 seconds]
    Frame Number: 1507
    Frame Length: 570 bytes (4560 bits)
    Capture Length: 570 bytes (4560 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ip:tcp:http:json]
Ethernet II, Src: c0:ff:a8:25:d2:06 (c0:ff:a8:25:d2:06), Dst: d0:0d:e5:3a:2f:59 (d0:0d:e5:3a:2f:59)
    Destination: d0:0d:e5:3a:2f:59 (d0:0d:e5:3a:2f:59)
        Address: d0:0d:e5:3a:2f:59 (d0:0d:e5:3a:2f:59)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: c0:ff:a8:25:d2:06 (c0:ff:a8:25:d2:06)
        Address: c0:ff:a8:25:d2:06 (c0:ff:a8:25:d2:06)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IP (0x0800)
Internet Protocol Version 4, Src: 172.18.1.130 (172.18.1.130), Dst: 172.20.134.67 (172.20.134.67)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00)
    Total Length: 556
    Identification: 0xefef (61423)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 56
    Protocol: TCP (6)
    Header checksum: 0x70f0 [validation disabled]
        [Good: False]
        [Bad: False]
    Source: 172.18.1.130 (172.18.1.130)
    Destination: 172.20.134.67 (172.20.134.67)
Transmission Control Protocol, Src Port: 56798 (56798), Dst Port: ca-audit-da (8025), Seq: 542, Ack: 116, Len: 504
    Source port: 56798 (56798)
    Destination port: ca-audit-da (8025)
    [Stream index: 10]
    Sequence number: 542    (relative sequence number)
    [Next sequence number: 1046    (relative sequence number)]
    Acknowledgment number: 116    (relative ack number)
    Header length: 32 bytes
    Flags: 0x018 (PSH, ACK)
        000. .... .... = Reserved: Not set
        ...0 .... .... = Nonce: Not set
        .... 0... .... = Congestion Window Reduced (CWR): Not set
        .... .0.. .... = ECN-Echo: Not set
        .... ..0. .... = Urgent: Not set
        .... ...1 .... = Acknowledgment: Set
        .... .... 1... = Push: Set
        .... .... .0.. = Reset: Not set
        .... .... ..0. = Syn: Not set
        .... .... ...0 = Fin: Not set
    Window size value: 229
    [Calculated window size: 229]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0xfb7d [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        No-Operation (NOP)
            Type: 1
                0... .... = Copy on fragmentation: No
                .00. .... = Class: Control (0)
                ...0 0001 = Number: No-Operation (NOP) (1)
        Timestamps: TSval 2167869235, TSecr 4192104776
            Kind: Timestamp (8)
            Length: 10
            Timestamp value: 2167869235
            Timestamp echo reply: 4192104776
    [SEQ/ACK analysis]
        [Bytes in flight: 504]
Hypertext Transfer Protocol
    POST /api/v2/alerts HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): POST /api/v2/alerts HTTP/1.1\r\n]
            [Message: POST /api/v2/alerts HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: POST
        Request URI: /api/v2/alerts
        Request Version: HTTP/1.1
    Host: 172.17.255.142:8025\r\n
    User-Agent: Prometheus/2.45.0\r\n
    Content-Length: 361\r\n
        [Content length: 361]
    Content-Type: application/json\r\n
    \r\n
    [Full request URI: http://172.17.255.142:8025/api/v2/alerts]
    [HTTP request 2/2]
    [Prev request in frame: 1130]
JavaScript Object Notation: application/json
    Array
        Object
            Member Key: "annotations"
                Object
                    Member Key: "description"
                        String value: 节点断联已超过1分钟！
                    Member Key: "summary"
                        String value: 节点失联
            Member Key: "endsAt"
                String value: 2024-01-23T03:39:51.075Z
            Member Key: "startsAt"
                String value: 2024-01-23T02:49:36.075Z
            Member Key: "generatorURL"
                String value: http://zonghe31:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1
            Member Key: "labels"
                Object
                    Member Key: "alertname"
                        String value: 实例存活告警
                    Member Key: "instance"
                        String value: 172.19.57.28:9100
                    Member Key: "job"
                        String value: server_node
                    Member Key: "severity"
                        String value: Disaster

可以看到第106行，告警信息传过来了，说明普罗米修斯传递的信息也没问题

  1. 日志调整为debug级，查看日志

通过以上排查证明了普罗米修斯和alertmanager都是正常运行的，哪么问题出在了哪里呢？

通过日志发现，alert接收到的告警信息都被标记成了resolved状态

正常情况状态应该为active，alert有个resolve_timeout配置项，默认5分钟，同一条告警信息，超过5分钟后没有再收到，就给标记成已解决状态，不再向外发送告警信息

于是想到了两台服务器的时间可能不一样，使用命令timedatectl查看，发现服务器的时间差了6分钟，将服务器时间修改一致后，能收到告警信息邮件了

修改时间命令：

date -s "yyyy-MM-dd HH:mm:ss"

PS：把resolve_timeout时间设置的长一些，发现没有用，还需要学习📖

小小小铭同学
关注
  • 7
    点赞
  • 踩
  • 9
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
Altermanager 发送告警报 x509: certificate signed by unknown authority 错误解决
01-08
这是alter manager配置默认需要证书，官方config解释： ...The default SMTP TLS requirement. Note that Go does not support unencrypted connections to remote SMTP endpoints. [ smtp_require_tls: | default = ...
alertmanager-0.15.3.linux-amd64.tar.gz
04-01
此外，还可以设置静默期（silences）来暂时阻止特定告警的发送。 在0.15.3版本中，Alertmanager可能已经包含了改进的性能和稳定性，以及对其他Prometheus组件的兼容性优化。不过，为了充分利用其功能，你需要确保与...
解决Prometheus发送报警alert manager没有收到告警，邮件告警没有发送
учёба
02-10 5598
第一步：确定alertmanager的配置文件没有写错 在alertmanager的安装目录中使用下面的命令进行查看 ./amtool check-config 配置文件名称 实际测试 [root@localhost alertmanager]# ./amtool check-config alertmanager.yml Checking 'alertmanager.yml' SUCCESS Found: - global config - route - 1 inhibit rules
记一次alertmanager发送邮件失败的处理过程
weixin_43557605的博客
03-23 6485
文章目录1 先验证smtp信息是否正确2 配置alertmanager配置文件并触发告警3 解决 smtp.plainAuth failed: wrong host name4 解决 dial tcp 127.0.0.1:5001: connect: connection refused5 解决 配置文件不对应的问题6 解决configmap跟挂载文件名不对应的问题 如下的过程记录了我遇到alertmanager不发送告警邮件后完整的排查过程，也贴有详细的报错信息，如果有时间可以通读，没时间可以直接在全文搜
Alertmanager告警恢复通知丢失排查
小太阳DENGJIE的博客
04-23 1150
Alertmanager告警恢复通知丢失问题排查
prometheus监控之alertmanager安装配置（2）接入电话报警微信、短信、邮件等告警
akzp123的博客
12-04 1万+
电话报警使用OPSALERT点击了解更多（支持电话报警、短信报警、邮件告警、微信报警）直接使用webhook配置即可，比较简单。 介绍 Prometheus 将数据采集和报警分成两个模块。报警规则配置在Prometheus Servers上，然后发送报警信息到AlertManger，Alertmanager 对收到的告警信息进行处理，包括去重，降噪，分组，沉默，抑制，策略路由，告警通知。 流程 Alertmanager 接收到告警，根据labels判断属于哪些Route（可存在多个Route，...
Alertmanager实现企业微信机器人webhook告警
最新发布
K-free的博客
01-15 2163
1. 说明 由于企业微信更新问题，现在已经无法直接使用创建应用后在alertmanager的配置文件中定义企业id及secret就可以发送告警信息了，除非填写备案后域名；为了我们这种个人开发者非常的不便，所以本文档是为了解决想使用企业微信告警但又无法备案的朋友；下面只是我的操作过程记录 如果没有自定义的需求可以直接使用我的镜像可以直接使用我的镜像：kfreesre/prometheus-fl...
Alertmanager 收到发送警报通知但是没有发出邮件
sanmao123456_的博客
03-20 5575
解决办法： 修改 alertmanager.yml 全局配置： smtp_require_tls: false SSL/TLS 学习 （TLS 是升级版的 SSL ，SSL被证明是有问题的。） 1.不使用 SSL/TLS 的 HTTP 通信的风险： 窃听 冒充 修改 2.对应的解决办法： 加密 身份验证 校验机制 3.TLS过程 TLS 基于 TCP 客户端发出请求 （1）支持的协议版...
alertmanager-webhook-logger:从Prometheus AlertManager Webhook通知程序生成（结构化）日志消息
05-11
Alertmanager Webhook记录器 从通知程序生成（结构化）日志消息。 基本原理 不提供任何警报历史记录。 记录警报是获取该历史记录的最简单解决方案。 结合使用诸如等的日志管理解决方案，它应适合大多数用例，以提供...
prometheus-alertmanager-cfg.yaml
04-14
prometheus结合altermanager的配置文件，其中关于k8s组件的静态监控配置请根据自身情况修改Ip地址。
从AlertManager接收HTTP Webhook通知，并将其插入到Elasticsearch索引中以进行搜索和分析-Golang开发
05-26
alertmanager2es alertmanager2es从AlertManager接收HTTP Webhook通知，并将其插入到Elasticsearch索引中以进行搜索和分析。 它作为守护程序运行。 ... 它作为守护程序运行。 警报作为警报组存储在Elasticsearch中。...
altermanager使用报错
weixin_30700099的博客
08-05 2167
报错如下： level=warn ts=2019-01-24T09:20:01.122920737Z caller=cluster.go:148 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"...
prometheus学习5altermanager
weixin_60092693的博客
08-10 649
仅自己学习笔记
AlertManager实现Email告警（使用Postman测试）
程序员小王的博客
03-20 3663
AlertManager实现Email告警（使用Postman测试）
alertmanager笔记
Re_Virtual的博客
02-11 907
prometheus alertmanager笔记，包含API调用，json格式和yml参数说明
Prometheus 监控报警系统 AlertManager 之邮件告警
热门推荐
哎_小羊的博客
08-04 6万+
Prometheus 是一套开源的系统监控、报警、时间序列数据库的组合，最初有 SoundCloud 开发的，后来随着越来越多公司使用，于是便独立成开源项目。Alertmanager 主要用于接收 Prometheus 发送的告警信息，它支持丰富的告警通知渠道，例如邮件、微信、钉钉、Slack 等常用沟通工具，而且很容易做到告警信息进行去重，降噪，分组等，是一款很好用的告警通知系统。本文介绍 Prometheus 监控报警系统 AlertManager 之邮件告警
prometheus使用Alertmanager预警（邮件）
yang.zhou欢迎各位
07-24 1万+
1.   Prometheus 安装 源码安装： # wget https://github.com/prometheus/prometheus/releases/download/v1.7.1/prometheus-1.7.1.linux-amd64.tar.gz # tar zxvf prometheus-1.7.1.linux-amd64.tar.gz # cd promethe
Prometheus学习笔记（三）Alertmanager报警模块集成钉钉/企业微信机器人
刘李404not_found的博客
08-06 7170
一、简介 Alertmanager是Prometheus的一个报警通知组件，需要结合Prometheus使用。Prometheus将监测到的异常事件发送给Alertmanager，Alertmanager发送异常事件的通知（邮件、webhook等）。 Prometheus安装文档 二、安装Alertmanager Alertmanager下载地址 # 解压安装包 tar -zxvf alertmanager-0.21.0.linux-amd64.tar.gz # 文件列表 ./ ├── alertmanag
docker安装altermanager
08-18
容器会将本地的alertmanager.yml文件挂载到容器内的指定位置，这样你就可以配置AlertManager的告警规则了。<span class="em">1</span><span class="em">2</span><span class="em">3 #### 引用[.reference_title] - *...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值