Prometheus&Grafana监控报警安装配置

最新推荐文章于 2024-05-29 13:28:44 发布

at sea .

最新推荐文章于 2024-05-29 13:28:44 发布

阅读量1.8k

点赞数

分类专栏：监控 Linux

本文链接：https://blog.csdn.net/qq_44078641/article/details/118970978

版权

Linux 同时被 2 个专栏收录

9 篇文章 1 订阅

订阅专栏

监控

3 篇文章 0 订阅

订阅专栏

一、软件版本

1、prometheus-2.26.0.linux-amd64
2、blackbox_exporter-0.18.0.linux-amd64
3、alertmanager-0.22.2.linux-amd64
4、grafana-7.3.7

二、Prometheus及相关组件安装配置

1、解压Prometheus

tar -xf  prometheus-2.26.0.linux-amd64.tar.gz

2、主配置文件修改

vim prometheus.yml

# my global config
global:
  #每隔15s去拉取agent端指标信息
  scrape_interval:     15s 
  #每隔15s检测报警规则
  evaluation_interval: 15s 


# Alertmanager configuration
#使用altermanager 报警
alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "127.0.0.1:9093"

#指定报警的规则文件
rule_files:
  - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  #指标名称
  - job_name: 'saas-服务状态'
    scrape_interval: 15s
    #指标的路径,blackbox的路径
    metrics_path: /probe
    params:
      module: [http_2xx]  # 模块对应 blackbox.yml 
    static_configs:
      - targets:
        - http://ip/JKJCDo    # http
        labels:
          a_service_name: 'saas-(测试)'
      - targets:
        - http://ip/JKJCDo    # http
        labels:
          a_service_name: 'saas-(API)'
          #如果是java应用,这个值用于Grafana面板跳转jvm指标
          c_java_svc_add: 'ip:port'
      - targets:
        - http://ip:port/path    # http
        labels:
          a_service_name: '(API,开票员)'
          c_java_svc_add: 'ip:port'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: b_service_address
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox安装在哪台机器

  - job_name: '管理服务状态'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]  
    static_configs:
      - targets:
        - http://ip:port/path     
        labels:
          a_service_name: '(需求系统)'
      - targets:
        - http://ip:port/path  
        labels:
          a_service_name: '(工单系统)'
      - targets:
        - http://ip:port/path       
        labels:
          a_service_name: '(综合业务管理平台)'
          c_java_svc_add: 'ip:port'
      - targets:
        - http://ip:port/path   
        labels:
          a_service_name: '(云盘)'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: b_service_address
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # blackbox安装在哪台机器


#prometheus
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:19092']
    
#主机状态
  - job_name: 'saas_node'
    scrape_interval: 15s
    static_configs:
    - targets: ['ip:port','ip:port','ip:port']
      labels:
        group: '(开票员,管理员,api)'
    - targets: ['ip:port']
      labels:
        group: '(综合业务管理平台)'
    - targets: ['ip:port']
      labels:
        group: '(发票上传)'
    - targets: ['ip:port']
      labels:
        group: '(物联网)'
    - targets: ['ip:port']
      labels:
        group: '(版式文件)'

        
  #java agent指标抓取
  - job_name: 'java应用状态'
    scrape_interval: 15s
    static_configs:
    - targets: ['ip:port']
      labels:
        service: '(综合业务管理平台)'
    - targets: ['ip:port']
      labels:
        service: '(开票员,api)'
                
#  - job_name: 'pushgateway'
#    static_configs:
#    - targets: ['127.0.0.1:19091']
#      labels:
#        instance: pushgateway

3、报警规则文件修改

vim first_rules.yml
groups:
  - name: SAAS主机在线状态
    rules:
    - alert: node-status
      expr: up{job="saas_node"} == 0
      for: 1m
      labels:
        severity: warning
      annotations:
        business: "{{ $labels.group }}"
        remarks: "SAAS主机:  {{ $labels.instance }} 已经离线超过一分钟!"
        recover: "SAAS主机:  {{ $labels.instance }} 已经恢复"
        values: "{{  $value }}"
        description: '"up{job="saas_node"} == 0"'
        summary: "{{ $labels.instance }}"
  - name: 发票系统应用状态
    rules:
    - alert: 发票系统应用状态
      expr: probe_success{job="saas-发票服务状态"} == 0
      for: 1m
      labels:
      annotations:
        #业务
        business: "{{ $labels.a_service_name }}"
        #备注
        remarks: "SAAS 应用: {{ $labels.a_service_name }} 已经离线超过一分钟！健康检查地址为: {{ $labels.b_service_address }}"
        recover: "SAAS 应用: {{ $labels.a_service_name }} 已经恢复！健康检查地址为: {{ $labels.b_service_address }}"
        #报警值
        values: "{{ $value }}"
        #报警规则
        description: '"probe_success{job="saas-发票服务状态"} == 0"'
        #正在报警的资源
        summary: "{{ $labels.instance }}"
  - name: 管理系统应用状态
    rules:
    - alert: 管理系统应用状态
      expr: probe_success{job="管理服务状态"} == 0
      for: 1m
      labels:
      annotations:
        #业务
        business: "{{ $labels.a_service_name }}"
        #备注
        remarks: "SAAS 应用: {{ $labels.a_service_name }} 已经离线超过一分钟！健康检查地址为: {{ $labels.b_service_address }}"
        recover: "SAAS 应用: {{ $labels.a_service_name }} 已经恢复！健康检查地址为: {{ $labels.b_service_address }}"
        #报警值
        values: "{{ $value }}"
        #报警规则
        description: '"probe_success{job="管理服务状态"} == 0"'
        #正在报警的资源
        summary: "{{ $labels.instance }}"
  - name: 主机CPU使用率监测
    rules:
    - alert: 主机CPU使用率监测 
      expr: 100 - ((avg by (instance,group)(irate(node_cpu_seconds_total{mode="idle"}[30s]))) *100) > 80
      for: 1m
      labels:
      annotations:
        #业务
        business: "{{ $labels.group }}"
        #备注
        remarks: "SAAS主机: {{ $labels.group }} 已经连续1分钟使用率超过80%！主机信息为: {{ $labels.instance }}"
        recover: "SAAS主机: {{ $labels.group }} 已经恢复！主机信息为: {{ $labels.instance }}"
        #报警值
        values: "{{ $value }}"
        #报警规则
        description: '"100 - ((avg by (instance,group)(irate(node_cpu_seconds_total{mode="idle"}[30s]))) *100) > 30"'
        #正在报警的资源
        summary: "{{ $labels.instance }}"
  - name: 主机内存使用率监测
    rules:
    - alert: 主机内存使用率监测 
      expr: ((node_memory_MemTotal_bytes -(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes) )/node_memory_MemTotal_bytes ) * 100 > 90
      for: 1m
      labels:
      annotations:
        #业务
        business: "{{ $labels.group }}"
        #备注
        remarks: "SAAS主机: {{ $labels.group }} 已经连续1分钟内存使用率超过90%！主机信息为: {{ $labels.instance }}"
        recover: "SAAS主机: {{ $labels.group }} 已经恢复！主机信息为: {{ $labels.instance }}"
        #报警值
        values: "{{ $value }}"
        #报警规则
        description: '"((node_memory_MemTotal_bytes -(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes) )/node_memory_MemTotal_bytes ) * 100 > 90"'
        #正在报警的资源
        summary: "{{ $labels.instance }}"
  - name: 主机磁盘使用率监测
    rules:
    - alert: 主机磁盘使用率监测 
      expr: max((node_filesystem_size_bytes{job=~"saas_node",fstype=~"ext.?|xfs"}-node_filesystem_free_bytes{job=~"saas_node",fstype=~"ext.?|xfs"}) *100/(node_filesystem_avail_bytes {job=~"saas_node",fstype=~"ext.?|xfs"}+(node_filesystem_size_bytes{job=~"saas_node",fstype=~"ext.?|xfs"}-node_filesystem_free_bytes{job=~"saas_node",fstype=~"ext.?|xfs"})))by(instance,group)  > 90
      for: 1m
      labels:
      annotations:
        #业务
        business: "{{ $labels.group }}"
        #备注
        remarks: "SAAS主机: {{ $labels.group }} 已经连续1分钟磁盘使用率超过90%！主机信息为: {{ $labels.instance }}"
        recover: "SAAS主机: {{ $labels.group }} 已经恢复！主机信息为: {{ $labels.instance }}"
        #报警值
        values: "{{ $value }}"
        #报警规则
        description: '"max((node_filesystem_size_bytes{job=~"saas_node",fstype=~"ext.?|xfs"}-node_filesystem_free_bytes{job=~"saas_node",fstype=~"ext.?|xfs"}) *100/(node_filesystem_avail_bytes {job=~"saas_node",fstype=~"ext.?|xfs"}+(node_filesystem_size_bytes{job=~"saas_node",fstype=~"ext.?|xfs"}-node_filesystem_free_bytes{job=~"saas_node",fstype=~"ext.?|xfs"})))by(instance) > 90 "'
        #正在报警的资源
        summary: "{{ $labels.instance }}"

4、blackbox配置

vim blackbox.yml
modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp

5、Alertmanager配置

5.1、Alertmanager主配置文件

vim alertmanager.yml
global:
  #resolve_timeout：解析超时时间
  resolve_timeout: 5m
  #smtp_smarthost: 使用email打开服务配置
  smtp_smarthost: 'smtp.163.com:465'
  #smtp_from：指定通知报警的邮箱
  smtp_from: 'ymy23123@163.com'
  #smtp_auth_username：邮箱用户名
  smtp_auth_username: 'ymy23123@163.com'
  #smtp_auth_password：授权密码
  smtp_auth_password: ''
  #smtp_require_tls：是否启用tls
  smtp_require_tls: false

templates:
  - '/home/yunwei1/software/prometheus/prometheus-2.26.0.linux-amd64/alertmanager-0.22.2.linux-amd64/template/*'
route:
  #group_by：采用哪个标签作为分组的依据
  group_by: ['alertname','instance']
  #group_wait：分组等待的时间  #当传入的警报创建了一组新的警报时，请至少等待多少秒发送初始通知
  group_wait: 30s
  #group_interval：上下两组发送告警的间隔时间 发送第一个通知时，请等待多少分钟发送一批已开始为该组触发的新警报
  group_interval: 1m 
  #repeat_interval：重复发送告警时间。默认1h 如果警报已成功发送，请等待多少小时以重新发送警报
  repeat_interval: 60m
  #receiver 定义谁来通知报警
  receiver: 'mail'
receivers:
# name：报警来源自定义名称
- name: 'mail'
# email_configs：通过邮箱发送报警
  email_configs:
# to：指定接收端email
  - to: '2312398095@qq.com' 
    #html: '{{ template "email2.to.html" . }}' 
    html: '{{ template "test.html" . }}' 
    #设置报警标题
    headers: 
      Subject: '【报警】 类型: {{ .GroupLabels.alertname }} ,实例: {{ range .Alerts }}  {{ .Annotations.summary }} {{ end }},发生告警,请及时查看!' 
    send_resolved: true
#- name: 'web.hook'
#  webhook_configs:
#  - url: 'http://127.0.0.1:5001/'
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['instance']

5.2、邮件推送模板

vim test.html
{{ define "test.html" }}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<div id="qm_con_body"><div id="mailContentContainer" class="qmbox qm_con_body_content qqmail_webmail_only" style="">


<style type="text/css">
    .qmbox body {
        margin: 0 auto;
        padding: 0;
        font-family: Microsoft Yahei, Tahoma, Arial;
        color: #333333;
        background-color: #fff;
        font-size: 12px;
    }

    .qmbox a {
        color: #00a2ca;
        line-height: 22px;
        text-decoration: none;
    }

    .qmbox a:hover {
        text-decoration: underline;
        color: #00a2ca;
    }

</style>






<table width="800" border="0" align="center" cellpadding="0" cellspacing="0" bgcolor="#ffffff">
    <tbody><tr>
        <td>
            <table width="800" border="0" align="center" cellpadding="0" cellspacing="0" height="40"></table>
        </td>
    </tr>

    <tr>
        <td>

            <table width="800" border="0" align="center" cellpadding="0" cellspacing="0" bgcolor="#373d41" height="48" style="">
                <tbody><tr>
                    <td width="74" height="48" border="0" align="center" valign="middle" style="padding-left:20px;">
                        <a href="http://monitor.newtimeai.com" target="_blank" data-spm-anchor-id="0.0.0.0" rel="noopener">
                            <img src="https://img-blog.csdnimg.cn/2022010701240754460.png" width="50%" height="50%" border="0">
                        </a>
                    </td>
                    <td width="703" height="48" colspan="2" align="right" valign="middle" style="color:#ffffff; padding-right:20px;">
                        <a href="http://fp.newtimeai.com" target="_blank" style="" rel="noopener">发票管理云平台</a>
                        &nbsp;&nbsp; <span style="color:#6c7479;">|</span>&nbsp;
                        <a href="http://monitor.newtimeai.com" target="_blank" style="" rel="noopener">监控中心</a>
                    </td>
                </tr>
                </tbody></table>

        </td>
    </tr>

    <tr>
        <td>

            <table width="800" border="0" align="left" cellpadding="0" cellspacing="0" style="   font-size:14px;color:#333333;  border:1px solid #edecec;  padding: 10px">
                <tbody>
                <tr>
                    <td height="56" border="0" align="left" colspan="2" style=" font-size:16px;vertical-align:bottom;">
                        尊敬的***管理员，您好：
                    </td>
                </tr>
                <tr>
                    <td height="30" border="0" align="left" colspan="2">&nbsp;</td>
                </tr>

                        <tr>
                           <td width="40" height="32" border="0" align="left" valign="middle" style=" width:40px; text-align:left;vertical-align:middle; line-height:32px;">
                            <img width="32" height="32" border="0" src="https://img-blog.csdnimg.cn/2022010701240759666.png">
                        </td>
                            <td height="32" style="word-wrap: break-word;word-break: normal;word-break:break-all;">
                                您的以下资源发生告警, <a href="" rel="noopener" target="_blank">{{ .Labels.alertname }}</a>
                            </td>
                        </tr>
                        <tr>
                            <td height="15" border="0" align="left" colspan="2">&nbsp;</td>
                        </tr>
                            <tr>
                                <td height="32" colspan="2" style="padding-left:40px;border-color: #666666">
                                    <table width="680" border="1" align="center" cellspacing="0" height="48" style="margin-bottom: 10px;margin-left: 0px;">
                                        <thead>
                                        <tr><th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            业务
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            正在报警的资源
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            当前值
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            触发时间
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            规则描述
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            备注
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            Region
                                        </th>
                                        </tr></thead>

                                        <tbody>
                                                                                        <tr>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
        {{ .Annotations.business }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
<a href="" rel="noopener" target="_blank">{{ .Annotations.summary }}</a>
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ .Annotations.values }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05"}}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ .Annotations.description }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ .Annotations.remarks }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
华北3(张家口)
    </td>
                                            </tr>
                                                                                    </tbody>
                                    </table>
                                </td>
                            </tr>

                        <tr>
                            <td height="32" colspan="2" style="padding-left:40px;word-wrap: break-word;word-break: normal;word-break:break-all;">
                                <a href="http://monitor.newtimeai.com">详情请登录智云监控</a>
                            </td>
                        </tr>

                <tr>
                    <td height="30" border="0" align="left" colspan="2">&nbsp;</td>
                </tr>
                <tr>
                    <td height="14" colspan="2">
                        <a href="http://monitor.newtimeai.com/login" rel="noopener" target="_blank">智云监控</a>
                    </td>
                </tr>

                <tr>
                    <td height="14" colspan="2">电话咨询：*****</td>
                </tr>


                <tr>
                    <td height="30" border="0" align="left" colspan="2">&nbsp;</td>
                </tr>
                <tr>
                    <td height="14" colspan="2" style="padding-bottom:16px; border-bottom:1px dashed #e5e5e5;">
                       ***有限公司
                    </td>
                </tr>
                <tr>
                    <td height="14" colspan="2" style="padding:8px 0 28px;color:#999999; font-size:12px;">
                        此为系统邮件请勿回复
                    </td>
                </tr>
                </tbody>
            </table>

        </td>
    </tr>

    <tr>
        <td>
            <table width="800" height="100" border="0" align="center" cellpadding="0" cellspacing="0">
                <tbody>
                <tr>
                    <td width="800" height="100" align="center" valign="middle">
                    </td>
                </tr>
                </tbody>
            </table>
        </td>
    </tr>
</tbody></table>
<style type="text/css">.qmbox style, .qmbox script, .qmbox head, .qmbox link, .qmbox meta {display: none !important;}</style></div></div>
</body>
</html>
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
</head>
<body>
<div id="qm_con_body"><div id="mailContentContainer" class="qmbox qm_con_body_content qqmail_webmail_only" style="">


<style type="text/css">
    .qmbox body {
        margin: 0 auto;
        padding: 0;
        font-family: Microsoft Yahei, Tahoma, Arial;
        color: #333333;
        background-color: #fff;
        font-size: 12px;
    }

    .qmbox a {
        color: #00a2ca;
        line-height: 22px;
        text-decoration: none;
    }

    .qmbox a:hover {
        text-decoration: underline;
        color: #00a2ca;
    }

</style>






<table width="800" border="0" align="center" cellpadding="0" cellspacing="0" bgcolor="#ffffff">
    <tbody><tr>
        <td>
            <table width="800" border="0" align="center" cellpadding="0" cellspacing="0" height="40"></table>
        </td>
    </tr>

    <tr>
        <td>

            <table width="800" border="0" align="center" cellpadding="0" cellspacing="0" bgcolor="#373d41" height="48" style="">
                <tbody><tr>
                    <td width="74" height="48" border="0" align="center" valign="middle" style="padding-left:20px;">
                        <a href="http://monitor.newtimeai.com" target="_blank" data-spm-anchor-id="0.0.0.0" rel="noopener">
                            <img src="https://img-blog.csdnimg.cn/2022010701240754460.png" width="50%" height="50%" border="0">
                        </a>
                    </td>
                    <td width="703" height="48" colspan="2" align="right" valign="middle" style="color:#ffffff; padding-right:20px;">
                        <a href="http://fp.newtimeai.com" target="_blank" style="" rel="noopener">发票管理云平台</a>
                        &nbsp;&nbsp; <span style="color:#6c7479;">|</span>&nbsp;
                        <a href="http://monitor.newtimeai.com" target="_blank" style="" rel="noopener">监控中心</a>
                    </td>
                </tr>
                </tbody></table>

        </td>
    </tr>

    <tr>
        <td>

            <table width="800" border="0" align="left" cellpadding="0" cellspacing="0" style="   font-size:14px;color:#333333;  border:1px solid #edecec;  padding: 10px">
                <tbody>
                <tr>
                    <td height="56" border="0" align="left" colspan="2" style=" font-size:16px;vertical-align:bottom;">
                        尊敬的***管理员，您好：
                    </td>
                </tr>
                <tr>
                    <td height="30" border="0" align="left" colspan="2">&nbsp;</td>
                </tr>

                        <tr>
                           <td width="40" height="32" border="0" align="left" valign="middle" style=" width:40px; text-align:left;vertical-align:middle; line-height:32px;">
                            <img width="32" height="32" border="0" src="https://img-blog.csdnimg.cn/2022010701240754739.png">
                        </td>
                            <td height="32" style="word-wrap: break-word;word-break: normal;word-break:break-all;">
                                您的以下资源已经恢复, <a href="" rel="noopener" target="_blank">{{ .Labels.alertname }}</a>
                            </td>
                        </tr>
                        <tr>
                            <td height="15" border="0" align="left" colspan="2">&nbsp;</td>
                        </tr>
                            <tr>
                                <td height="32" colspan="2" style="padding-left:40px;border-color: #666666">
                                    <table width="680" border="1" align="center" cellspacing="0" height="48" style="margin-bottom: 10px;margin-left: 0px;">
                                        <thead>
                                        <tr><th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            业务
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                             恢复的资源
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            当前值
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            恢复时间
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            规则描述
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            备注
                                        </th>
                                        <th style="text-align: center;word-wrap: break-word;min-width: 60px;">
                                            Region
                                        </th>
                                        </tr></thead>

                                        <tbody>
                                                                                        <tr>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
        {{ .Annotations.business }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
<a href="" rel="noopener" target="_blank">{{ .Annotations.summary }}</a>
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ .Annotations.values }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05"}}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ .Annotations.description }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
{{ .Annotations.remarks }}
    </td>
    <td style="text-align: center;word-wrap: break-word;word-break: normal;word-break:break-all;min-width: 60px;">
华北3(张家口)
    </td>
                                            </tr>
                                                                                    </tbody>
                                    </table>
                                </td>
                            </tr>

                        <tr>
                            <td height="32" colspan="2" style="padding-left:40px;word-wrap: break-word;word-break: normal;word-break:break-all;">
                                <a href="http://monitor.newtimeai.com">详情请登录智云监控</a>
                            </td>
                        </tr>

                <tr>
                    <td height="30" border="0" align="left" colspan="2">&nbsp;</td>
                </tr>
                <tr>
                    <td height="14" colspan="2">
                        <a href="http://monitor.newtimeai.com/login" rel="noopener" target="_blank">智云监控</a>
                    </td>
                </tr>

                <tr>
                    <td height="14" colspan="2">电话咨询：*****</td>
                </tr>


                <tr>
                    <td height="30" border="0" align="left" colspan="2">&nbsp;</td>
                </tr>
                <tr>
                    <td height="14" colspan="2" style="padding-bottom:16px; border-bottom:1px dashed #e5e5e5;">
                        北京***有限公司
                    </td>
                </tr>
                <tr>
                    <td height="14" colspan="2" style="padding:8px 0 28px;color:#999999; font-size:12px;">
                        此为系统邮件请勿回复
                    </td>
                </tr>
                </tbody>
            </table>

        </td>
    </tr>

    <tr>
        <td>
            <table width="800" height="100" border="0" align="center" cellpadding="0" cellspacing="0">
                <tbody>
                <tr>
                    <td width="800" height="100" align="center" valign="middle">
                    </td>
                </tr>
                </tbody>
            </table>
        </td>
    </tr>
</tbody></table>
<style type="text/css">.qmbox style, .qmbox script, .qmbox head, .qmbox link, .qmbox meta {display: none !important;}</style></div></div>
</body>
</html>
{{- end }}
{{- end }}
{{- end }}

6、prometheus及相关组件启动

#主prometheus
./prometheus --config.file=./prometheus.yml --web.listen-address=0.0.0.0:19092 --web.enable-lifecycle

#blackbox_exporter
/home/yunwei1/software/prometheus/prometheus-2.26.0.linux-amd64/blackbox_exporter-0.18.0.linux-amd64/blackbox_exporter --config.file=/home/yunwei1/software/prometheus/prometheus-2.26.0.linux-amd64/blackbox_exporter-0.18.0.linux-amd64/blackbox.yml

#alertmanager
./alertmanager --config.file=alertmanager.yml

#prometheus配置重载
curl -v -X POST http://localhost:19092/-/reload

三、Grafana安装配置

1、下载Grafana

wget https://dl.grafana.com/oss/release/grafana-7.3.7.linux-amd64.tar.gz

2、修改Grafana配置

vim 

#自定义的端口号
http_port = 13300
#饼状图表
path = /home/yunwei1/software/grafana/grafana-7.3.7/plugin/grafana-piechart-panel

3、启动服务

./bin/grafana-server

4、导入模板、效果展示

导入模板 14750
dashboard 效果如下:

请添加图片描述

报警效果图如下:

at sea .

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
1
评论
Prometheus&Grafana监控报警安装配置

一、软件版本1、prometheus-2.26.0.linux-amd642、blackbox_exporter-0.18.0.linux-amd643、alertmanager-0.22.2.linux-amd644、grafana-7.3.7二、Prometheus安装配置1、解压Prometheustar -xf prometheus-2.26.0.linux-amd64.tar.gz2、主配置文件修改vim prometheus.yml...
复制链接

扫一扫