Prometheus Node Critical 级别告警配置指南

最新推荐文章于 2024-08-22 00:20:06 发布

ivwdcwso

最新推荐文章于 2024-08-22 00:20:06 发布

阅读量253

点赞数 3

文章标签： prometheus k8s 云原生告警监控

本文链接：https://blog.csdn.net/u012172506/article/details/141405570

版权

在使用 Prometheus 监控系统时,合理配置 Critical 级别的告警规则对于及时发现和处理严重问题至关重要。本文将详细介绍一套针对 Node 节点的 Critical 级别告警配置,包括告警规则的解释、配置方法以及最佳实践。

什么是 Critical 级别告警?

Critical 级别告警代表最高优先级的告警,通常用于指示可能导致服务中断或严重影响系统性能的问题。这些告警需要立即引起关注并采取行动。

Node Critical 告警规则

以下是一套针对 Node 节点的 Critical 级别告警规则:

# Critical 级别告警

- alert: NodeDown
  expr: up{job="node-exporter"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Node {{ $labels.instance }} is down"
    description: "Node {{ $labels.instance }} has been down for more than 2 minutes"

- alert: NodeCPUUsageCritical
  expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 95
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Critical CPU usage on {{ $labels.instance }}"
    description: "CPU usage is above 95% on node {{ $labels.instance }} for 5 minutes"

- alert: NodeMemoryUsageCritical
  expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 95
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Critical memory usage on {{ $labels.instance }}"
    description: "Memory usage is above 95% on node {{ $labels.instance }} for 5 minutes"

- alert: NodeDiskSpaceCritical
  expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) < 5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Critical disk space on {{ $labels.instance }}"
    description: "Disk space usage is above 95% on node {{ $labels.instance }} for 5 minutes"

- alert: NodeRAIDDegraded
  expr: node_md_disks_required - ignoring(state) (node_md_disks{state="active"}) > 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "RAID array degraded on {{ $labels.instance }}"
    description: "RAID array {{ $labels.device }} on {{ $labels.instance }} is in degraded state"

- alert: NodeFileDescriptorsCritical
  expr: node_filefd_allocated / node_filefd_maximum * 100 > 95
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "File descriptors critical on {{ $labels.instance }}"
    description: "More than 95% of file descriptors are used on {{ $labels.instance }}"

- alert: NodeInodeCritical
  expr: node_filesystem_files_free{mountpoint ="/"} / node_filesystem_files{mountpoint ="/"} * 100 < 5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Inodes critical on {{ $labels.instance }}"
    description: "Less than 5% inodes free on {{ $labels.instance }}"

- alert: NodeOOMKillerExecuting
  expr: increase(node_vmstat_oom_kill[5m]) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "OOM killer executed on {{ $labels.instance }}"
    description: "OOM killer executed on {{ $labels.instance }}"

- alert: NodeHighCPUIOWait
  expr: avg by(instance) (rate(node_cpu_seconds_total{mode="iowait"}[5m])) * 100 > 50
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High CPU IOWait on {{ $labels.instance }}"
    description: "CPU IOWait is over 50% on {{ $labels.instance }} for 5 minutes"

- alert: NodeNetworkInterfaceDown
  expr: node_network_up == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Network interface down on {{ $labels.instance }}"
    description: "Network interface {{ $labels.device }} is down on {{ $labels.instance }}"