Helm部署Prometheus+Grafana

三省吾身8

已于 2023-11-07 11:32:04 修改

阅读量390

点赞数

分类专栏：服务部署文章标签： grafana prometheus

于 2023-09-26 18:09:08 首次发布

本文链接：https://blog.csdn.net/lishuailing123/article/details/133318352

版权

服务部署专栏收录该内容

11 篇文章 0 订阅

订阅专栏

一、部署Prometheus+Grafana

1、创建命名空间：

kubectl create ns prometheus

2、创建证书的Secret:

kubectl create secret tls sllme-com-pem  --cert=./sllme.com.pem --key=./sllme.com.key -n prometheus

3、部署prometheus

helm pull bitnami/kube-prometheus
修改values.yaml
prometheus:
  persistence:
    enabled: true
    storageClass: "azureblob-fuse-premium"
    size: 100Gi
helm upgrade --install prometheus bitnami/kube-prometheus -f ./kube-prometheus/values.yaml  -n prometheus

4、部署grafana

helm pull bitnami/grafana
修改values.yaml
persistence:
  enabled: true 
  storageClass: "azureblob-fuse-premium"
  annotations: {}
  existingClaim: ""
  accessMode: ReadWriteOnce
  accessModes: []
  size: 100Gi

helm upgrade --install grafana bitnami/grafana -f ./values.yaml  -n prometheus

5、创建grafana-ingress:

grafana-dev.sllme.com.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
  generation: 1
  name: grafana-dev.sllme.com
spec:
  rules:
  - host: grafana-dev.sllme.com
    http:
      paths:
      - backend:
          service:
            name: grafana
            port:
              number: 3000
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - grafana-dev.sllme.com
    secretName: sllme-com-pem
    
kubectl apply -f grafana-dev.sllme.com.yaml -n prometheus

6、登录到grafana:

配置数据源：http://prometheus-kube-prometheus-prometheus:9090 导入grafana模版即可。由于模板太长就不写了，可以参考阿里云的Kubernetes集群监控的模板。

二、配置监控

1、参考以上配置prometheus配置，进行修改alertmanager的内容

#备注以下内容是prometheus的values.yaml文件的内容，进行修改。

alertmanager:
    config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['instance']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'lux'
      routes:
        - match:
            alertname: Watchdog
          receiver: 'lux'
    receivers:
      - name: 'lux'
        webhook_configs:
          - url: 'http://prometheus-alert-center:8080/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=https://open.feishu.cn/open-apis/bot/v2/hook/c0c7e8d8-0bb9-4a72-97b5-247e21bbc50ba'

备注：修改此处的目的是将告警信息都转发到PrometheusAlert

prometheus-alert-center:8080 这个为下面告警组件的service名称

fsurl：为飞书的机器人地址

2、告警软件

我们采用github上开源的工具：PrometheusAlert进行告警 github地址：https://github.com/feiyu563/PrometheusAlert

3、部署服务

vim prometheus-alert.yaml

apiVersion: v1
data:
  app.conf: |
    appname = Lux_Alert
    login_user=lux
    login_password=xxxxxx
    httpaddr = "0.0.0.0"
    httpport = 8080
    runmode = dev
    #开启JSON请求
    copyrequestbody = true
    #告警消息标题
    title=XXX生产
    #转换Prometheus,graylog告警消息的时区为CST时区(如默认已经是CST时区，请勿开启)
    prometheus_cst_time=1
    #数据库驱动，支持sqlite3，mysql,postgres如使用mysql或postgres，请开启db_host,db_port,db_user,db_password,db_name的注释
    db_driver=sqlite3
    #db_host=127.0.0.1
    #db_port=3306
    #db_user=root
    #db_password=root
    #db_name=prometheusalert
    #是否开启告警记录 0为关闭,1为开启
    AlertRecord=0
    #是否开启告警记录定时删除 0为关闭,1为开启
    RecordLive=0
    RecordLiveDay=7
    # 是否将告警记录写入es7，0为关闭，1为开启
    alert_to_es=0
    # es地址，是[]string
    # beego.Appconfig.Strings读取配置为[]string，使用";"而不是","
    # to_es_pwd=password
    # 飞书 webhook告警
    open-feishu=1
    fsurl=https://open.feishu.cn/open-apis/bot/v2/hook/c0c7e8d8-0bb9-4a72-97b5-247e21bbc50ba
kind: ConfigMap
metadata:
  name: prometheus-alert-center-conf
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus-alert-center
    alertname: prometheus-alert-center
  name: prometheus-alert-center
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-alert-center
      alertname: prometheus-alert-center
  template:
    metadata:
      labels:
        app: prometheus-alert-center
        alertname: prometheus-alert-center
    spec:
      containers:
      - image: feiyu563/prometheus-alert
        name: prometheus-alert-center
        env:
        - name: TZ
          value: "Asia/Shanghai"
        ports:
        - containerPort: 8080
          name: http
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: prometheus-alert-center-conf-map
          mountPath: /app/conf/app.conf
          subPath: app.conf
        - name: prometheus-alert-center-conf-map
          mountPath: /app/user.csv
          subPath: user.csv
      volumes:
      - name: prometheus-alert-center-conf-map
        configMap:
          name: prometheus-alert-center-conf
          items:
          - key: app.conf
            path: app.conf
---
apiVersion: v1
kind: Service
metadata:
  labels:
    alertname: prometheus-alert-center
  name: prometheus-alert-center
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8080'  
spec:
  ports:
  - name: http
    port: 8080
    targetPort: http
  selector:
    app: prometheus-alert-center
    
kubectl apply -f prometheus-alert.yaml -n prometheus
##备注：我们采用飞书进行作为接受告警信息端
fsurl为飞书创建一个群，在群上创建一个机器人的地址。

4、创建告警规则

vim k8s-rules.yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus-name: kube-prometheus-prometheus
    managed-by: prometheus-operator          
  name: prometheus-k8s-rules
spec:
  groups:
  - name: node-exporter.rules
    rules:
    - alert: 节点内存过高
      expr: 100 - (node_memory_Buffers_bytes+node_memory_Cached_bytes+node_memory_MemFree_bytes) / node_memory_MemTotal_bytes*100 > 80
      for: 6m
      labels:
        severity: "Error"
      annotations:
        title: "主机内存过高"
        btn: "点击查看详情 :玫瑰:"
        link: "https://www.baidu.com"
        summary: "告警主机：{{ $labels.instance }}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：节点{{ $labels.instance }}内存 > 80%; value = [{{ $value }}]"

    - alert: 节点CPU负载过高  
      expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[4m])) * 100) > 80
      for: 6m
      labels:
        severity: "Error"
      annotations:
        summary: "告警主机： {{ $labels.instance }}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：节点{{ $labels.instance }}CPU > 80%; value[{{ $value }}]"

    - alert: 节点硬盘资源不足
      expr:  (node_filesystem_size_bytes{device="rootfs",mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"}) / node_filesystem_size_bytes{mountpoint="/"} * 100 > 80
      for: 6m
      labels:
        severity:  "Error"
      annotations:
        summary: "告警主机：{{$labels.job}}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：节点{{ $labels.instance }}硬盘(< 20% left); value[{{ $value }}] \n  partition = {{$labels.mountpoint}}"

    - alert: 节点硬盘资源不足
      expr: (node_filesystem_size_bytes{mountpoint="/home"} - node_filesystem_free_bytes{mountpoint="/home"}) / node_filesystem_size_bytes{mountpoint="/home"} * 100 > 80
      for: 6m
      labels:
        severity:  "Error"
      annotations:
        summary: "告警主机：{{$labels.job}}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：节点{{ $labels.instance }}/home硬盘 (< 20% left); value[{{ $value }}] \n  partition = {{$labels.mountpoint}}"

    - alert: node-exporter down  
      expr: up{job="nodes-exporter"} == 0
      for: 30s
      labels:
        team: node
        severity: "Error"
      annotations:
        summary: "告警主机：{{$labels.instance}}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：node-exporter down"

    - alert: 节点硬盘读取率异常
      expr: sum by (job,instance) (irate(node_disk_read_bytes_total[6m])) / 1024 / 1024 > 50
      for: 6m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警主机：{{$labels.job}}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：节点{{ $labels.instance }}磁盘读取率 (> 50 MB/s) VALUE = {{ $value }}"


    - alert: 节点网络吞吐量输入异常
      expr: sum by (job,instance) (irate(node_network_receive_bytes_total[6m])) / 1024 / 1024 > 50
      for: 6m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警主机：{{$labels.job}}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：Host network interfaces are probably receiving too much data (> 50 MB/s) VALUE = {{ $value }}"

    - alert: 节点网络吞吐量输出异常 
      expr: sum by (job,instance) (irate(node_network_transmit_bytes_total[6m])) / 1024 / 1024 > 50
      for: 6m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警主机：{{$labels.job}}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：Host network interfaces are probably sending too much data (> 50 MB/s) VALUE = {{ $value }}"

    - alert: TCP_ESTABLISHED过高  
      expr: node_netstat_Tcp_CurrEstab > 5000
      for: 6m
      labels:
        name: Tcp_Established
        severity: "Warning"
      annotations:
        summary: "告警主机： {{ $labels.instance }}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：TCP_ESTABLISHED大于5000,当前值[{{ $value }}]."

    - alert: TCP_TIME_WAIT大于10000  
      expr: node_sockstat_TCP_tw > 10000
      for: 5m
      labels:
        name: Tcp_Time_Wait
        severity: "Warning"
      annotations:
        summary: "告警主机： {{ $labels.instance }}"
        address: "告警地址：{{ $labels.instance }}"
        description: "告警问题：TCP_TIME_WAIT大于10000,当前值[{{ $value }}]."

      
  - name: k8s.rules
    rules:
    - expr: |
        sum(rate(container_cpu_usage_seconds_total{job="kubelet", image!="", container!="POD"}[4m])) by (namespace)
      record: namespace:container_cpu_usage_seconds_total:sum_rate
    - expr: |
        sum by (namespace, pod, container) (
          rate(container_cpu_usage_seconds_total{job="kubelet", image!="", container!="POD"}[4m])
        ) * on (namespace, pod) group_left(node) max by(namespace, pod, node) (kube_pod_info)
      record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate
    - expr: |
        container_memory_working_set_bytes{job="kubelet", image!=""}
        * on (namespace, pod) group_left(node) max by(namespace, pod, node) (kube_pod_info)
      record: node_namespace_pod_container:container_memory_working_set_bytes
    - expr: |
        container_memory_rss{job="kubelet", image!=""}
        * on (namespace, pod) group_left(node) max by(namespace, pod, node) (kube_pod_info)
      record: node_namespace_pod_container:container_memory_rss
    - expr: |
        container_memory_cache{job="kubelet", image!=""}
        * on (namespace, pod) group_left(node) max by(namespace, pod, node) (kube_pod_info)
      record: node_namespace_pod_container:container_memory_cache
    - expr: |
        container_memory_swap{job="kubelet", image!=""}
        * on (namespace, pod) group_left(node) max by(namespace, pod, node) (kube_pod_info)
      record: node_namespace_pod_container:container_memory_swap
    - expr: |
        sum(container_memory_usage_bytes{job="kubelet", image!="", container!="POD"}) by (namespace)
      record: namespace:container_memory_usage_bytes:sum
    - expr: |
        sum by (namespace) (
            sum by (namespace, pod) (
                max by (namespace, pod, container) (
                    kube_pod_container_resource_requests_memory_bytes{job="prometheus-kube-state-metrics"}
                ) * on(namespace, pod) group_left() max by (namespace, pod) (
                    kube_pod_status_phase{phase=~"Pending|Running"} == 1
                )
            )
        )
      record: namespace:kube_pod_container_resource_requests_memory_bytes:sum
    - expr: |
        sum by (namespace) (
            sum by (namespace, pod) (
                max by (namespace, pod, container) (
                    kube_pod_container_resource_requests_cpu_cores{job="prometheus-kube-state-metrics"}
                ) * on(namespace, pod) group_left() max by (namespace, pod) (
                  kube_pod_status_phase{phase=~"Pending|Running"} == 1
                )
            )
        )
      record: namespace:kube_pod_container_resource_requests_cpu_cores:sum
    - expr: |
        sum(
          label_replace(
            label_replace(
              kube_pod_owner{job="prometheus-kube-state-metrics", owner_kind="ReplicaSet"},
              "replicaset", "$1", "owner_name", "(.*)"
            ) * on(replicaset, namespace) group_left(owner_name) kube_replicaset_owner{job="prometheus-kube-state-metrics"},
            "workload", "$1", "owner_name", "(.*)"
          )
        ) by (namespace, workload, pod)
      labels:
        workload_type: deployment
      record: mixin_pod_workload
    - expr: |
        sum(
          label_replace(
            kube_pod_owner{job="prometheus-kube-state-metrics", owner_kind="DaemonSet"},
            "workload", "$1", "owner_name", "(.*)"
          )
        ) by (namespace, workload, pod)
      labels:
        workload_type: daemonset
      record: mixin_pod_workload
    - expr: |
        sum(
          label_replace(
            kube_pod_owner{job="prometheus-kube-state-metrics", owner_kind="StatefulSet"},
            "workload", "$1", "owner_name", "(.*)"
          )
        ) by (namespace, workload, pod)
      labels:
        workload_type: statefulset
      record: mixin_pod_workload

  - name: node.rules
    rules:
    - expr: sum(min(kube_pod_info) by (node))
      record: ':kube_pod_info_node_count:'
    - expr: |
        max(label_replace(kube_pod_info{job="prometheus-kube-state-metrics"}, "pod", "$1", "pod", "(.*)")) by (node, namespace, pod)
      record: 'node_namespace_pod:kube_pod_info:'
    - expr: |
        count by (node) (sum by (node, cpu) (
          node_cpu_seconds_total{job="node-exporter"}
        * on (namespace, pod) group_left(node)
          node_namespace_pod:kube_pod_info:
        ))
      record: node:node_num_cpu:sum
    - expr: |
        sum(
          node_memory_MemAvailable_bytes{job="node-exporter"} or
          (
            node_memory_Buffers_bytes{job="node-exporter"} +
            node_memory_Cached_bytes{job="node-exporter"} +
            node_memory_MemFree_bytes{job="node-exporter"} +
            node_memory_Slab_bytes{job="node-exporter"}
          )
        )
      record: :node_memory_MemAvailable_bytes:sum

  - name: kube-prometheus-node-recording.rules
    rules:
    - expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait"}[3m])) BY
        (instance)
      record: instance:node_cpu:rate:sum
    - expr: sum((node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"}))
        BY (instance)
      record: instance:node_filesystem_usage:sum
    - expr: sum(rate(node_network_receive_bytes_total[3m])) BY (instance)
      record: instance:node_network_receive_bytes:rate:sum
    - expr: sum(rate(node_network_transmit_bytes_total[3m])) BY (instance)
      record: instance:node_network_transmit_bytes:rate:sum
    - expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait"}[4m])) WITHOUT
        (cpu, mode) / ON(instance) GROUP_LEFT() count(sum(node_cpu_seconds_total)
        BY (instance, cpu)) BY (instance)
      record: instance:node_cpu:ratio
    - expr: sum(rate(node_cpu_seconds_total{mode!="idle",mode!="iowait"}[4m]))
      record: cluster:node_cpu:sum_rate4m
    - expr: cluster:node_cpu_seconds_total:rate4m / count(sum(node_cpu_seconds_total)
        BY (instance, cpu))
      record: cluster:node_cpu:ratio

  - name: kubernetes-apps
    rules:
    - alert: KubePodCrashLooping 
      expr: |
        rate(kube_pod_container_status_restarts_total{job="prometheus-kube-state-metrics",container!="fluentd",namespace!="efk"}[4m]) * 60 * 5 > 0
      for: 2m
      labels:
        severity: "Error"
      annotations:
        summary: "告警事件：Pod，{{ $labels.pod }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: '告警问题：Pod Crash Looping ({{ $labels.container }}) is restarting {{ printf "%.2f" $value }} times / 5 minutes'

    - alert: KubePodNotReady 
      expr: |
        sum by (namespace, pod) (max by(namespace, pod) (kube_pod_status_phase{job="prometheus-kube-state-metrics", phase=~"Pending|Unknown"}) * on(namespace, pod) group_left(owner_kind) max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})) > 0
      for: 2m
      labels:
        severity: "Error"
      annotations: 
        summary: "告警事件：Pod，{{ $labels.pod }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Pod {{ $labels.namespace }}/{{ $labels.pod }} has been in a non-ready state for longer than 2 minutes"

    - alert: KubeDeploymentGenerationMismatch 
      expr: |
        kube_deployment_status_observed_generation{job="prometheus-kube-state-metrics"}
          !=
        kube_deployment_metadata_generation{job="prometheus-kube-state-metrics"}
      for: 5m
      labels:
        severity: "Error"
      annotations:
        summary: "告警事件：deployment，{{ $labels.deployment }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Deployment generation for {{ $labels.namespace }}/{{ $labels.deployment
          }} does not match, this indicates that the Deployment has failed but has not been rolled back"

    - alert: KubeDeploymentReplicasMismatch 
      expr: |
        kube_deployment_spec_replicas{job="prometheus-kube-state-metrics"}
          !=
        kube_deployment_status_replicas_available{job="prometheus-kube-state-metrics"}
      for: 30m
      labels:
        severity: "Error"
      annotations:
        summary: "告警事件：deployment，{{ $labels.deployment }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Deployment {{ $labels.namespace }}/{{ $labels.deployment }} has not
          matched the expected number of replicas for longer than 30 minutes."

    - alert: KubeStatefulSetReplicasMismatch 
      expr: |
        kube_statefulset_status_replicas_ready{job="prometheus-kube-state-metrics"}
          !=
        kube_statefulset_status_replicas{job="prometheus-kube-state-metrics"}
      for: 5m
      labels:
        severity: "Error"
      annotations:
        summary: "告警事件：StatefulSet；{{ $labels.statefulset }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} has
          not matched the expected number of replicas for longer than 15 minutes"

    - alert: KubeStatefulSetReplicasMismatch 
      expr: |
        kube_statefulset_status_observed_generation{job="prometheus-kube-state-metrics"}
          !=
        kube_statefulset_metadata_generation{job="prometheus-kube-state-metrics"}
      for: 5m
      labels:
        severity: "Error"
      annotations:
        summary: "告警事件：{{ $labels.statefulset }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：StatefulSet generation for {{ $labels.namespace }}/{{ $labels.statefulset
          }} does not match, this indicates that the StatefulSet has failed but has
          not been rolled back"

    - alert: KubeStatefulSetUpdateNotRolledOut 
      expr: |
        max without (revision) (
          kube_statefulset_status_current_revision{job="prometheus-kube-state-metrics"}
            unless
          kube_statefulset_status_update_revision{job="prometheus-kube-state-metrics"}
        )
          *
        (
          kube_statefulset_replicas{job="prometheus-kube-state-metrics"}
            !=
          kube_statefulset_status_replicas_updated{job="prometheus-kube-state-metrics"}
        )
      for: 5m
      labels:
        severity: "Critical"
      annotations:
        summary: "告警事件：StatefulSet，{{ $labels.statefulset }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：StatefulSet {{ $labels.namespace }}/{{ $labels.statefulset }} update
          has not been rolled out."

    - alert: KubeDaemonSetRolloutStuck 
      expr: |
        kube_daemonset_status_number_ready{job="prometheus-kube-state-metrics",daemonset!="fluentd"}
          /
        kube_daemonset_status_desired_number_scheduled{job="prometheus-kube-state-metrics"} < 1.00
      for: 15m
      labels:
        severity: "Critical"
      annotations:
        summary: "告警事件：daemonset，{{ $labels.daemonset }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Only {{ $value | humanizePercentage }} of the desired Pods of DaemonSet
          {{ $labels.namespace }}/{{ $labels.daemonset }} are scheduled and ready."

    - alert:  KubeContainerWaiting 
      expr: |
        sum by (namespace, pod, container) (kube_pod_container_status_waiting_reason{job="prometheus-kube-state-metrics"}) > 0
      for: 1h
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：Pod，{{ $labels.pod }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Pod {{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container}}
          has been in waiting state for longer than 1 hour."

    - alert: KubeDaemonSetNotScheduled 
      expr: |
        kube_daemonset_status_desired_number_scheduled{job="prometheus-kube-state-metrics"}
          -
        kube_daemonset_status_current_number_scheduled{job="prometheus-kube-state-metrics"} > 0
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：daemonset， {{ $labels.daemonset }}'"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：'{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset
          }} are not scheduled.'"

    - alert: KubeDaemonSetMisScheduled 
      expr: |
        kube_daemonset_status_number_misscheduled{job="prometheus-kube-state-metrics"} > 0
      for: 5m
      labels:
        severity: "Warning"
      annotations: 
        summary: "告警事件：DaemonSet，{{ $labels.daemonset }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：'{{ $value }} Pods of DaemonSet {{ $labels.namespace }}/{{ $labels.daemonset
          }} are running where they are not supposed to run.'"

    - alert: KubeCronJobRunning 
      expr: |
        time() - kube_cronjob_next_schedule_time{job="prometheus-kube-state-metrics"} > 3600
      for: 20m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：CronJob，{{ $labels.cronjob }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：CronJob {{ $labels.namespace }}/{{ $labels.cronjob }} is taking more
        than 20m to complete."

    - alert: KubeJobCompletion 
      expr: |
        kube_job_spec_completions{job="prometheus-kube-state-metrics"} - kube_job_status_succeeded{job="prometheus-kube-state-metrics"}  > 0
      for: 1h
      labels:
        severity: "Warning"
      annotations:
        title: "Job任务未完成"
        btn: "点击查看详情 :玫瑰:"
        link: "https://www.baidu.com"
        summary: "告警事件： Job，{{ $labels.job_name }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题： Job {{ $labels.namespace }}/{{ $labels.job_name }} is taking more
          than one hour to complete."

    - alert: KubeJobFailed 
      expr: |
        kube_job_failed{job="prometheus-kube-state-metrics"}  > 0
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        title: "Job失败"
        btn: "点击查看详情 :玫瑰:"
        link: "https://www.baidu.com"
        summary: "告警事件：Job，{{ $labels.job_name }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete."

    - alert: KubeHpaReplicasMismatch 
      expr: |
        (kube_hpa_status_desired_replicas{job="prometheus-kube-state-metrics"}
          !=
        kube_hpa_status_current_replicas{job="prometheus-kube-state-metrics"})
          and
        changes(kube_hpa_status_current_replicas[4m]) == 0
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：HPA，{{ $labels.hpa }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：HPA {{ $labels.namespace }}/{{ $labels.hpa }} has not matched the
          desired number of replicas for longer than 15 minutes."

    - alert: KubeHpaMaxedOut 
      expr: |
        kube_hpa_status_current_replicas{job="prometheus-kube-state-metrics"}
          ==
        kube_hpa_spec_max_replicas{job="prometheus-kube-state-metrics"}
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：HPA，{{ $labels.hpa }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：HPA {{ $labels.namespace }}/{{ $labels.hpa }} has been running at
          max replicas for longer than 15 minutes."

  - name: kubernetes-resources 
    rules:
    - alert: KubeCPUOvercommit
      expr: |
        sum(namespace:kube_pod_container_resource_requests_cpu_cores:sum)
          /
        sum(kube_node_status_allocatable_cpu_cores)
          >
        (count(kube_node_status_allocatable_cpu_cores)-1) / count(kube_node_status_allocatable_cpu_cores)
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件："
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Cluster has overcommitted CPU resource requests for Pods and cannot
          tolerate node failure."

    - alert: KubeMemOvercommit 
      expr: |
        sum(namespace:kube_pod_container_resource_requests_memory_bytes:sum)
          /
        sum(kube_node_status_allocatable_memory_bytes)
          >
        (count(kube_node_status_allocatable_memory_bytes)-1)
          /
        count(kube_node_status_allocatable_memory_bytes)
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary:  "告警事件：pod"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Cluster has overcommitted memory resource requests for Pods and cannot
          tolerate node failure."

    - alert:  KubeCPUOvercommit 
      expr: |
        sum(kube_resourcequota{job="prometheus-kube-state-metrics", type="hard", resource="cpu"})
          /
        sum(kube_node_status_allocatable_cpu_cores)
          > 1.5
      for: 5m
      labels:
        severity:  "Warning"
      annotations:
        summary: "告警事件：namespaces"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题： Cluster has overcommitted CPU resource requests for Namespaces"

    - alert: KubeMemOvercommit 
      expr: |
        sum(kube_resourcequota{job="prometheus-kube-state-metrics", type="hard", resource="memory"})
          /
        sum(kube_node_status_allocatable_memory_bytes{job="node-exporter"})
          > 1.5
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：namespaces"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Cluster has overcommitted memory resource requests for Namespaces"

    - alert: KubeQuotaExceeded 
      expr: |
        kube_resourcequota{job="prometheus-kube-state-metrics", type="used"}
          / ignoring(instance, job, type)
        (kube_resourcequota{job="prometheus-kube-state-metrics", type="hard"} > 0)
          > 0.90
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件： {{ $labels.namespace }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Namespace {{ $labels.namespace }} is using {{ $value | humanizePercentage }} of its {{ $labels.resource }} quota"

    - alert: CPUThrottlingHigh 
      expr: |
        sum(increase(container_cpu_cfs_throttled_periods_total{container!="",namespace!="efk" }[4m])) by (container, pod, namespace)
          /
        sum(increase(container_cpu_cfs_periods_total{}[4m])) by (container, pod, namespace)
          > ( 80 / 100 )
      for: 5m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：{{ $labels.pod }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：{{ $value | humanizePercentage }} throttling of CPU in namespace {{ $labels.namespace }} for container {{ $labels.container }} in pod {{$labels.pod }}"

  - name: kubernetes-storage
    rules:
    - alert: KubePersistentVolumeUsageCritical 
      expr: |
        kubelet_volume_stats_available_bytes{job="kubelet"}
          /
        kubelet_volume_stats_capacity_bytes{job="kubelet"}
          < 0.03
      for: 2m
      labels:
        severity: "Warning"
      annotations:
        summary: "告警事件：持久卷"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题： The PersistentVolume claimed by {{ $labels.persistentvolumeclaim
          }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage
          }} free."
      
  - name: kubernetes-system
    rules:
    - alert: KubeVersionMismatch 
      annotations:
        summary: "告警事件：{{ $value }}"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：There are {{ $value }} different semantic versions of Kubernetes
          components running."
      expr: |
        count(count by (gitVersion) (label_replace(kubernetes_build_info{job!~"kube-dns|coredns"},"gitVersion","$1","gitVersion","(v[0-9]*.[0-9]*.[0-9]*).*"))) > 1
      for: 5m
      labels:
        severity: "Warning"

    - alert: KubeClientErrors 
      annotations:
        summary: "告警事件：Client error"
        address: "告警主机：{{ $labels.job }}"
        description: "告警问题：Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance
          }}' is experiencing {{ $value | humanizePercentage }} errors.'"
      expr: |
        (sum(rate(rest_client_requests_total{code=~"5.."}[4m])) by (instance, job)
          /
        sum(rate(rest_client_requests_total[4m])) by (instance, job))
        > 0.01
      for: 5m
      labels:
        severity: "Error"

  - name: kubernetes-system-kubelet 
    rules:
    - alert: KubeNodeNotReady
      annotations:
        summary: "告警事件：NodeNotReady"
        address: "告警节点：{{ $labels.node }}"
        description: "告警问题：'{{ $labels.node }} has been unready for more than 2 minutes.'"
      expr: |
        kube_node_status_condition{job="prometheus-kube-state-metrics",condition="Ready",status="true"} == 0
      for: 2m
      labels:
        severity: "Warning"
        
    - alert: KubeNodeUnreachable 
      annotations:
        summary: "告警事件：KubeNodeUnreachable"
        address: "告警主机：{{ $labels.node }}"
        description: "告警问题： '{{ $labels.node }} is unreachable and some workloads may be rescheduled.'"
      expr: |
        kube_node_spec_taint{job="prometheus-kube-state-metrics",key="node.kubernetes.io/unreachable",effect="NoSchedule"} == 1
      for: 2m
      labels:
        severity: "Warning"

    - alert: KubeletTooManyPods 
      annotations:
        summary: "告警事件：TooManyPods"
        address: "告警主机：{{ $labels.node }}"
        description: "告警问题：Kubelet is running at {{ $value | humanizePercentage
          }} of its Pod capacity."
      expr: |
        max(max(kubelet_running_pod_count{job="kubelet"}) by(instance) * on(instance) group_left(node) kubelet_node_name{job="kubelet"}) by(node) / max(kube_node_status_capacity_pods{job="prometheus-kube-state-metrics"}) by(node) > 0.95
      for: 5m
      labels:
        severity: "Warning"

    - alert: KubeletDown 
      annotations:
        summary: "告警事件：KubeletDown"
        address: "命名空间：{{ $labels.namespace }}"
        description: "告警问题：Kubelet has disappeared from Prometheus target discovery."
      expr: |
        absent(up{job="kubelet"} == 1)
      for: 5m
      labels:
        severity: "Warning"
        

kubectl apply -f k8s-rules.yaml -n prometheus

5、配置告警信息格式

登录PrometheusAlert

kubectl port-forward svc/prometheus-alert-center 8080:8080 -n prometheus 访问：http://127.0.0.1:8080

配置告警模版

模版内容如下：

{{ range $k,$v:=.alerts }} 状态：**{{$v.status}}** {{if eq $v.status "resolved"}}（恢复）
告警名称：[{{$v.labels.alertname}}]
告警级别：{{$v.labels.severity}}
{{$v.annotations.address}}
{{$v.annotations.summary}}
**{{$v.annotations.description}}**
⏱ : {{GetCSTtime  $v.startsAt}}
⏲ : {{GetCSTtime  $v.endsAt}} 
{{else}}（报警）
告警名称：[{{$v.labels.alertname}}]
告警级别：{{$v.labels.severity}}
{{$v.annotations.address}}
{{$v.annotations.summary}}
**{{$v.annotations.description}}**
⏱ : {{GetCSTtime  $v.startsAt}}
{{end}}
{{end}}

看一下告警的效果

【完成】

三省吾身8

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Helm部署Prometheus+Grafana

配置数据源：http://prometheus-kube-prometheus-prometheus:9090 导入grafana模版即可。由于模板太长就不写了，可以参考阿里云的Kubernetes集群监控的模板。prometheus-alert-center:8080 这个为下面告警组件的service名称。#备注以下内容是prometheus的values.yaml文件的内容，进行修改。备注：修改此处的目的是将告警信息都转发到PrometheusAlert。fsurl：为飞书的机器人地址。
复制链接

扫一扫