玩转rancher 2.6之 monitor监控

目录

前言

1. 安装monitor

2. 安装prometheus-webhook-dingtalk

2.1 配置钉钉告警配置文件

 2.2 创建钉钉告警模板

2.3 创建dingtalk configmap配置文件

2.4 安装dingtalk

2.5 dingtalk调用方法

3. 配置告警

3.1 配置告警接收者

3.2 配置路由默认告警接收者

4. 测试

---

前言

参考RancherLabs 文章： ( Rancher2.6全新Monitoring快速入门_RancherLabs的博客-CSDN博客

对rancher monitor使用过程中的配置讲解补充，补充了alertmanger发送钉钉告警的实现过程，通过安装dingtalk插件完成钉钉机器人告警推送，详细如下

1. 安装monitor

集群-集群工具Monitoring

根据实际需求修改部署要求：如持久化，数据保存时长等

 

安装完毕后可在： 集群-监控，查看仪表盘

分别可查看alertmanger、grafana和prometheus web界面

 

2. 安装prometheus-webhook-dingtalk

2.1 配置钉钉告警配置文件

注意：先创建钉钉机器人，安全验证配置secret，用于下面的配置文件中

创建dingtalk config.yml

# url和secret改为自己的机器人webhook地址和secret
templates:
  - /etc/prometheus-webhook-dingtalk/dingding.tmpl
targets:
  k8s:
    url: https://oapi.dingtalk.com/robot/send?access_token="你的钉钉webhook"
    secret: "你的钉钉webhook secret"
    message:
      title: '{{ template "ops.title" . }}'
      text: '{{ template "ops.content" . }}'

 2.2 创建钉钉告警模板

dingding.tmpl

{{ define "__subject" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]
{{ end }}
{{ define "__alert_list" }}{{ range . }}
---
    [告警程序]: gz-test-k8s 
    [告警级别]: {{ index .Labels.severity }} 
    [告警类型]: {{ index .Labels.alertname }} 
    [故障实例]: {{ index .Labels.instance }} 
    [告警主题]: {{ index .Annotations.summary }}
    [告警详情]: {{ index .Annotations.description }}
    [触发时间]: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}  
{{ end }}{{ end }}
{{ define "__resolved_list" }}{{ range . }}
---
    [告警程序]: gz-test-k8s
    [告警级别]: {{ index .Labels.severity }} 
    [告警类型]: {{ index .Labels.alertname }} 
    [故障实例]: {{ index .Labels.instance }} 
    [告警详情]: {{ index .Annotations.description }}
    [状态]: 恢复正常
    [触发时间]: {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}  
    [恢复时间]: {{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{ end }}{{ end }}
{{ define "ops.title" }}
{{ template "__subject" . }}
{{ end }}
{{ define "ops.content" }}
{{ if gt (len .Alerts.Firing) 0 }}
**侦测到{{ .Alerts.Firing | len  }}个故障**
{{ template "__alert_list" .Alerts.Firing }}
---
{{ end }}
{{ if gt (len .Alerts.Resolved) 0 }}
**恢复{{ .Alerts.Resolved | len  }}个故障**
{{ template "__resolved_list" .Alerts.Resolved }}
{{ end }}
{{ end }}
{{ define "ops.link.title" }}{{ template "ops.title" . }}{{ end }}
{{ define "ops.link.content" }}{{ template "ops.content" . }}{{ end }}
{{ template "ops.title" . }}
{{ template "ops.content" . }}

2.3 创建dingtalk configmap配置文件

将以上的配置文件保存为文件，通过kubectl创建configmap

$  kubectl create configmap dingtalk-cm --from-file=config.yml=config.yml --from-file=dingding.tmpl=dingding.tmpl -n cattle-monitoring-system
$  kubectl get cm dingtalk-cm  -n cattle-monitoring-system
NAME          DATA   AGE
dingtalk-cm   2      87m 

2.4 安装dingtalk

部署文件 dingtalk.yaml

apiVersion: v1
kind: Service
metadata:
  name: dingtalk
  namespace: cattle-monitoring-system
spec:
  selector:
    app: dingtalk
  ports:
    - name: http
      protocol: TCP
      port: 8060
      targetPort: 8060
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dingtalk
  namespace: cattle-monitoring-system
  labels:
    app: dingtalk
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  selector:
    matchLabels:
      app: dingtalk
  template:
    metadata:
      labels:
        app: dingtalk
    spec:
      restartPolicy: "Always"
      containers:
      - name: dingtalk
        image: timonwong/prometheus-webhook-dingtalk
        imagePullPolicy: "IfNotPresent"
        volumeMounts:
          - name: dingtalk-conf
            mountPath: /etc/prometheus-webhook-dingtalk/
        resources:
          limits:
            cpu: "400m"
            memory: "500Mi"
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
        - containerPort: 8060
          name: http
          protocol: TCP 
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 5
          initialDelaySeconds: 30
          successThreshold: 1
          tcpSocket:
            port: 8060
        livenessProbe:
          tcpSocket:
            port: 8060
          initialDelaySeconds: 30
          periodSeconds: 10
      volumes:
        - name: dingtalk-conf
          configMap:
            name: dingtalk-cm

部署到命名空间下：cattle-monitoring-system

> kubectl get deployments.apps dingtalk -n cattle-monitoring-system 
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
dingtalk   1/1     1            1           68m
> kubectl get svc dingtalk -n cattle-monitoring-system 
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
dingtalk   ClusterIP   10.43.245.38   <none>        8060/TCP   69m

2.5 dingtalk调用方法

# 和alertmanager部署在同个命名空间则可以使用以下的方式调用
# http://dingtalk:8060/dingtalk/"你的钉钉webhook名字"/send
# 如配置了k8s的接收者，地址如下
http://dingtalk:8060/dingtalk/k8s/send

3. 配置告警

3.1 配置告警接收者

集群-监控-Alerting-Routes and Receivers-接收者-创建

 

创建一个默认接收者，同时接收邮件和钉钉告警

 

3.2 配置路由默认告警接收者

集群-监控-Alerting-Routes and Receivers-路由

root匹配全部告警规则，不可删除

 

 

4. 测试

rancher monitor安装后，默认配置了针对组件、节点、pod的告警策略，可在 集群-监控-Advanced-PrometheusRule 查看，修改或新增，具体按实际需要

在本配置了接收者和路由后，可以看到当触发告警时，收到对应的邮件通知

 

钉钉告警

 

觉得有用点个关注吧~欢迎一起讨论