Kubernetes事件采集、告警（ARM64环境）

说明

k8s的事件默认只保存1小时，而事件对于排查集群故障有很大作用，因此很多时候可能需要将k8s事件保存更长时间来进行问题分析。kubernetes-event-exporter这个工具可以用来导出k8s事件以用于观察分析或者告警。

目前

1. 通过kubernetes-event-export将k8s 事件收集到elasticsearch，然后通过kibana进行查看分析
2. 通过kubernetes-event-export筛选Warning事件，并将事件以webhook形式告警通知到蓝信群中

配置

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-exporter
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: event-exporter
        version: v1
    spec:
      serviceAccountName: event-exporter
      containers:
        - name: event-exporter
          image: toyangdon/kubernetes-event-exporter:20220526
          imagePullPolicy: IfNotPresent
          args:
            - -conf=/data/config.yaml
          volumeMounts:
            - mountPath: /data
              name: cfg
      volumes:
        - name: cfg
          configMap:
            name: event-exporter-cfg
  selector:
    matchLabels:
      app: event-exporter
      version: v1

role.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: kube-system
  name: event-exporter
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: event-exporter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    namespace: kube-system
    name: event-exporter

config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: kube-system
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
        - match:
            - receiver: "dump"
        - match:
            - receiver: "alert"
          drop:
            - type: "Normal"
    receivers:
      - name: "dump"
        elasticsearch:
          hosts:
            - "http://elasticsearch:9200"
          indexFormat: "k8s-{2006-01-02}"
      - name: "alert"
        webhook:
          endpoint: "https://apigw-cec.cec.com.cn/v1/bot/hook/messages/create?hook_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
          headers:
            Content-Type: "application/json"
          layout:
            msgType: "text"
            msgData:
              text:
                content: 'k8s事件告警  message: "{{ .Message }}" reason: "{{ .Reason }}" type: "{{ .Type }}" count: "{{ .Count }}" kind: "{{ .InvolvedObject.Kind }}" name: "{{ .InvolvedObject.Name }}" namespace: "{{ .Namespace }}" component: "{{ .Source.Component }}"  host: "{{ .Source.Host }}" labels: "{{ toJson .InvolvedObject.Labels}}" lastTimestamp: "{{ .LastTimestamp }}"'

效果

kibana上查询事件

蓝信群中Warning事件告警