k8s笔记23--使用kubernetes-event-exporter采集集群events日志

1 介绍

kubernetes-event-exporter 是一个用于采集k8s事件的工具，它允许我们将经常遗漏的 Kubernetes 事件导出到第三方平台或者数据库，以便用于可观察性或警报目的。
event-exporter 可以将k8s事件存储到 Opsgenie、Webhooks、kafka、es等十几种平台|数据库，本文基于该工具介绍两种最常见的采集方式，方法一将事件日志采集到 kafka，方法二将事件日志直接采集到 es集群。

2 部署测试

2.1 写入kafka

部署脚本：

apiVersion: v1
kind: Namespace
metadata:
  name: lens-metrics
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: lens-metrics
  name: event-exporter
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: event-exporter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    namespace: lens-metrics
    name: event-exporter
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: lens-metrics
data:
  config.yaml: |该工具允许将经常遗漏的Kubernetes事件导出到各种输出，以便用于可观察性或警报目的。你不会相信你错过了什么。
    logLevel: error
    logFormat: json
    route:
      routes:
      - match:
        - receiver: "kafka"
    receivers:
      - name: "kafka"
        kafka:
          clientId: "kubernetes"
          topic: "k8s-event-log"
          brokers:
          - "192.168.2.11:9092"
          compressionCodec: "gzip"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-exporter
  namespace: lens-metrics
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: event-exporter
        version: v1
    spec:
      serviceAccountName: event-exporter
      containers:
      - name: event-exporter
        image: opsgenie/kubernetes-event-exporter:0.9
        imagePullPolicy: IfNotPresent
        args:
        - -conf=/data/config.yaml
        volumeMounts:
        - mountPath: /data
          name: cfg
      volumes:
        - name: cfg
          configMap:
            name: event-exporter-cfg
  selector:
    matchLabels:
      app: event-exporter
      version: v1

采集数据：
服务部署后，很快就采集到kafka，如下所示：

offset 4077的具体数据如下所示：

{
	"metadata": {
		"name": "calico-node-4p4jb.171cf6be0f3d946d",
		"namespace": "kube-system",
		"uid": "53d89898-44f3-471b-a3fc-9362567ac846",
		"resourceVersion": "4225520",
		"creationTimestamp": "2022-10-11T08:34:18Z",
		"managedFields": [
			{
				"manager": "kubelet",
				"operation": "Update",
				"apiVersion": "v1",
				"time": "2022-10-11T08:34:18Z"
			}
		]
	},
	"reason": "Unhealthy",
	"message": "Readiness probe failed: 2022-10-11 08:34:18.377 [INFO][503] confd/health.go 180: Number of node(s) with BGP peering established = 0\ncalico/node is not ready: BIRD is not ready: BGP not established with 192.168.2.12\n",
	"source": {
		"component": "kubelet",
		"host": "kmaster"
	},
	"firstTimestamp": "2022-10-11T08:34:18Z",
	"lastTimestamp": "2022-10-11T08:34:18Z",
	"count": 1,
	"type": "Warning",
	"eventTime": null,
	"reportingComponent": "",
	"reportingInstance": "",
	"involvedObject": {
		"kind": "Pod",
		"namespace": "kube-system",
		"name": "calico-node-4p4jb",
		"uid": "069010fe-ca85-4ede-a4c5-033f02975433",
		"apiVersion": "v1",
		"resourceVersion": "4223433",
		"fieldPath": "spec.containers{calico-node}",
		"labels": {
			"controller-revision-hash": "84f97c77db",
			"k8s-app": "calico-node",
			"pod-template-generation": "2"
		}
	}
}

2.2 写入es

部署脚本：

apiVersion: v1
kind: Namespace
metadata:
  name: lens-metrics
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: lens-metrics
  name: event-exporter
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: event-exporter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    namespace: lens-metrics
    name: event-exporter
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-cfg
  namespace: lens-metrics
data:
  config.yaml: |
    logLevel: error
    logFormat: json
    route:
      routes:
      - match:
        - receiver: "es"
    receivers:
      - name: "es"
        elasticsearch:
          hosts:
            - http://your-es-ip:9200
          index: kube-events
          indexFormat: "kube-events-{2006-01-02}"
          username: your-user
          password: your-pwd
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-exporter
  namespace: lens-metrics
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: event-exporter
        version: v1
    spec:
      serviceAccountName: event-exporter
      containers:
      - name: event-exporter
        image: opsgenie/kubernetes-event-exporter:0.9
        imagePullPolicy: IfNotPresent
        args:
        - -conf=/data/config.yaml
        volumeMounts:
        - mountPath: /data
          name: cfg
      volumes:
        - name: cfg
          configMap:
            name: event-exporter-cfg
  selector:
    matchLabels:
      app: event-exporter
      version: v1

采集数据：
服务部署后，很快就采集到es，如下所示：

其中一条数据如下：

{
    "metadata":{
        "name":"event-exporter-f5b88ddb-vqrwc.171f1c2cb500b852",
        "namespace":"lens-metrics",
        "uid":"374afa8d-70dd-43f5-bd04-757ad3bdf5ec",
        "resourceVersion":"4501276",
        "creationTimestamp":"2022-10-18T08:22:45Z",
        "managedFields":[
            {
                "manager":"kubelet",
                "operation":"Update",
                "apiVersion":"v1",
                "time":"2022-10-18T08:22:45Z"
            }
        ]
    },
    "reason":"Created",
    "message":"Created container event-exporter",
    "source":{
        "component":"kubelet",
        "host":"kmaster"
    },
    "firstTimestamp":"2022-10-18T08:22:45Z",
    "lastTimestamp":"2022-10-18T08:22:45Z",
    "count":1,
    "type":"Normal",
    "eventTime":null,
    "reportingComponent":"",
    "reportingInstance":"",
    "involvedObject":{
        "kind":"Pod",
        "namespace":"lens-metrics",
        "name":"event-exporter-f5b88ddb-vqrwc",
        "uid":"41fb89bf-8f54-4810-af81-68893f6ed02a",
        "apiVersion":"v1",
        "resourceVersion":"4501249",
        "fieldPath":"spec.containers{event-exporter}",
        "labels":{
            "app":"event-exporter",
            "pod-template-hash":"f5b88ddb",
            "version":"v1"
        },
        "annotations":{
            "cni.projectcalico.org/containerID":"2199b2c0056cfb84979a0d94429b0c209877c5593b65fb0e2b890419d228e7c7",
            "cni.projectcalico.org/podIP":"10.224.189.42/32",
            "cni.projectcalico.org/podIPs":"10.224.189.42/32"
        }
    }
}

3 注意事项

1. 本文第一种方法将事件日志写入到 kafka， 然后再根据需要消费kafka数据，并存放到用户自己的系统中。

4 说明

resmoio/kubernetes-event-exporter
kubernetes-event-exporter