Kubernetes 审计日志采集与分析最佳实践

观测云

于 2024-05-17 10:17:57 发布

阅读量1.1k

点赞数 20

文章标签： kubernetes 审计日志

本文链接：https://blog.csdn.net/DataFlux/article/details/138997945

版权

Kubernetes 审计日志概述

Kubernetes 在 1.7 版本中发布了审计（Audit）日志功能，审计（Audit）提供了安全相关的时序操作记录（包括时间、来源、操作结果、发起操作的用户、操作的资源以及请求/响应的详细信息等），通过审计日志，我们能够非常清晰的知道 K8S 集群到底发生了什么事情，包括但不限于：

当前/历史上集群发生了哪些变更事件。
这些变更操作者是谁，是系统组件还是用户，是哪个系统组件/用户。
重要变更事件的详细内容是什么，比如修改了 POD 中的哪个参数。
事件的结果是什么，成功还是失败。
操作用户来自哪里，集群内还是集群外。

审计日志格式

Kubernetes 审计日志是以 json 格式的方式进行输出的，如下所示：

{
        "kind": "Event",
        "apiVersion": "audit.k8s.io/v1",
        "level": "Metadata",
        "auditID": "432064d5-bda6-4957-a9e8-369f364e5748",
        "stage": "ResponseComplete",
        "requestURI": "/api/v1/namespaces/tigera-operator/configmaps/operator-lock",
        "verb": "update",
        "user": {
                "username": "system:serviceaccount:tigera-operator:tigera-operator",
                "uid": "8ab017b1-c57d-4aa4-a52f-5acec0c1e72e",
                "groups": ["system:serviceaccounts", "system:serviceaccounts:tigera-operator", "system:authenticated"],
                "extra": {
                        "authentication.kubernetes.io/pod-name": ["tigera-operator-d7957f5cc-hv5p8"],
                        "authentication.kubernetes.io/pod-uid": ["2e859424-41ca-4f64-a1c6-b0ae3395e480"]
                }
        },
        "sourceIPs": ["10.0.16.202"],
        "userAgent": "operator/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election",
        "objectRef": {
                "resource": "configmaps",
                "namespace": "tigera-operator",
                "name": "operator-lock",
                "uid": "5f5b044f-9d7e-4c89-b703-f85affe91a25",
                "apiVersion": "v1",
                "resourceVersion": "13338825"
        },
        "responseStatus": {
                "metadata": {},
                "code": 200
        },
        "requestReceivedTimestamp": "2024-05-09T01:51:24.165036Z",
        "stageTimestamp": "2024-05-09T01:51:24.167660Z",
        "annotations": {
                "authorization.k8s.io/decision": "allow",
                "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"tigera-operator\" of ClusterRole \"tigera-operator\" to ServiceAccount \"tigera-operator/tigera-operator\""
        }
}

审计日志阶段

审计日志根据日志策略可以选择在事件执行的某个阶段记录，目前支持的事件阶段有：

RequestReceived 接收到事件且在分配给对应 handler 前记录。
ResponseStarted 开始响应数据的 Header 但在响应数据 Body 发送前记录，这种一般应用在持续很长的操作事件，例如 watch 操作。
ResponseComplete 事件响应完毕后记录。
Panic 内部出现 panic 时记录。

审计日志等级

审计日志根据日志策略可以选择事件保存的等级，根据等级不同，APIServer 记录日志的详细程度也不同。目前支持的事件等级有：

None 不记录日志。
Metadata 只记录 Request 的一些 metadata (例如 user, timestamp, resource, verb 等)，但不记录 Request 或 Response 的 body。
Request 记录 Request 的 metadata 和 body。
RequestResponse 最全记录方式，会记录所有的 metadata、Request和 Response 的 Body。

审计日志策略

APIServer 支持对每类不同的资源设置不同的审计日志策略，包括日志记录阶段以及日志记录等级，目前官方以及很多云厂商都会提供日志策略，一般都遵循以下原则：

在收到请求后不立即记录日志，当返回体 header 发送后才开始记录。
对于大量冗余的 kube-proxy watch 请求，kubelet 和 system:nodes 对于 node 的 get 请求，kube 组件在 kube-system 下对于 endpoint 的操作，以及 apiserver 对于 namespaces 的 get 请求等不作审计。
对于 /healthz，/version，/swagger* 等只读 url 不作审计。
对于可能包含敏感信息或二进制文件的 secrets，configmaps，tokenreviews 接口的日志等级设为 metadata，该 level 只记录请求事件的用户、时间戳、请求资源和动作，而不包含请求体和返回体。
对于一些如 authenticatioin、rbac、certificates、autoscaling、storage 等敏感接口，根据读写记录相应的请求体和返回体。

开启审计日志策略

以 K8S 1.24 为例，开启审计日志策略。

1、登陆主节点服务器

cd /etc/kubernetes

2、新增 audit-policy.yml

# Copyright © 2022 sealos.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following requests were manually identified as high-volume and low-risk,
  # so drop them.
  - level: None
    users: [ "system:kube-proxy" ]
    verbs: [ "watch" ]
    resources:
      - group: "" # core
        resources: [ "endpoints", "services" ]
  - level: None
    users: [ "system:unsecured" ]
    namespaces: [ "kube-system" ]
    verbs: [ "get" ]
    resources:
      - group: "" # core
        resources: [ "configmaps" ]
  - level: None
    users: [ "kubelet" ] # legacy kubelet identity
    verbs: [ "get" ]
    resources:
      - group: "" # core
        resources: [ "nodes" ]
  - level: None
    userGroups: [ "system:nodes" ]
    verbs: [ "get" ]
    resources:
      - group: "" # core
        resources: [ "nodes" ]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: [ "get", "update" ]
    namespaces: [ "kube-system" ]
    resources:
      - group: "" # core
        resources: [ "endpoints" ]
  - level: None
    users: [ "system:apiserver" ]
    verbs: [ "get" ]
    resources:
      - group: "" # core
        resources: [ "namespaces" ]
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Don't log events requests.
  - level: None
    resources:
      - group: "" # core
        resources: [ "events" ]
  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    resources:
      - group: "" # core
        resources: [ "secrets", "configmaps" ]
      - group: authentication.k8s.io
        resources: [ "tokenreviews" ]
  # Get repsonses can be large; skip them.
  - level: Request
    verbs: [ "get", "list", "watch" ]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for known APIs
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
      - group: "autoscaling.alibabacloud.com"
  # Default level for all other requests.
  - level: Metadata

对应的策略信息可以按照实际需求进行调整。

API Server 开启审计日志

进入目录 /etc/kubernetes/manifests，先备份 kube-apiserver.yaml 文件，并且备份的文件不能放在 /etc/kubernetes/manifests/ 下，调整文件内容：

1、在 spec.containers.command 下添加命令：

  - command:
    - kube-apiserver
    - --advertise-address=10.0.16.204
    - --allow-privileged=true
    - --audit-log-format=json
    - --audit-log-maxage=7
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-policy-file=/etc/kubernetes/audit-policy.yml

2、在 spec.containers.volumeMounts 下添加：

    - mountPath: /etc/kubernetes
      name: audit
    - mountPath: /var/log/kubernetes
      name: audit-log

3、在 spec.volumes 下添加：

  - hostPath:
      path: /etc/kubernetes
      type: DirectoryOrCreate
    name: audit
  - hostPath:
      path: /var/log/kubernetes
      type: DirectoryOrCreate
    name: audit-log

4、生效

API Server 被改动后，会自动重启，耐心等待几分钟即可。

5、验证

执行以下命令，看看是否有 audit.log 文件产生，如果有则证明已经生效。

ls /var/log/kubernetes

观测云采集 K8S 审计日志

接入步骤：

部署 DataKit 采集器
采集 K8S 审计日志
查看 K8S 审计日志报表

部署 DataKit 采集器

K8s 安装 - 观测云文档

采集 K8S 审计日志

K8S 审计日志存储在对应 master 节点的 /var/log/kubernetes 目录下，这里采用 annotation 的方式进行采集。

创建 pod

k8s-audit-log.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kube-audit-log
  annotations:
        datakit/logs: |
          [
            {
              "disable": false,
              "type": "file",
              "path":"/var/log/kubernetes/audit.log",
              "source":  "k8s-audit",
              "tags" : {
                "atype": "kube-audit-log"
              }
            }
          ]

spec:
  containers:
  - name: kube-audit-log
    image: busybox
#    command: ["sleep", "1"]
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        i=$((i+1));
        sleep 10;
      done
    volumeMounts:
    - mountPath: /var/log/kubernetes
      name: datakit-vol-opt
  volumes:
  - name: datakit-vol-opt
    hostPath: 
      path: /var/log/kubernetes
  nodeSelector:
     kubernetes.io/hostname: k8s-master
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"

需要注意当前 pod 只能运行在 master 节点上。

执行

kubectl apply -f k8s-audit-log.yaml

查看

等几分钟后就可以在观测云上查看到对应的日志了。

由于是 json 格式，观测云支持通过 @+json字段名 的方式进行搜索，如 @verb:update⁠

查看 K8S 审计日志报表

审计日志采集上来后，通过观测云 pipeline 的能力，可以对审计日志关键字段进行提取，从而对审计日志进行进一步数据分析。

新建 pipeline

1、选择对应的日志来源 k8s-audit

2、Pipeline 名称：kubelet-audit

3、定义解析规则

abc = load_json(_)

add_key(kind, abc["kind"])
add_key(level, abc["level"])
add_key(stage, abc["stage"])
add_key(verb, abc["verb"])
add_key(auditID, abc["auditID"])
add_key(username, abc["user"]["username"])
add_key(responseCode, abc["responseStatus"]["code"])
if abc["responseStatus"]["code"]==200 {
  add_key(status, "OK")
}else{
  add_key(status, "FAIL")
}
add_key(sourceIP_0,abc["sourceIPs"][0])
  
add_key(namespace,abc["objectRef"]["namespace"])
add_key(node,abc["objectRef"]["name"])

4、点击获取脚本测试

5、保存