What are Kubernetes audit logs?
Audit logs record requests to the Kubernetes API. Because the API server processes all changes to Kubernetes state—and serves as the gatekeeper to the backend database that stores this state—the API server is an ideal point for capturing all changes that occur within a cluster. The API server processes requests from various internal and external components, including:
- The control plane (built-in controllers, the scheduler)
- Node daemons (the kubelet, kube-proxy, and others)
- Cluster services (e.g., the cluster autoscaler, kube-state-metrics, CoreDNS, etc.)
- Users making
kubectlrequests - Applications, controllers, and operators that send requests through a kube client
- Even the API server itself
Audit logs help keep tabs on all of this complexity by recording which users or services requested access to cluster resources—and why the API server authorized or rejected those requests.
Audit logs are structured in JSON. Each log contains rich metadata in the form of key attributes like the HTTP method, the URL path requested, and the user who sent the request. For instance, you can inspect these attributes to obtain more information about an unusual request, or use a monitoring service to analyze API traffic to detect trends or identify possible threats.
Why should you monitor Kubernetes audit logs?
Although Kubernetes audit logs are not enabled by default, we recommend leveraging this feature to debug issues in your cluster. Kubernetes events are key to understanding the behavior of a cluster. By default events are stored in the Kubernetes backend database, etcd, with only one hour of retention. Audit logs allow you to capture events, create/read/update/delete actions, and even the heartbeats sent from the nodes to the API server.
By persisting audit logs to longer term storage, it’s possible to go back in time and answer questions such as, “Why was this pod evicted?” and “What lifecycle operations occur when we update a deployment?”
You can also set up automated alerts to keep tabs on any unexpected activity in your Kubernetes audit logs, such as create/update/patch/delete requests during an operational freeze period. Below, we’ll walk through some examples of using audit logs to monitor:
Kubernetes 审计日志
您所知，Kubernetes 有一个控制平面来管理整个集群生命周期，这个控制平面由多个组件组成。在我看来，Kubernetes API Server 是其中最关键的组件。基本上，我们认为 Kubernetes API Server 是集群的大脑。所以有时我们想知道大脑内部发生了什么。😅
此时，“Kubernetes 审计日志”出现了。
Kubernetes 审计日志是解释大脑内部发生了什么的记录，但如果我们从技术角度来看它们，官方文档说“Kubernetes 审计日志提供了一组与安全相关的、按时间顺序排列的记录，记录了集群中的操作顺序。” ，因此通过收集和分析它们，我们可以回答以下问题：
- 发生了什么？🐵
- 什么时候发生的？🙈
- 谁发起的？🙊
- 它是怎么发生的？🙉
在Kubernetes集群中，API Server的审计日志记录了哪些用户、哪些服务请求操作集群资源，并且可以编写不同规则，控制忽略、存储的操作日志。
事件和阶段
当某个人或组件向 Kubernetes API 服务器发出请求时，该请求将经历一个或多个阶段：
| 阶段 | 说明 |
|---|---|
| RequestReceived | 审核处理程序已收到请求。 |
| ResponseStarted | 已发送响应标头，但尚未发送响应正文。 |
| ResponseComplete | 响应正文已完成，不再发送任何字节。 |
| Panic | 内部服务器出错，请求未完成。 |
请求的每个阶段都会生成一个事件，该事件根据策略进行处理。策略指定是否应将事件记录为日志条目，如果需要记录，应将哪些数据包含在日志条目中。
审核级别
Kubernetes 审核政策文件包含一个规则列表。在政策文件中，与事件匹配的第一项规则设置事件的审核级别。
规则可以指定以下审核级别之一：
| 级别 | 说明 |
|---|---|
| None | 不为事件创建日志条目。 |
| Metadata | 创建日志条目。包括元数据，但不包括请求正文或响应正文。 |
| Request | 创建日志条目。包括元数据和请求正文，但不包括响应正文。 |
| RequestResponse | 创建日志条目。包括元数据、请求正文和响应正文。 |
下面是一个示例规则。如果事件与规则匹配，Kubernetes API 服务器将在 Request 级别创建一个日志条目。
每个level就是每个规则，如果不写规则，那么所有日志都会记录。可以使用白名单去实现。不同的级别代表记录的日志包含哪些。
- level: Request
userGroups: ["system:nodes"]
verbs: ["update","patch"]
resources:
- group: "" # core
resources: ["nodes/status", "pods/status"]
omitStages:
- "RequestReceived"如果满足以下所有条件，则事件与规则匹配：
- 事件与政策文件中前面的任何规则都不匹配。
- 进行调用的身份属于
system:nodes用户组。 - 该调用是
update请求或patch请求。 - 请求针对
nodes/status资源或pods/status资源。 - 事件不是针对调用的
RequestReceived阶段。
审计策略（审计阶段 + 审计级别）
审计策略定义了kubernetes哪些资源的事件被记录以及被记录事件的数据规则，对应参数audit-policy-file，文件配置会涉及到两个十分重要的名词解释：审计阶段和审计级别。
审计阶段
及记录事件的时机，这里分为四种类型：
RequestReceived：一旦接收到请求即刻记录审计事件ResponseStarted：响应头发出，响应消息体未发出之前，一般适用于长连接或者耗时任务的场景ResponseComplete：响应消息体完成Panic：出现panic
审计级别
代表记录审计日志的完整程度，常见的也有四种：
None：不记录审计日志；Metadata：记录请求的原子数据信息比如请求的user、method、时间、资源类型等，但不包括请求体和响应体；Request：记录包括Metadata和请求体，但不包括响应体；RequestResponse：记录包括Metadata、请求体、响应体；
当然，除了上述两个关键字之外，审计策略还应包括具体的规则(rules)，比如请求的user、method、resource等，并且这里规则的配置跟kubernetes role规则的写法基本一致，举一个简单的审计策略配置的例子，只针对pod的创建和删除记录审计日志：
# cat /etc/kubernetes/pki/audit-policy.yaml
apiVersion: audit.k8s.io/v1beta1 # 必须字段，固定写法
kind: Policy # 必须字段，固定写法
omitStages:
- "RequestReceived" # 审计阶段
rules: # rule按顺序匹配
- level: Request # 审计级别
verbs:
- create
- delete
resources:
- group: ""
resources:
- pods
- level: NoneCreate a file called audit-policy.yaml with the following recommended policy:
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Generate audit events only for ResponseComplete or panic stages of a request.
omitStages:
- "RequestReceived"
- "ResponseStarted"
rules:
# Audit on pod exec/attach events
- level: Request
resources:
- group: ""
resources: ["pods/exec", "pods/attach"]
# Audit on pod creation events
- level: Request
resources:
- group: ""
resources: ["pods"]
verbs: ["create"]
# Audit on changes to the twistlock namespace (defender daemonset)
- level: Request
verbs: ["create", "update", "patch", "delete"]
namespaces: ["twistlock"]
# Default catch all rule
- level: NoneapiVersion: audit.k8s.io/v1
kind: Policy
rules:
# do not log requests to the following
- level: None
nonResourceURLs:
- "/healthz*"
- "/logs"
- "/metrics"
- "/swagger*"
- "/version"
# limit level to Metadata so token is not included in the spec/status
- level: Metadata
omitStages:
- RequestReceived
resources:
- group: authentication.k8s.io
resources:
- tokenreviews
# extended audit of auth delegation
- level: RequestResponse
omitStages:
- RequestReceived
resources:
- group: authorization.k8s.io
resources:
- subjectaccessreviews
# log changes to pods at RequestResponse level
- level: RequestResponse
omitStages:
- RequestReceived
resources:
- group: "" # core API group; add third-party API services and your API services if needed
resources: ["pods"]
verbs: ["create", "patch", "update", "delete"]
# log everything else at Metadata level
- level: Metadata
omitStages:
- RequestReceived
Kubernetes 审核政策摘要
-
通常，
create、update和delete等写入请求会在RequestResponse级别记录。 -
通常，
get、list和watch事件会在Metadata级别记录。 -
有些事件被视为特殊情况。如需查看被视为特殊情况的请求的确切列表，请参阅政策脚本。在撰写本文时，以下这些是特殊情况：
-
由
kubelet、system:node-problem-detector或system:serviceaccount:kube-system:node-problem-detector发出的针对nodes/status资源或pods/status资源的 update 和 patch 请求会在 Request 级别记录。 -
由
system:nodes组中的任何身份发出的针对nodes/status资源或pods/status资源的 update 和 patch 请求会在 Request 级别记录。 -
system:serviceaccount:kube-system:namespace-controller发出的deletecollection请求会在 Request 级别记录。 -
针对
secrets资源、configmaps资源或tokenreviews资源的请求会在 Metadata 级别记录。
-
-
某些请求是不会被记录的。如需查看系统不会记录的请求的确切列表，请参阅政策脚本中的
level: None规则。截至撰写本文时为止，以下请求将不会进行记录：-
system:kube-proxy发出的监视endpoints资源、services资源或services/status资源的请求。 -
system:unsecured发出的针对kube-system命名空间中configmaps资源的 get 请求。 -
kubelet发出的针对nodes资源或nodes/status资源的 get 请求。 -
system:nodes组中的任何身份发出的针对nodes资源或nodes/status资源的 get 请求。 -
system:kube-controller-manager、system:kube-scheduler或system:serviceaccount:endpoint-controller发出的针对kube-system命名空间中endpoints资源的 get 和 update 请求。 -
system:apiserver发出的针对namespaces资源、namespaces/status资源或namespaces/finalize资源的 get 请求。 -
system:kube-controller-manager发出的针对metrics.k8s.io组中任何资源的 get 和 list 请求。 -
对与
/healthz*、/version或/swagger*匹配的网址发出的请求。
-
如何配置 Kubernetes 审计日志
你的 Kubernetes 审计政策指定要在审核日志中捕获的 API 请求类型。每个 Kubernetes API 调用由阶段组成，包括RequestReceived（API 服务器收到请求）和ResponseComplete（API 服务器生成响应）。有些阶段并不适用于所有类型的请求——例如，一个watch操作从RequestReceived到ResponseStarted阶段移动而不是直接到阶段，ResponseComplete因为它不断地观察更新并相应地接收响应，而不是只接收一个响应。
在您的审计策略中，您可以配置每个请求的哪些阶段应该生成日志，以及每种类型资源所需的详细级别。
尽管您的审核日志策略会根据您的用例而有所不同，但我们有一些通用准则可供分享：
- 该
RequestReceived阶段通常是多余的，因此您可以使用omitStages忽略它。该RequestReceived阶段会跟踪 API 服务器何时首次收到请求，但此信息已requestReceivedTimestamp在该ResponseComplete阶段的审计日志属性中捕获。 - 使用
level: None忽略不捕捉相关信息，如端点/healthz和/metrics。在Datadog，我们也忽略了/logs，/swagger*和/version。 - 该
RequestResponse级别捕获请求的主体和结果响应。尽管这可以提供很多有价值的信息，但请记住，具有大响应负载的请求可能会给 API 服务器带来大量负载，并增加审计日志所需的存储空间。
设置您的审计策略
配置 Kubernetes 审计策略后，使用--audit-policy-file标志指向文件，并使用--audit-log-path指定 API 服务器应输出审计日志的文件路径。如果不指定路径，API 服务器会将日志输出到stdout.
- --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
- --audit-log-path=/var/log/k8s_audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=10
- --audit-log-maxsize=100
- --audit-log-path=/var/log/k8s_audit.log 这个是将日志写到本地文件
- --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml 必须得配置记录的策略，没有这个策略是不会记录任何日志的
[root@master kubernetes]# mkdir -p audit
[root@master kubernetes]# cd audit/
[root@master audit]# pwd
/etc/kubernetes/audit - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
- --audit-log-path=/var/log/k8s_audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=10
- --audit-log-maxsize=100
volumes:
- hostPath:
path: /etc/kubernetes/audit
type: DirectoryOrCreate
name: audit
- hostPath:
path: /var/log/k8s_audit.log
type: FileOrCreate
name: audit-log
- mountPath: /etc/kubernetes/audit
name: audit
- mountPath: /var/log/k8s_audit.log
name: audit-log[root@master audit]# cat audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
# Log pod changes at RequestResponse level
- level: RequestResponse
resources:
- group: ""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources: ["pods"]
忽略请求接收这一步，因为这一块也没有什么信息可以记录的，节省资源
omitStages:
- "RequestReceived"这里只记录pod的
[root@master audit]# tail -f /var/log/k8s_audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"e4770504-4335-4a56-972a-fee7f79e0fce","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/pods?allowWatchBookmarks=true\u0026resourceVersion=25814662\u0026timeout=8m20s\u0026timeoutSeconds=500\u0026watch=true","verb":"watch","user":{"username":"system:serviceaccount:kubesphere-monitoring-system:prometheus-k8s","uid":"a67057d6-5c9e-4499-9b43-186c8275b992","groups":["system:serviceaccounts","system:serviceaccounts:kubesphere-monitoring-system","system:authenticated"]},"sourceIPs":["192.168.0.3"],"userAgent":"Prometheus/discovery","objectRef":{"resource":"pods","namespace":"default","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Success","message":"Connection closed early","code":200},"requestReceivedTimestamp":"2021-07-30T02:22:28.564604Z","stageTimestamp":"2021-07-30T02:30:48.565100Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"kubesphere-prometheus-k8s\" of ClusterRole \"kubesphere-prometheus-k8s\" to ServiceAccount \"prometheus-k8s/kubesphere-monitoring-system\""}}
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "RequestResponse",
"auditID": "e4770504-4335-4a56-972a-fee7f79e0fce",
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces/default/pods?allowWatchBookmarks=true\u0026resourceVersion=25814662\u0026timeout=8m20s\u0026timeoutSeconds=500\u0026watch=true",
"verb": "watch",
"user": {
"username": "system:serviceaccount:kubesphere-monitoring-system:prometheus-k8s",
"uid": "a67057d6-5c9e-4499-9b43-186c8275b992",
"groups": ["system:serviceaccounts", "system:serviceaccounts:kubesphere-monitoring-system", "system:authenticated"]
},
"sourceIPs": ["192.168.0.3"],
"userAgent": "Prometheus/discovery",
"objectRef": {
"resource": "pods",
"namespace": "default",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"status": "Success",
"message": "Connection closed early",
"code": 200
},
"requestReceivedTimestamp": "2021-07-30T02:22:28.564604Z",
"stageTimestamp": "2021-07-30T02:30:48.565100Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"kubesphere-prometheus-k8s\" of ClusterRole \"kubesphere-prometheus-k8s\" to ServiceAccount \"prometheus-k8s/kubesphere-monitoring-system\""
}
}内部的组件不记录日志 ，因为规则是从上到下进行匹配的，所以不需要记录的规则一般往上面放着，只要上面匹配了，就不会记录了。
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
- "RequestReceived"
rules:
- level: None
users:
- system:apiserver
- system:kube-controller-manager
- system:kube-scheduler
- system:kube-proxy
- kubeletYou can use a minimal audit policy file to log all requests at the Metadata level:
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
..................................... #这里一大堆的
- level: Metadata
代表所有规则按照metadata去采集收集审计日志方案：
扫一扫
2445
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
