OpenShift 4 - 利用 File Integrity Operator 实现对集群节点进行入侵检测（附视频）

《OpenShift / RHEL / DevSecOps 汇总目录》
说明：本文已经在OpenShift 4.9环境中验证

File Integrity Operator 功能介绍

OpenShift 的 File Integrity Operator 可以在集群节点上持续地运行文件完整性检查。它部署了一个 DaemonSet，在每个节点上初始化并运行有特权的AIDE（Advanced Intrusion Detection Environment - 高级入侵检测环境）容器，提供自 DaemonSet 初始运行以后被修改的文件记录。

File Integrity Operator 根据从配置文件中找到的正则表达式规则创建一个数据库。一旦数据库被初始化，它就可以用来验证文件的完整性。它有几种消息摘要算法用于检查文件的完整性，另外文件属性也可以被检查出不一致的地方。

安装 File Integrity Operator

在 OpenShift 上使用缺省配置安装 File Integrity Operator ，缺省会将资源安装在 openshift-file-integrity 项目下。

安装后可以看到 File Integrity Operator 实例：

创建 FileIntegrity 对象

1. 在 OpenShift 中根据以下内容创建FileIntegrity对象。其中 spec.config.gracePeriod 是两次 AIDE 完整性检查之间暂停的秒数。由于在节点中频繁进行 AIDE 检查需要大量资源，因此可以指定较长的时间间隔（默认为 900 秒）。

apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:
  name: example-fileintegrity
  namespace: openshift-file-integrity
spec:
  config:
    gracePeriod: 900
  debug: false
  tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
      operator: Exists

2. 如果没有指定，FileIntegrity 会在所有节点运行一个 Daemonset Pod 以进行完整性检查。执行命令可以看到 Daemonset 运行在集群的 5 个节点中。

$ oc get daemonset -n openshift-file-integrity
NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
aide-example-fileintegrity   5         5         5       5            5           <none>          7h41m
 
$ oc get pod -n openshift-file-integrity -o wide -l app=aide-example-fileintegrity
NAME                               READY   STATUS    RESTARTS   AGE     IP            NODE                                         NOMINATED NODE   READINESS GATES
aide-example-fileintegrity-2vpbq   1/1     Running   1          7h45m   10.128.2.15   ip-10-0-241-123.us-east-2.compute.internal   <none>           <none>
aide-example-fileintegrity-czf4w   1/1     Running   1          7h45m   10.131.0.11   ip-10-0-134-47.us-east-2.compute.internal    <none>           <none>
aide-example-fileintegrity-vx5q7   1/1     Running   1          7h45m   10.128.0.6    ip-10-0-173-213.us-east-2.compute.internal   <none>           <none>
aide-example-fileintegrity-wd4pm   1/1     Running   1          7h45m   10.129.0.10   ip-10-0-173-152.us-east-2.compute.internal   <none>           <none>
aide-example-fileintegrity-wx42d   1/1     Running   1          7h45m   10.130.0.8    ip-10-0-135-39.us-east-2.compute.internal    <none>           <none>

3. 查看内容为 “FileIntegrityStatus” 的事件，可以看到当前已经是 “Active”。

$ oc get event -n openshift-file-integrity --field-selector reason=FileIntegrityStatus
LAST SEEN   TYPE     REASON                OBJECT                                MESSAGE
2m28s       Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Pending
2m28s       Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Initializing
2m13s       Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Active

4. 在example-fileintegrity完成首次扫描后，可以执行以下命令确认所有节点的 fileintegritynodestatus 的状态都是 “Succeeded”，这说明被监控节点的目标文件没有变化。

$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal    ip-10-0-134-47.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal    ip-10-0-135-39.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal   ip-10-0-173-152.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal   ip-10-0-173-213.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal   ip-10-0-241-123.us-east-2.compute.internal   Succeeded

5. 另外查看事件也能看到此时所有目标都 “no changes to node”。

$ oc get events -n openshift-file-integrity --field-selector reason=NodeIntegrityStatus
LAST SEEN   TYPE     REASON                OBJECT                                MESSAGE
7m4s        Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-134-47.us-east-2.compute.internal
7m3s        Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-241-123.us-east-2.compute.internal
7m2s        Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-173-152.us-east-2.compute.internal
7m          Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-135-39.us-east-2.compute.internal
6m59s       Normal   NodeIntegrityStatus   fileintegrity/example-fileintegrity   no changes to node ip-10-0-173-213.us-east-2.compute.internal

修改节点配置文件

1. 进入到一个Node的RHCOS内部。

$ NODE=ip-10-0-135-39.us-east-2.compute.internal
$ NODESTATUS=example-fileintegrity-${NODE}
$ oc debug node/${NODE}

2. 执行命令，更新 “** /host/etc/resolv.conf**” 文件。

sh-4.2# echo "# integrity test" >> /host/etc/resolv.conf
sh-4.2# exit

确认文件完整性发生变化

1. 执行以下命令，或在 OpenShift 控制台上查看 example-fileintegrity 的事件。确认提示有一个节点的完整性检查失败，其中完整性变化是：“a:0,c:1,r:0”，即：0个新增（a）；1个变化（c）；0个删除（r）。

$ oc get events -n openshift-file-integrity --field-selector reason=NodeIntegrityStatus,type=Warning
LAST SEEN   TYPE      REASON                OBJECT                                MESSAGE
17m         Warning   NodeIntegrityStatus   fileintegrity/example-fileintegrity   node ip-10-0-135-39.us-east-2.compute.internal has changed! a:0,c:1,r:0 log:openshift-file-integrity/aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed

2. 还可执行以下命令，确认其中一个的 STATUS 已经变为 “Failed”。

$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal    ip-10-0-134-47.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal    ip-10-0-135-39.us-east-2.compute.internal    Failed
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal   ip-10-0-173-152.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal   ip-10-0-173-213.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal   ip-10-0-241-123.us-east-2.compute.internal   Succeeded

3. 执行以下命令，可以在该节点的 fileintegritynodestatus 中查到记录的完整性检查结果。可以看到完整性检查开始是 “Secceeded”，然后变成了“Failed”，这是由 1 个 filesChanged 引起的。File Integrity Operator 将完整性检查详细结果存放在 “openshift-file-integrity” 项目中名为 “aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed” 的 ConfigMap 对象中了。

$ oc get fileintegritynodestatus/${NODESTATUS} -n openshift-file-integrity -ojsonpath='{.results}' | jq -r
[
  {
    "condition": "Succeeded",
    "lastProbeTime": "2022-03-05T03:19:42Z"
  },
  {
    "condition": "Failed",
    "filesChanged": 1,
    "lastProbeTime": "2022-03-05T03:35:27Z",
    "resultConfigMapName": "aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed",
    "resultConfigMapNamespace": "openshift-file-integrity"
  }
]

4. 从 ConfigMap 对象中查看完整性检查结果失败说明，其中提示 “/hostroot/etc/resolv.conf” 发生了变化。

$ oc describe cm aide-${NODESTATUS}-failed

Name:         aide-example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal-failed
Namespace:    openshift-file-integrity
Labels:       file-integrity.openshift.io/node=ip-10-0-135-39.us-east-2.compute.internal
              file-integrity.openshift.io/owner=example-fileintegrity
              file-integrity.openshift.io/result-log=
Annotations:  file-integrity.openshift.io/files-added: 0
              file-integrity.openshift.io/files-changed: 1
              file-integrity.openshift.io/files-removed: 0
 
Data
====
integritylog:
----
Start timestamp: 2022-03-05 03:50:27 +0000 (AIDE 0.16)
AIDE found differences between database and filesystem!!
 
Summary:
  Total number of entries:  35211
  Added entries:              0
  Removed entries:            0
  Changed entries:            1
 
---------------------------------------------------
Changed entries:
---------------------------------------------------
 
f   ...    .C... : /hostroot/etc/resolv.conf
 
---------------------------------------------------
Detailed information about changes:
---------------------------------------------------
 
File: /hostroot/etc/resolv.conf
  SHA512   : Xl2pzxjmRPtW8bl6Kj49SkKOSBVJgsCI | 4M9rtSPrVj1Q34k0r/j3sZlreZT6uieC
             Fw3VVMAbuGgHwINR5Fi5CYGuLhNmmDNh | 850ZVM9KcJ5DYB5Gf0IGbuojvIF1exdI
             dC7kQIKcAjFwB3Ib+UaOzw==         | vXZwgl8BvjmcezDPzmTGPA==
 
---------------------------------------------------
The attributes of the (uncompressed) database(s):
---------------------------------------------------
 
/hostroot/etc/kubernetes/aide.db.gz
  MD5      : OeZ96l1VZFyigc/hL7vv5w==
  SHA1     : a8EFf5Ylgi1h86rq549XFFea6CM=
  RMD160   : SV7z0d1kjNxzSRn0Pu+6Hps02Ew=
  TIGER    : HHLY/HeLmpTjF3/bNyTjcZ3+/AlDfYrK
  SHA256   : DsDlSlIxzysKry78/mYualTEt6BqtjJJ
             nIoRhHwy1jo=
  SHA512   : iIIJZZKx36suzJG2BsdG78jAonAm1EpI
             ZKmz40Gw5h6eWIu88+VyFlStpkOLCeFn
             3j8uZda/6Qgste6zBQV/Sg==
 
End timestamp: 2022-03-05 03:51:10 +0000 (run time: 0m 43s)
 
BinaryData
====
 
Events:  <none>

接受对节点文件的修改

如果这个修改是必要的，则可以接受这一修改。

1. 执行命令重置完整性对比数据库。

$ oc annotate fileintegrity/example-fileintegrity file-integrity.openshift.io/re-init=

2. 然后查看由 FileIntegrityStatus 引发的事件。

$ oc get events -n openshift-file-integrity --field-selector reason=FileIntegrityStatus
LAST SEEN   TYPE     REASON                OBJECT                                MESSAGE
54s         Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Initializing
47s         Normal   FileIntegrityStatus   fileintegrity/example-fileintegrity   Active

3. 可以再次进入前面操作的 RHCOS，确认已经在 “/host/etc/kubernetes/” 目录中对 “aide.db.gz”和“aide.log”进行了备份。

sh-4.4# chroot /host
sh-4.4# ls -lR /etc/kubernetes/aide.*
-rw-------. 1 root root 1953565 Mar  5 10:47 /etc/kubernetes/aide.db.gz
-rw-------. 1 root root 1953538 Mar  5 10:46 /etc/kubernetes/aide.db.gz.backup-20220305T10_46_29
-rw-------. 1 root root 1953565 Mar  5 10:47 /etc/kubernetes/aide.db.gz.new
-rw-------. 1 root root    1012 Mar  5 10:31 /etc/kubernetes/aide.log
-rw-------. 1 root root    1012 Mar  5 10:46 /etc/kubernetes/aide.log.backup-20220305T10_46_29
-rw-------. 1 root root     778 Mar  5 10:47 /etc/kubernetes/aide.log.new

4. 过一段时间后每个节点的 DaemonSet 会完成一次新的完整性对比。再此过程会中会基于修改后的文件重新建完整性对比数据库，这样对比状态又恢复成 “Succeeded” 了。

$ oc get fileintegritynodestatus -n openshift-file-integrity
NAME                                                               NODE                                         STATUS
example-fileintegrity-ip-10-0-134-47.us-east-2.compute.internal    ip-10-0-134-47.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-135-39.us-east-2.compute.internal    ip-10-0-135-39.us-east-2.compute.internal    Succeeded
example-fileintegrity-ip-10-0-173-152.us-east-2.compute.internal   ip-10-0-173-152.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-173-213.us-east-2.compute.internal   ip-10-0-173-213.us-east-2.compute.internal   Succeeded
example-fileintegrity-ip-10-0-241-123.us-east-2.compute.internal   ip-10-0-241-123.us-east-2.compute.internal   Succeeded

通过Prometheus监控

File Integrity 提供了 Prometheus 接口，运行以下命令可以从 Prometheus 接口访问到当前集群文件完整性的状态。其中：
“file_integrity_operator_node_failed gauge” 是以往完整性失败的情况。
“file_integrity_operator_node_status_total counter” 是当前集群节点完整性的统计。

$ oc run --rm -i --restart=Never --image=registry.fedoraproject.org/fedora-minimal:latest -n openshift-file-integrity metrics-test -- bash -c 'curl -ks -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://metrics.openshift-file-integrity.svc:8585/metrics-fio'

If you don't see a command prompt, try pressing enter.
# HELP file_integrity_operator_node_failed A gauge that is set to 1 when a node has unresolved integrity failures, and 0 when it is healthy
# TYPE file_integrity_operator_node_failed gauge
file_integrity_operator_node_failed{node="ip-10-0-134-47.us-east-2.compute.internal"} 1
file_integrity_operator_node_failed{node="ip-10-0-135-39.us-east-2.compute.internal"} 0
# HELP file_integrity_operator_node_status_total The total number of FileIntegrityNodeStatus transitions, per condition and node
# TYPE file_integrity_operator_node_status_total counter
file_integrity_operator_node_status_total{condition="Failed",node="ip-10-0-134-47.us-east-2.compute.internal"} 1
file_integrity_operator_node_status_total{condition="Succeeded",node="ip-10-0-135-39.us-east-2.compute.internal"} 1
。。。

定制 File Integrity 配置

1. 当创建一个 FileIntegrity，它的详细配置保存在对应名称的 ConfigMap 对象中。可以执行以下命令查看该 ConfigMap 的内容，其中使用正则表达式列出了所有受 FileIntegrity 保护的目标，默认配置包含以下目录中的文件 “/root”、“/boot”、“/usr”、“/etc”，而不包含的目录文件有 “/var”、“/opt”、“/etc/”。

$ oc describe cm/example-fileintegrity -n openshift-file-integrity
Name:         example-fileintegrity
Namespace:    openshift-file-integrity
Labels:       file-integrity.openshift.io/aide-conf=
              file-integrity.openshift.io/owner=example-fileintegrity
Annotations:  <none>
 
Data
====
aide.conf:
----
@@define DBDIR /hostroot/etc/kubernetes
@@define LOGDIR /hostroot/etc/kubernetes
database=file:@@{DBDIR}/aide.db.gz
database_out=file:@@{DBDIR}/aide.db.gz.new
gzip_dbout=yes
verbose=5
report_url=file:@@{LOGDIR}/aide.log.new
report_url=stdout
PERMS = p+u+g+acl+selinux+xattrs
CONTENT_EX = sha512+ftype+p+u+g+n+acl+selinux+xattrs
 
/hostroot/boot/        CONTENT_EX
/hostroot/root/\..* PERMS
/hostroot/root/   CONTENT_EX
!/hostroot/root/\.kube
!/hostroot/usr/src/
!/hostroot/usr/tmp/
 
/hostroot/usr/    CONTENT_EX
 
# OpenShift specific excludes
!/hostroot/opt/
!/hostroot/var
!/hostroot/etc/NetworkManager/system-connections/
!/hostroot/etc/mtab$
!/hostroot/etc/.*~
!/hostroot/etc/kubernetes/static-pod-resources
!/hostroot/etc/kubernetes/aide.*
!/hostroot/etc/kubernetes/manifests
!/hostroot/etc/docker/certs.d
!/hostroot/etc/selinux/targeted
!/hostroot/etc/openvswitch/conf.db
!/hostroot/etc/kubernetes/cni/net.d/*
!/hostroot/etc/machine-config-daemon/currentconfig$
!/hostroot/etc/pki/ca-trust/extracted/java/cacerts$
!/hostroot/etc/cvo/updatepayloads
 
# Catch everything else in /etc
/hostroot/etc/    CONTENT_EX
 
BinaryData
====
 
Events:  <none>

2. 可以将该 ConfigMap 内容导出成文件。

$ oc extract cm/example-fileintegrity -n openshift-file-integrity --keys=aide.conf

3. 修改后再根据修改后的文件生成一个新的 ConfigMap 对象。

$ oc create cm master-aide-conf -n openshift-file-integrity --from-file=aide.conf

4. 最后可以根据以下YAML只在 “master” 节点上根据建的 “master-aide-conf” 配置创建 FileIntegrity 对象。

node-role.kubernetes.io/master: “” 在所有 master 节点上调度 AIDE；name 和 namespace 属性指向配置映射

apiVersion: fileintegrity.openshift.io/v1alpha1
kind: FileIntegrity
metadata:
  name: master-fileintegrity
  namespace: openshift-file-integrity
spec:
  nodeSelector:
      node-role.kubernetes.io/master: ""
  config:
      name: master-aide-conf
      namespace: openshift-file-integrity

演示视频

视频

参考

https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.9/html-single/security_and_compliance/index
https://github.com/openshift/file-integrity-operator
https://aide.github.io/