CKS-最小化服务漏洞-pod security policy(PSP)

最小化服务漏洞

使用第三方的开源软件、准入控制器来提升系统的安全性

PSP （pod security policy） pod的安全策略

我们创建的pod基本上是没有任何限制的
1.此pod里的进程，可以以任何用户身份运行
2.可以使用任一类型的存储
3.比如使用hostPath，可以使用任何任一类型的宿主机目录
4.使用宿主机的网络空间
5.pod里也可以使用特权运行

我想限制：
不允许使用特权运行？
不允许使用emptyDir类型的存储？
如果使用hostPath存储的话，只能映射到宿主机的/tmp目录

PSP 启动（默认关闭） PSP开启后 用户即使有权限在创建pod时不会直接创建，而是会在psp中询问是否能创建

1.启动PSP，启动之后任何用户就不能创建pod了，要配置后才可以

root@vms81:~/cks# vim /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
root@vms81:~/cks# systemctl restart kubelet.service

root@vms81:~/cks# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "audit-pod" is forbidden: PodSecurityPolicy: no providers available to validate pod request

2.运用psp

测试禁止特权运行

2.1 创建一个psp

root@vms81:~/day3# vim psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: mypsp1   # 名为mypsp1
spec:
  privileged: false  # Don't allow privileged pods!   # 不允许使用特权运行
  # The rest fills in some required fields.
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'
root@vms81:~/day3# kubectl apply -f psp.yaml

给john一个权限，可以对pod进行管理
john–权限1
master

crole1 cbind1 ---john create,delete,get,list
root@vms81:~/day3# kubectl create clusterrole crole1 --verb=create,delete,get,list --resource=pod --dry-run=client -o yaml > crole1.yaml
root@vms81:~/day3# kubectl apply -f crole1.yaml
clusterrole.rbac.authorization.k8s.io/crole1 created

root@vms81:~/day3# kubectl create clusterrolebinding  cbind1 --clusterrole=crole1 --user=john
clusterrolebinding.rbac.authorization.k8s.io/cbind1 created

client

root@vms82:~# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: []
**提示psp无法接纳pod，但是用master是可以创建这个pod的，这边报错原因是开启psp后，用户即使有权限在创建pod时不会直接创建，而是会在psp中询问是否能创建，由于john这个用户不是特殊用户没有资格访问psp**

john–权限2
master

crole2 cbind2 --john 访问mypsp1

用户访问psp是需要授权的

root@vms81:~/day3# kubectl create clusterrole crole2 --verb=use --resource=psp --resource-name=mypsp1
clusterrole.rbac.authorization.k8s.io/crole2 created

root@vms81:~/day3# kubectl create clusterrolebinding cbind2 --clusterrole=crole2 --user=john
clusterrolebinding.rbac.authorization.k8s.io/cbind2 created

client 成功创建pod

root@vms82:~# kubectl apply -f pod1.yaml
pod/pod1 created

测试是否允许使用hostNetwork（默认允许）

root@vms81:~/day3# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used]
**开启PSP后 hostNetwork: true pod设置了true，创建时报错PSP 禁止**

root@vms81:~/day3# vim psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: mypsp1
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  hostNetwork: true                                    # 允许使用hostNetwork
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - '*'

pod 可以正常创建

测试是否允许使用存储（默认允许）

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: mypsp1
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  hostNetwork: true
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - 'emptyDir'      # PSP指定只能使用emptyDir

当pod使用的是hostPath时就会报错

root@vms81:~/day3# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]

问题，即使pod内改为了emptyDir挂载，在创建时还是报错，原因是在运行pod时会默认挂载sa到容器，而这个挂载模式是 projected ，所以还是会报错

root@vms81:~/day3# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[1]: Invalid value: "projected": projected volumes are not allowed to be used]

如何解决

方法一： automountServiceAccountToken: false 新增，以后凡是使用这个sa1的pod，都不自动挂载token
方法二： PSP规则加上 projected 这个挂载模式

root@vms81:~/day3# vim psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: mypsp1
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  hostNetwork: true
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - 'emptyDir'
  - 'projected'    # sa挂载的默认模式

在PSP中指定hostPath的挂载目录#########

root@vms81:~/day3# vim psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: mypsp1
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  hostNetwork: true
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
  - 'emptyDir'
  - 'projected'
  - 'hostPath'                                 # 指定了允许使用
  allowedHostPaths:                            # 指定了允许使用挂载的目录
  - pathPrefix: /tmp
  - pathPrefix: /xxx                           # 设置多目录允许

root@vms81:~/day3# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[0].hostPath.pathPrefix: Invalid value: "/xx": is not allowed to be used]

pod里面就不能再设置挂载目录了，只能挂载到PSP允许的目录，

测试容器内只能以某个用户运行

root@vms81:~/day3# vim psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: mypsp1
spec:
  privileged: false  # Don't allow privileged pods!
  # The rest fills in some required fields.
  hostNetwork: true
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    #rule: RunAsAny
    rule: MustRunAs          # 设置必须要满足要求的用户才可以运行
    ranges:                  # 设置了UID的范围
    - min: 1500
      max: 2000
  fsGroup:
    rule: RunAsAny
  volumes:
  - 'emptyDir'
  - 'projected'
  - 'hostPath'
  allowedHostPaths:
  - pathPrefix: /tmp

当pod指定使用1000的UID用户运行时就会报错

spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
  volumes:
  - name: v1
    hostPath:
...
root@vms81:~/day3# kubectl apply -f pod1.yaml
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [{1500 2000]]}

需求：如何实现 — john用户可以特权访问day3（ns）的用户，但是不能特权访问day4（ns）的用户

PSP中 允许的优先级要高于拒绝的优先级，所以可以设定一个针对所有ns 的都不能特权运行的mypsp1，再创建一个可以对day3 特权运行的mypsp2

1. 创建psp
   mypsp1 — privileged: false，用于clusterrole mypsp2 — privileged: true，用于role

root@vms81:~/day3# kubectl get psp
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME     PRIV    CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
mypsp1   false          RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            *
mypsp2   true           RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            *

2. 创建一个不可以特权访问所有ns的PSP
   给john一个pod的相关权限（clusterrole 就相当于针对所有ns）

root@vms81:~/day3# kubectl create clusterrole crole1 --verb=create,delete,get,list --resource=pod
clusterrole.rbac.authorization.k8s.io/crole1 created
root@vms81:~/day3# kubectl create clusterrolebinding cbind1 --clusterrole=crole1 --user=john
clusterrolebinding.rbac.authorization.k8s.io/cbind1 created

给john一个mypsp1的相关权限

root@vms81:~/day3# kubectl create clusterrole crole2 --verb=use --resource=psp  --resource-name=mypsp1
clusterrole.rbac.authorization.k8s.io/crole2 created
root@vms81:~/day3# kubectl create clusterrolebinding cbind2 --clusterrole=crole2 --user=john
clusterrolebinding.rbac.authorization.k8s.io/cbind2 created

给john一个mypsp2的相关权限

root@vms81:~/day3# kubectl create role role3 --verb=use --resource=psp  --resource-name=mypsp2 
role.rbac.authorization.k8s.io/role2 created
root@vms81:~/day3# kubectl create rolebinding rbind3 --role=role3 --user=john
rolebinding.rbac.authorization.k8s.io/rbind2 created

client pod 以特权创建测试

root@vms82:~# vim pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pod1
  name: pod1
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: pod1
    resources: {
    securityContext:                    # 开启特权
      privileged: true
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}}

实现需求，在day4下无法创建，在day3下可以创建

root@vms82:~# kubectl apply -f pod1.yaml -n day4
Error from server (Forbidden): error when creating "pod1.yaml": pods "pod1" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]
root@vms82:~# kubectl apply -f pod1.yaml -n day3
pod/pod1 created

开启PSP 对deployment的影响

开启PSP后deploy可以正常运行，但是pod不会创建

root@vms81:~/day3# kubectl describe deployments.apps deploy1
...
NewReplicaSet:     deploy1-cf4694566 (0/1 replicas created)  **没发启动成功**
...

原因是
1.deploy在运行时不是直接创建pod，而是先运行replicaSet（kubectl get rc ），再由replicaSet去创建pod
2.但是replicaSet以sa的身份去创建pod的，由于开启了PSP，sa没有权限去访问psp，所以pod就无法成功创建
解决方案：给这个sa授权可以访问PSP
给sa一个mypsp1的相关权限，由于上面 crole2 已创建并拥有 --resource=psp --resource-name=mypsp1 权限

root@vms81:~/day3# kubectl create clusterrolebinding  cbing22 --clusterrole=crole2 --group=system:serviceaccounts -n kube-system
clusterrolebinding.rbac.authorization.k8s.io/cbing22 created

–group=system:serviceaccounts 授权给所有sa 指定组

root@vms81:~/day3# kubectl get pod
NAME                      READY   STATUS    RESTARTS   AGE
deploy1-cf4694566-bxwld   1/1     Running   0          22m
pod1                      1/1     Running   0          3d18h