kubernetes v1.8.8中 RBAC DENY 解决办法

现象

在node节点上日志总是报以下错误

[root@node-01 ssl]# journalctl -f -u kubelet
-- Logs begin at 六 2018-04-21 16:27:19 CST. --
4月 21 18:49:56 node-01 kubelet[11776]: E0421 18:49:56.130755   11776 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:422: Failed to list *v1.Node: nodes is forbidden: User "system:node:192.168.1.35" cannot list nodes at the cluster scope
4月 21 18:49:56 node-01 kubelet[11776]: E0421 18:49:56.138287   11776 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:413: Failed to list *v1.Service: services is forbidden: User "system:node:192.168.1.35" cannot list services at the cluster scope
4月 21 18:49:56 node-01 kubelet[11776]: E0421 18:49:56.141303   11776 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:node:192.168.1.35" cannot list pods at the cluster scope
4月 21 18:49:57 node-01 kubelet[11776]: E0421 18:49:57.135185   11776 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:422: Failed to list *v1.Node: nodes is forbidden: User "system:node:192.168.1.35" cannot list nodes at the cluster scope
4月 21 18:49:57 node-01 kubelet[11776]: E0421 18:49:57.143383   11776 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:413: Failed to list *v1.Service: services is forbidden: User "system:node:192.168.1.35" cannot list services at the cluster scope
4月 21 18:49:57 node-01 kubelet[11776]: E0421 18:49:57.149522   11776 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: pods is forbidden: User "system:node:192.168.1.35" cannot list pods at the cluster scope
4月 21 18:49:58 node-01 kubelet[11776]: E0421 18:49:58.127377   11776 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "192.168.1.35" not found
4月 21 18:49:58 node-01 kubelet[11776]: E0421 18:49:58.139449   11776 reflector.go:205] k8s.io/kubernet

在master节点上报如下错误

4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.422528     582 rbac.go:116] RBAC DENY: user "system:node:192.168.1.35" groups ["system:nodes" "system:authenticated"] cannot "list" resource "nodes" cluster-wide
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.422731     582 wrap.go:42] GET /api/v1/nodes?fieldSelector=metadata.name%3D192.168.1.35&resourceVersion=0: (735.866µs) 403 [[kubelet/v1.8.8 (linux/amd64) kubernetes/2f73858] 192.168.1.35:33630]
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.430153     582 rbac.go:116] RBAC DENY: user "system:node:192.168.1.35" groups ["system:nodes" "system:authenticated"] cannot "list" resource "services" cluster-wide
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.430389     582 wrap.go:42] GET /api/v1/services?resourceVersion=0: (733.459µs) 403 [[kubelet/v1.8.8 (linux/amd64) kubernetes/2f73858] 192.168.1.35:33630]
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.436612     582 rbac.go:116] RBAC DENY: user "system:node:192.168.1.35" groups ["system:nodes" "system:authenticated"] cannot "list" resource "pods" cluster-wide
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.436757     582 wrap.go:42] GET /api/v1/pods?fieldSelector=spec.nodeName%3D192.168.1.35&resourceVersion=0: (502.574µs) 403 [[kubelet/v1.8.8 (linux/amd64) kubernetes/2f73858] 192.168.1.35:33630]
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.843132     582 wrap.go:42] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (3.456149ms) 200 [[kube-scheduler/v1.8.8 (linux/amd64) kubernetes/2f73858/leader-election] 192.168.1.36:47708]
4月 21 18:49:14 master-01 kube-apiserver[582]: I0421 18:49:14.857052     582 wrap.go:42] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (10.143296ms) 200 [[kube-scheduler/v1.8.8 (linux/amd64) kubernetes/2f73858/leader-election] 192.168.1.36:47708]
4月 21 18:49:15 master-01 kube-apiserver[582]: I0421 18:49:15.328868     582 wrap.go:42] GET /api/v1/namespaces?resourceVersion=4312&timeoutSeconds=496&watch=true: (8m16.000980289s) 200 [[kube-controller-manager/v1.8.8 (linux/amd64) kubernetes/2f73858/shared-informers] 192.168.1.36:47666]
4月 21 18:49:15 master-01 kube-apiserver[582]: I0421 18:49:15.333133     582 rest.go:362] Starting watch for /api/v1/namespaces, rv=4312 labels= fields= timeout=6m56s
4月 21 18:49:15 master-01 kube-apiserver[582]: I0421 18:49:15.427169     582 rbac.go:116] RBAC DENY: user "system:node:192.168.1.35" groups ["system:nodes" "system:authenticated"] cannot "list" resource "nodes" cluster-wide
4月 21 18:49:15 master-01 kube-apiserver[582]: I0421 18:49:15.427411     582 wrap.go:42] GET /api/v1/nodes?fieldSelector=metadata.name%3D192.168.1.35&resourceVersion=0: (1.0657ms) 403 [[kubelet/v1.8.8 (linux/amd64) kubernetes/2f73858] 192.168.1.35:33630]
4月 21 18:49:15 master-01 kube-apiserver[582]: I0421 18:49:15.434277     582 rbac.go:116] RBAC DENY: user "system:node:192.168.1.35" groups ["system:nodes" "system:authenticated"] cannot "list" resource "services" cluster-wide

原因

1.8版本之前.开启rbac后,apiserver默认绑定system:nodes组到system:node的clusterrole。v1.8之后,此绑定默认不存在,需要手工绑定,否则kubelet启动后会报认证错误，使用kubectl get nodes查看无法成为Ready状态。

默认角色与默认角色绑定

API Server会创建一组默认的 ClusterRole和 ClusterRoleBinding对象。 这些默认对象中有许多包含 system:前缀，表明这些资源由Kubernetes基础组件”拥有”。 对这些资源的修改可能导致非功能性集群（non-functional cluster） 。一个例子是 system:node ClusterRole对象。这个角色定义了kubelets的权限。如果这个角色被修改，可能会导致kubelets无法正常工作。
所有默认的ClusterRole和ClusterRoleBinding对象都会被标记为kubernetes.io/bootstrapping=rbac-defaults。
使用命令kubectl get clusterrolebinding和kubectl get clusterrole可以查看系统中的角色与角色绑定

使用命令kubectl get clusterrolebindings system:node -o yaml或kubectl describe clusterrolebindings system:node查看system:node角色绑定的详细信息：

[root@node-01 kubernetes]# kubectl describe clusterrolebindings system:node
Name:         system:node
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind  Name  Namespace
  ----  ----  ---------
[root@node-01 kubernetes]# kubectl describe clusterrolebindings system:node
Name:         system:node
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind  Name  Namespace
  ----  ----  ---------

system:node角色默认绑定为空

创建角色绑定

[root@node-01 kubernetes]#   kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --user=system:node:192.168.1.35
clusterrolebinding "kubelet-node-clusterbinding" created

[root@node-01 kubernetes]# kubectl get clusterrolebinding kubelet-node-clusterbinding -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: 2018-04-21T10:56:48Z
  name: kubelet-node-clusterbinding
  resourceVersion: "5624"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kubelet-node-clusterbinding
  uid: b03edc2f-4552-11e8-917b-080027587c6b
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:node:192.168.1.35
  [root@node-01 kubernetes]# kubectl delete clusterrolebindings kubelet-node-clusterbinding
clusterrolebinding "kubelet-node-clusterbinding" deleted

[root@node-01 kubernetes]# kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --group=system:nodes
clusterrolebinding "kubelet-node-clusterbinding" created
[root@node-01 kubernetes]# kubectl describe clusterrolebindings system:node
Name:         system:node
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind  Name  Namespace
  ----  ----  ---------
[root@node-01 kubernetes]# kubectl describe kubelet-node-clusterbinding
the server doesn't have a resource type "kubelet-node-clusterbinding"
[root@node-01 kubernetes]# kubectl describe clusterrolebindings kubelet-node-clusterbinding
Name:         kubelet-node-clusterbinding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind   Name          Namespace
  ----   ----          ---------
  Group  system:nodes  

查看节点

[root@node-01 kubernetes]# kubectl get nodes 
NAME           STATUS    ROLES     AGE       VERSION
192.168.1.35   Ready     <none>    1h        v1.8.8

参考
kubernetes v1.9.0中 RBAC DENY 解决办法
service-account-permissions

kubernetes v1.8.4 RBAC DENY 解决办法