kubernetes v1.9.0中 RBAC DENY 解决办法

1 问题

在node1上，安装kubelet后，查看日志：

root@node1:~# tailf /var/log/syslog
Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.080370   17646 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach
Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084014   17646 kubelet_node_status.go:431] Recording NodeHasSufficientDisk event message for node 192.168.122.3
Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084061   17646 kubelet_node_status.go:431] Recording NodeHasSufficientMemory event message for node 192.168.122.3
Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084084   17646 kubelet_node_status.go:431] Recording NodeHasNoDiskPressure event message for node 192.168.122.3
Jan 15 15:01:07 node1 kubelet[17646]: I0115 15:01:07.084106   17646 kubelet_node_status.go:82] Attempting to register node 192.168.122.3
Jan 15 15:01:07 node1 kubelet[17646]: E0115 15:01:07.086790   17646 kubelet_node_status.go:106] Unable to register node "192.168.122.3" with API server: nodes is forbidden: User "system:node:192.168.122.3" cannot create nodes at the cluster scope
Jan 15 15:01:07 node1 kubelet[17646]: E0115 15:01:07.309503   17646 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "192.168.122.3" not found

在master结点上查看日志，apiserver日志报错，大量的组件都是RBAC DENY状态。：

root@master:~# tailf /var/log/syslog
Jan 18 14:39:39 master kube-apiserver[2638]: I0118 14:39:39.885925    2638 rbac.go:116] RBAC DENY: user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"] cannot "list" resource "services" cluster-wide
Jan 18 14:39:39 master kube-apiserver[2638]: I0118 14:39:39.886076    2638 wrap.go:42] GET /api/v1/services?limit=500&resourceVersion=0: (490.18µs) 403 [[kubelet/v1.9.0 (linux/amd64) kubernetes/925c127] 192.168.122.3:33354]
Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.097660    2638 rbac.go:116] RBAC DENY: user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"] cannot "list" resource "pods" cluster-wide
Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.098047    2638 wrap.go:42] GET /api/v1/pods?fieldSelector=spec.nodeName%3D192.168.122.3&limit=500&resourceVersion=0: (930.577µs) 403 [[kubelet/v1.9.0 (linux/amd64) kubernetes/925c127] 192.168.122.3:33354]
Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.359008    2638 wrap.go:42] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (16.594085ms) 200 [[kube-scheduler/v1.9.0 (linux/amd64) kubernetes/925c127/leader-election] 192.168.122.2:57472]
Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.367652    2638 wrap.go:42] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler: (7.997847ms) 200 [[kube-scheduler/v1.9.0 (linux/amd64) kubernetes/925c127/leader-election] 192.168.122.2:57472]
Jan 18 14:39:40 master kube-apiserver[2638]: I0118 14:39:40.749509    2638 rbac.go:116] RBAC DENY: user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"] cannot "list" resource "nodes" cluster-wide

2 原因：

按照k8s官方文档(https://kubernetes.io/docs/admin/authorization/rbac/#service-account-permissions)，存在如下的clusterrolebing。

 

 1.8版本之前.开启rbac后,apiserver默认绑定system:nodes组到system:node的clusterrole。v1.8之后,此绑定默认不存在,需要手工绑定,否则kubelet启动后会报认证错误，使用kubectl get nodes查看无法成为Ready状态。

 

默认角色与默认角色绑定

API Server 会创建一组默认的 ClusterRole 和 ClusterRoleBinding 对象。 这些默认对象中有许多包含 system: 前缀，表明这些资源由 Kubernetes 基础组件”拥有”。 对这些资源的修改可能导致非功能性集群（ non-functional cluster ） 。一个例子是 system:node ClusterRole 对象。这个角色定义了 kubelets 的权限。如果这个角色被修改，可能会导致 kubelets 无法正常工作。
所有默认的ClusterRole 和 ClusterRoleBinding 对象都会被标记为 kubernetes.io/bootstrapping=rbac-defaults 。

使用命令kubectl get clusterrolebinding和kubectl get clusterrole可以查看系统中的角色与角色绑定

 

使用命令kubectl get clusterrolebindings system:node -o yaml或kubectl describe clusterrolebindings system:node查看system:node角色绑定的详细信息：

root@master:~# kubectl describe clusterrolebindings system:node
Name:         system:node
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind  Name  Namespace
  ----  ----  ---------

system:node角色默认绑定为空。

 

 

创建角色绑定

在整个集群中授予 ClusterRole ，包括所有命名空间。

从错误日志中可看出信息：user "system:node:192.168.122.3" groups ["system:nodes" "system:authenticated"]或者从kubelet.kubeconfig查看使用的用户。

在整个集群范围内将 system:node ClusterRole 授予用户”system:node:192.168.122.3”或组”system:nodes”：

root@master:~# kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --user=system:node:192.168.122.3
clusterrolebinding "kubelet-node-clusterbinding" created
root@master:~# kubectl describe clusterrolebindings kubelet-node-clusterbinding
Name:         kubelet-node-clusterbinding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind  Name                       Namespace
  ----  ----                       ---------
  User  system:node:192.168.122.3  
root@master:~#
root@master:~# kubectl delete clusterrolebindings kubelet-node-clusterbinding
clusterrolebinding "kubelet-node-clusterbinding" deleted
root@master:~#
root@master:~# kubectl create clusterrolebinding kubelet-node-clusterbinding --clusterrole=system:node --group=system:nodes
clusterrolebinding "kubelet-node-clusterbinding" created
root@master:~#
root@master:~# kubectl describe clusterrolebindings kubelet-node-clusterbinding
Name:         kubelet-node-clusterbinding
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  system:node
Subjects:
  Kind   Name          Namespace
  ----   ----          ---------
  Group  system:nodes  

此时节点状态变更为Ready

root@master:~# kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
192.168.122.3   Ready     <none>    1h        v1.9.0

查看日志都已经正常