一次kubesphere ingress故障的解决

最新推荐文章于 2024-05-27 16:27:42 发布

scwang18

最新推荐文章于 2024-05-27 16:27:42 发布

阅读量3.5k

点赞数

分类专栏： k8s 文章标签： kubernetes

本文链接：https://blog.csdn.net/scwang18/article/details/124531761

版权

k8s 专栏收录该内容

12 篇文章 1 订阅

订阅专栏

本文记录了在Kubesphere 3.2.1环境下，遇到ingress控制器由于镜像版本不兼容及权限问题导致无法正常工作的问题。解决方案包括升级ingresscontroller镜像至v1.1.0以解决版本兼容性，以及创建并绑定ClusterRole来赋予ingress控制器更新configmap的权限，从而修复ingress不能获取address的错误。

摘要由CSDN通过智能技术生成

一次kubesphere ingress故障的解决

前言

采用单 master 方式安装 Kubesphere 3.2.1 以后, 启动了集群 gateway, 但是 ingress 总是不能正常工作，排查了很久，终于解决问题

问题1: ingress controller 镜像版本过低，启动失败。

kubesphere 3.2.1 默认的 ingress controller 是 kubesphere/nginx-ingress-controller:v0.48.1。我在安装 kubesphere 时，选择安装的 kubernetes 1.23.0, 因此存在兼容性问题。需要修改默认的 nginx ingress controller 版本，查看 docekrhub 上 kubesphere 的最新 nginx ingress controller 版本是 v1.1.0。
通过修改 configmap ks-router-config 中的 nginx ingress controller 的 image 版本号为 v1.1.0，再重启 kubersphere controller，可以解决该问题。
这个地方我走了很多弯路，发现 ks-config 的配置需要在重启 kubesphere controller 才会生效。
kubesphere 采用 helm 方式安装额 nginx ingress controller, 但是该 helm 是包含在 kubesphere controller 镜像中，我尝试修改 helm 包的 values.yaml，替换新的镜像文件，实际上也不能生效，

问题2：ingress 不能正确获取到 address

查看 nginx ingress controller 的 pod 日志，发现是无法完成 ingress controller 选举，提示没有权限更新 ingress-controller-leader-kubesphere-router-kubesphere-system
通过增加可以 update 该 configmap 的 clusterrole, 绑定到 sa kubesphere-router-kubesphere-system, 问题解决。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-controller-configmap-update
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["ingress-controller-leader-kubesphere-router-kubesphere-system"]
  verbs: ["update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubesphere-router-kubesphere-system-update-configmap
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-controller-configmap-update
subjects:
- kind: ServiceAccount
  name: kubesphere-router-kubesphere-system
  namespace: kubesphere-controls-system

排查权限过程如下

################################################################################################################################

[root@ks-master ~]# kcs get clusterrolebindings |grep kubesphere-router-kubesphere-system
kubesphere-router-kubesphere-system                    ClusterRole/kubesphere-router-kubesphere-system                                    6m48s

################################################################################################################################

[root@ks-master ~]# kubectl describe clusterrolebindings kubesphere-router-kubesphere-system
Name:         kubesphere-router-kubesphere-system
Labels:       app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=0.48.1
              helm.sh/chart=ingress-nginx-3.35.0
Annotations:  meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
              meta.helm.sh/release-namespace: kubesphere-controls-system
              operator-sdk/primary-resource: kubesphere-controls-system/kubesphere-router-kubesphere-system-ingress
              operator-sdk/primary-resource-type: Nginx.gateway.kubesphere.io
Role:
  Kind:  ClusterRole
  Name:  kubesphere-router-kubesphere-system
Subjects:
  Kind            Name                                 Namespace
  ----            ----                                 ---------
  ServiceAccount  kubesphere-router-kubesphere-system  kubesphere-controls-system

################################################################################################################################

[root@ks-master ~]# kcs describe clusterrole kubesphere-router-kubesphere-system
Name:         kubesphere-router-kubesphere-system
Labels:       app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=0.48.1
              helm.sh/chart=ingress-nginx-3.35.0
Annotations:  meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
              meta.helm.sh/release-namespace: kubesphere-controls-system
              operator-sdk/primary-resource: kubesphere-controls-system/kubesphere-router-kubesphere-system-ingress
              operator-sdk/primary-resource-type: Nginx.gateway.kubesphere.io
PolicyRule:
  Resources                           Non-Resource URLs  Resource Names  Verbs
  ---------                           -----------------  --------------  -----
  events                              []                 []              [create patch]
  services                            []                 []              [get list watch]
  ingresses.extensions                []                 []              [get list watch]
  ingressclasses.networking.k8s.io    []                 []              [get list watch]
  ingresses.networking.k8s.io         []                 []              [get list watch]
  nodes                               []                 []              [list watch get]
  configmaps                          []                 []              [list watch]
  endpoints                           []                 []              [list watch]
  pods                                []                 []              [list watch]
  secrets                             []                 []              [list watch]
  ingresses.extensions/status         []                 []              [update]
  ingresses.networking.k8s.io/status  []                 []              [update]


################################################################################################################################
################################################################################################################################

[root@ks-master ~]# kcs get rolebindings 
NAME                                  ROLE                                       AGE
kubesphere-router-kubesphere-system   Role/kubesphere-router-kubesphere-system   7m50s
nginx-ingress-role-nisa-binding       Role/system:kubesphere-router-role         11d

################################################################################################################################

[root@ks-master ~]# kcs describe role kubesphere-router-kubesphere-system
Name:         kubesphere-router-kubesphere-system
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=0.48.1
              helm.sh/chart=ingress-nginx-3.35.0
Annotations:  meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
              meta.helm.sh/release-namespace: kubesphere-controls-system
PolicyRule:
  Resources                           Non-Resource URLs  Resource Names                                                         Verbs
  ---------                           -----------------  --------------                                                         -----
  events                              []                 []                                                                     [create patch]
  configmaps                          []                 []                                                                     [get list watch create]
  endpoints                           []                 []                                                                     [get list watch]
  pods                                []                 []                                                                     [get list watch]
  secrets                             []                 []                                                                     [get list watch]
  services                            []                 []                                                                     [get list watch]
  ingresses.extensions                []                 []                                                                     [get list watch]
  ingressclasses.networking.k8s.io    []                 []                                                                     [get list watch]
  ingresses.networking.k8s.io         []                 []                                                                     [get list watch]
  configmaps                          []                 [ingress-controller-leader-kubesphere-router-kubesphere-system-nginx]  [get update]
  namespaces                          []                 []                                                                     [get]
  ingresses.extensions/status         []                 []                                                                     [update]
  ingresses.networking.k8s.io/status  []                 []                                                                     [update]

################################################################################################################################

[root@ks-master ~]# kcs describe role system:kubesphere-router-role
Name:         system:kubesphere-router-role
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  kubernetes.io/created-by: kubesphere.io/ks-router
              meta.helm.sh/release-name: ks-core
              meta.helm.sh/release-namespace: kubesphere-system
PolicyRule:
  Resources   Non-Resource URLs  Resource Names                     Verbs
  ---------   -----------------  --------------                     -----
  configmaps  []                 []                                 [get create]
  configmaps  []                 [ingress-controller-leader-nginx]  [get update]
  endpoints   []                 []                                 [get]
  namespaces  []                 []                                 [get]
  pods        []                 []                                 [get]
  secrets     []                 []                                 [get]