一次kubesphere ingress故障的解决
前言
采用单 master
方式安装 Kubesphere 3.2.1
以后, 启动了集群 gateway
, 但是 ingress
总是不能正常工作,排查了很久,终于解决问题
问题1: ingress controller 镜像版本过低,启动失败。
-
kubesphere 3.2.1
默认的ingress controller
是kubesphere/nginx-ingress-controller:v0.48.1
。我在安装kubesphere
时,选择安装的kubernetes 1.23.0
, 因此存在兼容性问题。需要修改默认的nginx ingress controller
版本,查看docekrhub
上kubesphere
的最新nginx ingress controller
版本是v1.1.0
。 -
通过 修改
configmap
ks-router-config
中的nginx ingress controller
的image
版本号为v1.1.0
,再重启kubersphere controller
,可以解决该问题。 -
这个地方我走了很多弯路,发现
ks-config
的配置需要在重启kubesphere controller
才会生效。 -
kubesphere
采用helm
方式安装额nginx ingress controller
, 但是该helm
是包含在kubesphere controller
镜像中,我尝试修改helm
包的values.yaml
,替换新的镜像文件,实际上也不能生效,
问题2:ingress 不能正确获取到 address
- 查看
nginx ingress controller
的pod
日志,发现是无法完成ingress controller
选举,提示没有权限更新ingress-controller-leader-kubesphere-router-kubesphere-system
- 通过增加 可以
update
该configmap
的clusterrole
, 绑定到sa
kubesphere-router-kubesphere-system
, 问题解决。
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ingress-controller-configmap-update
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["ingress-controller-leader-kubesphere-router-kubesphere-system"]
verbs: ["update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubesphere-router-kubesphere-system-update-configmap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-controller-configmap-update
subjects:
- kind: ServiceAccount
name: kubesphere-router-kubesphere-system
namespace: kubesphere-controls-system
排查权限过程如下
################################################################################################################################
[root@ks-master ~]# kcs get clusterrolebindings |grep kubesphere-router-kubesphere-system
kubesphere-router-kubesphere-system ClusterRole/kubesphere-router-kubesphere-system 6m48s
################################################################################################################################
[root@ks-master ~]# kubectl describe clusterrolebindings kubesphere-router-kubesphere-system
Name: kubesphere-router-kubesphere-system
Labels: app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.48.1
helm.sh/chart=ingress-nginx-3.35.0
Annotations: meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
meta.helm.sh/release-namespace: kubesphere-controls-system
operator-sdk/primary-resource: kubesphere-controls-system/kubesphere-router-kubesphere-system-ingress
operator-sdk/primary-resource-type: Nginx.gateway.kubesphere.io
Role:
Kind: ClusterRole
Name: kubesphere-router-kubesphere-system
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount kubesphere-router-kubesphere-system kubesphere-controls-system
################################################################################################################################
[root@ks-master ~]# kcs describe clusterrole kubesphere-router-kubesphere-system
Name: kubesphere-router-kubesphere-system
Labels: app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.48.1
helm.sh/chart=ingress-nginx-3.35.0
Annotations: meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
meta.helm.sh/release-namespace: kubesphere-controls-system
operator-sdk/primary-resource: kubesphere-controls-system/kubesphere-router-kubesphere-system-ingress
operator-sdk/primary-resource-type: Nginx.gateway.kubesphere.io
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch]
services [] [] [get list watch]
ingresses.extensions [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
nodes [] [] [list watch get]
configmaps [] [] [list watch]
endpoints [] [] [list watch]
pods [] [] [list watch]
secrets [] [] [list watch]
ingresses.extensions/status [] [] [update]
ingresses.networking.k8s.io/status [] [] [update]
################################################################################################################################
################################################################################################################################
[root@ks-master ~]# kcs get rolebindings
NAME ROLE AGE
kubesphere-router-kubesphere-system Role/kubesphere-router-kubesphere-system 7m50s
nginx-ingress-role-nisa-binding Role/system:kubesphere-router-role 11d
################################################################################################################################
[root@ks-master ~]# kcs describe role kubesphere-router-kubesphere-system
Name: kubesphere-router-kubesphere-system
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=kubesphere-router-kubesphere-system-ingress
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=0.48.1
helm.sh/chart=ingress-nginx-3.35.0
Annotations: meta.helm.sh/release-name: kubesphere-router-kubesphere-system-ingress
meta.helm.sh/release-namespace: kubesphere-controls-system
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch]
configmaps [] [] [get list watch create]
endpoints [] [] [get list watch]
pods [] [] [get list watch]
secrets [] [] [get list watch]
services [] [] [get list watch]
ingresses.extensions [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
configmaps [] [ingress-controller-leader-kubesphere-router-kubesphere-system-nginx] [get update]
namespaces [] [] [get]
ingresses.extensions/status [] [] [update]
ingresses.networking.k8s.io/status [] [] [update]
################################################################################################################################
[root@ks-master ~]# kcs describe role system:kubesphere-router-role
Name: system:kubesphere-router-role
Labels: app.kubernetes.io/managed-by=Helm
Annotations: kubernetes.io/created-by: kubesphere.io/ks-router
meta.helm.sh/release-name: ks-core
meta.helm.sh/release-namespace: kubesphere-system
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
configmaps [] [] [get create]
configmaps [] [ingress-controller-leader-nginx] [get update]
endpoints [] [] [get]
namespaces [] [] [get]
pods [] [] [get]
secrets [] [] [get]