415: Unsupported Media Type:kubeflow - cert-manager 无法创建出来cert证书
kubeflow、cert-manager
现象:kubeflow - cert-manager 无法创建出来cert证书,导致kubeflow相关pod无法正常启动,kubeflow相关pod的describe。
Unable to attach or mount volumes: unmounted volumes=[webhook-cert]
排查发现webhook-cert卷没有被创建出来,由于相关cert没有被cert-manager创建出来
cert-manager具体报错
415: Unsupported Media Type
cert-manager的日志
kubectl logs -f cert-maanger-xxxx -n cert-manager
E1120 02:16:41.022114 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/katib-webhook-cert: 415: Unsupported Media Type" "key"="kubeflow/katib-webhook-cert"
E1120 02:16:41.022145 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/webhook-certs: 415: Unsupported Media Type" "key"="kubeflow/admission-webhook-cert"
E1120 02:16:41.022411 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/webhook-server-tls: 415: Unsupported Media Type" "key"="kubeflow/kfp-cache-cert"
E1120 02:16:41.022492 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/kserve-webhook-server-cert: 415: Unsupported Media Type" "key"="kubeflow/serving-cert"
E1120 02:16:42.025948 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/webhook-certs: 415: Unsupported Media Type" "key"="kubeflow/admission-webhook-cert"
E1120 02:16:42.025957 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/webhook-server-tls: 415: Unsupported Media Type" "key"="kubeflow/kfp-cache-cert"
E1120 02:16:42.025948 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/kserve-webhook-server-cert: 415: Unsupported Media Type" "key"="kubeflow/serving-cert"
E1120 02:16:42.025989 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/katib-webhook-cert: 415: Unsupported Media Type" "key"="kubeflow/katib-webhook-cert"
E1120 02:16:44.028589 1 controller.go:167] cert-manager/certificates-issuing "msg"="re-queuing item due to error processing" "error"="failed to apply secret kubeflow/webhook-certs: 415: Unsupported Media Type" "key"="kubeflow/admission-webhook-cert"
原因和解决方案:
最终排查发现是,kube-apiserver的一个配置导致的。
将kube-apiserver的feature-getes=ServerSideApply=false,改成true,将ServerSideApply打开,重启一下,kubeflow就能正常运行了。