kubernetes 从v1.7.6升级到v1.11.2版本--kube-apiserver错误分析

错误日志

[root@master-47-35 ~]# kubectl logs -f kube-apiserver-master-47-34  -n kube-system 
...
E0912 06:36:05.215312       1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: OpenAPI spec does not exists
...

查原因

查看api-versions

[root@master-47-35 ~]# kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
custom-metrics.metrics.k8s.io/v1alpha1
events.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

虽然有创建这个custom-metrics.metrics.k8s.io/v1alpha1，但是在v1.11.1,已经不支持这个版本了kubernetes release-1.11，在v1.7.6版本支持kubernetes release-1.7
对比以下两幅图，就可以看出了，在k8s v1.11.x已经
custom-metrics.metrics.k8s.io/v1alpha1
改为
custom-metrics.metrics.k8s.io/v2beta1
以及custom-metrics.metrics.k8s.io/v2beta2
所以就会报错

解决办法

删除v1alpha1.custom-metrics.metrics.k8s.ioapi组

vi v1alpha1.custom-metrics.metrics.k8s.io.yaml

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1alpha1.custom-metrics.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: custom-metrics.metrics.k8s.io
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: api
    namespace: custom-metrics
  version: v1alpha1

kubectl delete -f v1alpha1.custom-metrics.metrics.k8s.io.yaml


apiservice.apiregistration.k8s.io "v1alpha1.custom-metrics.metrics.k8s.io" deleted

查看api-versions

[root@master-47-35 custom-metrics]# kubectl api-versions
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
apps/v1beta1
apps/v1beta2
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
batch/v1
batch/v1beta1
certificates.k8s.io/v1beta1
events.k8s.io/v1beta1
extensions/v1beta1
monitoring.coreos.com/v1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1beta1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

删除成功，验证
重启kube-apiserver

[root@master-47-35 custom-metrics]# docker ps | grep api
b5ce2cb0371f        harbor.enncloud.cn/enncloud/hyperkube-amd64@sha256:2466e6219aba9d88dcef04c0f5578a6df5e9cc0d9f1913357585bc71436b4237   "/apiserver --enab..."   About a minute ago   Up About a minute                       k8s_kube-apiserver_kube-apiserver-master-47-35_kube-system_497141a7e067956aaaa0199ce88060ed_2
dee06861bece        harbor.enncloud.cn/paas/pause-amd64:3.1                                                                               "/pause"                 24 hours ago         Up 24 hours                             k8s_POD_kube-apiserver-master-47-35_kube-system_497141a7e067956aaaa0199ce88060ed_2
[root@master-47-35 custom-metrics]# docker logs -f b5ce2cb0371f
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
[restful] 2018/09/12 07:29:35 log.go:33: [restful/swagger] listing is available at https://10.39.47.35:6443/swaggerapi
[restful] 2018/09/12 07:29:35 log.go:33: [restful/swagger] https://10.39.47.35:6443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/09/12 07:29:37 log.go:33: [restful/swagger] listing is available at https://10.39.47.35:6443/swaggerapi
[restful] 2018/09/12 07:29:37 log.go:33: [restful/swagger] https://10.39.47.35:6443/swaggerui/ is mapped to folder /swagger-ui/
E0912 07:29:41.072852       1 available_controller.go:311] v1.monitoring.coreos.com failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1.monitoring.coreos.com": StorageError: invalid object, Code: 4, Key: /registry/apiregistration.k8s.io/apiservices/v1.monitoring.coreos.com, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 6e717648-b65d-11e8-89a3-5254e98192ae, UID in object meta: 
E0912 07:29:41.119161       1 autoregister_controller.go:190] v1.monitoring.coreos.com failed with : apiservices.apiregistration.k8s.io "v1.monitoring.coreos.com" already exists
E0912 07:29:44.605439       1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0912 07:30:44.605888       1 controller.go:111] loading OpenAPI spec for "v1alpha1.custom-metrics.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[X-Content-Type-Options:[nosniff] Content-Type:[text/plain; charset=utf-8]]

还有报错 但是不影响使用