the server is currently unable to handle the request (get pods.metrics.k8s.io)

最新推荐文章于 2024-07-29 10:13:00 发布

爱码后

最新推荐文章于 2024-07-29 10:13:00 发布

阅读量6.4k

点赞数 3

分类专栏： Kubernetes 文章标签： kubernetes

本文链接：https://blog.csdn.net/zhuyongru/article/details/114069775

版权

Kubernetes 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

出现问题：

[root@k97 kube-prometheus]# kubectl top pods -n monitoring
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

分析问题：

[root@k97 kube-prometheus]# systemctl status kube-apiserver.service -l
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since 三 2021-02-24 13:28:04 CST; 21h ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 24783 (kube-apiserver)
    Tasks: 22
   Memory: 742.9M
   CGroup: /system.slice/kube-apiserver.service
           └─24783 /opt/k8s/bin/kube-apiserver --advertise-address=172.16.160.97 --default-not-ready-toleration-seconds=360 --default-unreachable-toleration-seconds=360 --feature-gates=DynamicAuditing=true --max-mutating-requests-inflight=2000 --max-requests-inflight=4000 --default-watch-cache-size=200 --delete-collection-workers=2 --encryption-provider-config=/etc/kubernetes/encryption-config.yaml --etcd-cafile=/etc/kubernetes/cert/ca.pem --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem --etcd-servers=https://172.16.160.97:2379,https://172.16.160.98:2379,https://172.16.160.99:2379 --bind-address=172.16.160.97 --secure-port=6443 --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem --insecure-port=0 --audit-dynamic-configuration --audit-log-maxage=15 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-truncate-enabled --audit-log-path=/data/k8s/k8s/kube-apiserver/audit.log --audit-policy-file=/etc/kubernetes/audit-policy.yaml --profiling --anonymous-auth=false --client-ca-file=/etc/kubernetes/cert/ca.pem --enable-bootstrap-token-auth --requestheader-allowed-names="aggregator" --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem --requestheader-extra-headers-prefix="X-Remote-Extra-" --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --service-account-key-file=/etc/kubernetes/cert/ca.pem --authorization-mode=Node,RBAC --runtime-config=api/all=true --enable-admission-plugins=NodeRestriction --allow-privileged=true --apiserver-count=3 --event-ttl=168h --kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem --kubelet-https=true --kubelet-timeout=10s --proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem --proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem --service-cluster-ip-range=10.254.0.0/16 --service-node-port-range=30000-32767 --logtostderr=true --v=2

2月 25 11:20:20 k97.mayocase.com kube-apiserver[24783]: E0225 11:20:20.780208   24783 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: Get https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.254.25.154:443: connect: connection refused
2月 25 11:20:20 k97.mayocase.com kube-apiserver[24783]: E0225 11:20:20.941013   24783 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: Get https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.254.25.154:443: connect: connection refused
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: E0225 11:20:21.261890   24783 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: Get https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.254.25.154:443: connect: connection refused
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: I0225 11:20:21.611511   24783 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: W0225 11:20:21.611568   24783 handler_proxy.go:99] no RequestInfo found in the context
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: E0225 11:20:21.611595   24783 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: I0225 11:20:21.611602   24783 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2月 25 11:20:21 k97.mayocase.com kube-apiserver[24783]: E0225 11:20:21.902823   24783 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: Get https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.254.25.154:443: connect: connection refused
2月 25 11:20:23 k97.mayocase.com kube-apiserver[24783]: E0225 11:20:23.183771   24783 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: Get https://10.254.25.154:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.254.25.154:443: connect: connection refused
[root@k97 kube-prometheus]#

划重点：

loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable

[root@k97 kube-prometheus]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  creationTimestamp: "2021-02-25T03:11:04Z"
  labels:
    app.kubernetes.io/component: metrics-adapter
    app.kubernetes.io/name: prometheus-adapter
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.8.3
  name: v1beta1.metrics.k8s.io
  resourceVersion: "130415"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: ed613a1d-f3bb-4a88-a65f-7be1e5884c2c
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: prometheus-adapter
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2021-02-25T03:25:58Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available
[root@k97 kube-prometheus]#

解决问题：

我什么都没有做，只是抽根烟，等一会，等下就正常了。

[root@k97 kube-prometheus]# kubectl top pods -n monitoring
NAME                                   CPU(cores)   MEMORY(bytes)   
alertmanager-main-0                    2m           31Mi            
blackbox-exporter-65bb654645-5ggjz     0m           22Mi            
kube-state-metrics-5c7599595f-gcbnn    0m           50Mi            
node-exporter-8lczk                    2m           26Mi            
prometheus-k8s-0                       18m          230Mi           
prometheus-operator-7b8945c8dd-vfhgw   0m           51Mi            
[root@k97 kube-prometheus]#

其它方法：

kubectl delete apiservice v1beta1.metrics.k8s.io