k8s v1.27.4 部署metrics-serverv:0.6.4，kube-prometheus，镜像下载问题处理

已于 2023-09-15 10:49:43 修改
阅读量2.1k 收藏 4
点赞数 1
分类专栏： Kubernetes 文章标签： kubernetes 容器 云原生
于 2023-08-18 15:43:43 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_39438868/article/details/132363171
版权

只有一个问题，原来的httpGet存活、就绪检测一直不通过，于是改为tcpSocket后pod正常。

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

修改后的yaml文件，镜像修改为阿里云

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
#        - --cert-dir=/tmp
#        - --secure-port=4443
#        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
#        - --kubelet-use-node-status-port
#        - --metric-resolution=15s
        - --cert-dir=/tmp
        - --secure-port=4443
        - --metric-resolution=30s
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --requestheader-username-headers=X-Remote-User
        - --requestheader-group-headers=X-Remote-Group
        - --requestheader-extra-headers-prefix=X-Remote-Extra-
        image: registry.aliyuncs.com/google_containers/metrics-server:v0.6.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          tcpSocket:
            port: 4443
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          tcpSocket:
            port: 4443
          initialDelaySeconds: 20
          periodSeconds: 10
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

kubectl apply -f components.yaml

在这里插入图片描述

在这里插入图片描述
部署kube-prometheus
兼容1.27的为main分支
在这里插入图片描述
只克隆main分支

git clone  --single-branch --branch main https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus

如果直接运行kubectl apply -f manifests/setup/，会有元数据注解太长的错误
Error from server (Invalid): error when creating "manifests/setup/0prometheusCustomResourceDefinition.yaml": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

应该使用以下方式
1. kubectl apply -f manifests/setup/
2. kubectl apply --server-side -f manifests/setup 

CRD正常创建后然后再执行:
kubectl apply -f manifests

镜像下载使用这个项目的方法:https://github.com/DaoCloud/public-image-mirror,在镜像前面加：m.daocloud.io

prometheus-k8s-0 pod报权限不足问题

ts=2023-08-21T03:14:34.582Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"

处理：
修改prometheus-clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.26.0
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  verbs:
  - get

跨节点pind不通monitoring下的pod ip，prometheus target异常问题处理，

删除kube-prometheus/manifests下的netowrkprolic
alertmanager-networkPolicy.yaml
blackboxExporter-networkPolicy.yaml
grafana-networkPolicy.yaml
kubeStateMetrics-networkPolicy.yaml
nodeExporter-networkPolicy.yaml
prometheusAdapter-networkPolicy.yaml
prometheus-networkPolicy.yaml
prometheusOperator-networkPolicy.yaml

使用ServiceMonitor添加监控：
以ingress-nginx为例
修改ingress-nginx.yaml的service

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: '10254' #新增内容
    prometheus.io/scrape: 'true' #新增内容
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.8.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  - appProtocol: http  #新增内容
    name: prometheus  #新增内容
    port: 10254  #新增内容
    protocol: TCP  #新增内容
    targetPort: 10254  #新增内容
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
  

# ingress-nginx-monitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ingress-nginx  # ServiceMonitor名称
  namespace: monitoring # ServiceMonitor所在名称空间
spec:
  endpoints:  # prometheus所采集Metrics地址配置，endpoints为一个数组，可以创建多个，但是每个endpoints包含三个字段interval、path、port
    - interval: 15s # prometheus采集数据的周期，单位为秒
      path: /metrics # prometheus采集数据的路径
      port: prometheus # prometheus采集数据的端口，这里为port的name，主要是通过spec.selector中选择对应的svc，在选中的svc中匹配该端口
  namespaceSelector: # 需要发现svc的范围
    any: true     # 有且仅有一个值true，当该字段被设置时，表示监听所有符合selector所选择的svc
  selector:
    matchLabels:  # 选择svc的标签
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
      app.kubernetes.io/version: 1.8.1
      
kubectl apply -f ingress-nginx-monitor.yaml
确认ingress-nginx servicemonitors已添加
kubectl -n monitoring get servicemonitors.monitoring.coreos.com 
NAME                      AGE
alertmanager-main         70m
blackbox-exporter         70m
coredns                   70m
grafana                   70m
ingress-nginx             24m
kube-apiserver            70m
kube-controller-manager   70m
kube-scheduler            70m
kube-state-metrics        70m
kubelet                   70m
node-exporter             70m
prometheus-adapter        70m
prometheus-k8s            70m
prometheus-operator       70m

在这里插入图片描述
监控外部ETCD：
etcd配置文件添加：ETCD_LISTEN_METRICS_URLS=“http://0.0.0.0:2381”
etcd-job.yaml

- job_name: "etcd-server"
  scrape_interval: 30s
  scrape_timeout: 10s
  static_configs:
    - targets: ["172.16.0.157:2381"]
      labels:
        instance: etcdserver

创建secret

kubectl create secret generic etcd-secret --from-file=etcd-job.yaml -n monitoring

kube-prometheus/manifests/prometheus-prometheus.yaml 末尾添加配置：

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.46.0
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - apiVersion: v2
      name: alertmanager-main
      namespace: monitoring
      port: web
  enableFeatures: []
  externalLabels: {}
  image: quay.io/prometheus/prometheus:v2.46.0
  nodeSelector:
    kubernetes.io/os: linux
  podMetadata:
    labels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
      app.kubernetes.io/version: 2.46.0
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleNamespaceSelector: {}
  ruleSelector: {}
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: 2.46.0
  additionalScrapeConfigs:  #新增
    name: etcd-secret  #新增
    key: etcd-job.yaml  #新增

kubectl apply -f kube-prometheus/manifests/prometheus-prometheus.yaml

grafana导入 3070模板
在这里插入图片描述
http存活检测

kind: Probe
apiVersion: monitoring.coreos.com/v1
metadata:
  name: example-com-website
  namespace: monitoring
spec:
  interval: 60s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:19115
  targets:
    staticConfig:
      static:
      - https://www.baidu.com

在这里插入图片描述
在这里插入图片描述
添加proxy监控，默认为http

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: kube-proxy
    app.kubernetes.io/part-of: kube-prometheus
  name: kube-proxy
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: http-metrics
    scheme: http
    tlsConfig:
      insecureSkipVerify: true
  jobLabel: app.kubernetes.io/name
  namespaceSelector:
    matchNames:
    - kube-system
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-proxy
---
apiVersion: v1
kind: Endpoints
metadata:
  name: kube-proxy
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-proxy
subsets:
- addresses:
  - ip: 172.16.0.157
    targetRef:
      kind: Node
      name: master1
  - ip: 172.16.0.124
    targetRef:
      kind: Node
      name: node1
  - ip: 172.16.0.46
    targetRef:
      kind: Node
      name: node2
  ports:
  - name: http-metrics
    port: 10249
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: kube-proxy
  namespace: kube-system
  labels:
    app.kubernetes.io/name: kube-proxy
spec:
  type: ClusterIP 
  clusterIP: None
  ports:
  - name: http-metrics
    port: 10249
    targetPort: 10249
    protocol: TCP

在这里插入图片描述

x10n9
关注
专栏目录
Metrics Server 配置和安全注意事项
firmament0422的博客
10-26 315
Metrics Server 配置
metrics-server v0.6.1镜像包和安装文件
03-07
metrics-server v0.6.1镜像包和安装文件
metrics-server_v0.7.0 docker 镜像下载
最新发布
qq_31380685的博客
08-16 138
共享资源
docker离线镜像 metrics-server.tar
04-26
kubernetes性能监控工具metrics-server离线镜像， k8s.gcr.io/metrics-server-amd64:v0.3.6 ，使用方法：docker load -i metrics-server.tar
metrics-server v0.5.2镜像包和安装文件
01-27
metrics-server v0.5.2镜像包和安装文件
K8S部署metrics-server-v0.6.0资源清单及镜像文件
01-24
原文链接：https://blog.csdn.net/m0_37814112/article/details/122666995 说明： 1、支持支持的Kubernetes版本1.19+ 2、包含yaml文件和镜像，直接apply即可部署完成
k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0镜像
04-15
docker load -i kube-state-metrics-v2.3.0.tar.gz 上传
k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.4.2镜像
05-22
docker load -i kube-state-metrics-v2.4.2.tar.gz docker images|grep k8s.gcr.io/kube-state-metrics/kube-state-metrics
k8s-gcr-io-metrics-server-metrics-server-v0.4.1.rar
12-14
k8s.gcr.io/metrics-server/metrics-server:v0.4.1 components.yaml 博客地址：https://blog.csdn.net/shenhonglei1234/category_7260434.html
k8s.gcr.io/metrics-server/metrics-server:v0.4.3
03-24
kubectl get hpa targets 显示unknown 需要的镜像
k8s镜像文件
04-12
k8s.gcr.io的相关images,版本v1.14.1使用说明在里面··
k8s-v1.17.0镜像包
08-06
为了节省下载镜像的时间，我这里已经给大家都下载好了。kubeadm安装所需要镜像包，版本：v1.17.0。亲测可用
kubernetes-k8s v1.21.2版本镜像包
11-01
IT运维、k8s、kubernetes搭建(包含kube-apiserver-v1.21.2、kube-controller-manager-v1.21.2、kube-scheduler-v1.21.2、coredns、pause、etcd、calico。。。。）
k8s 组件镜像
12-13
kube-apiserver-amd64_v1.7.2 kube-controller-manager-amd64_v1.7.2 kube-scheduler-amd64_v1.7.2 k8s-dns-dnsmasq-nanny-amd64_1.14.4 k8s-dns-kube-dns-amd64_1.14.4 k8s-dns-sidecar-amd64_1.14.4
k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.5.0
12-23
docker save -o kube-state-metrics:v2.5.0.tar.gz k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.5.0 docker load -i kube-state-metrics:v2.5.0.tar.gz
k8s镜像下载，docker海外镜像下载
ysj的博客
04-19 3525
利用阿里云镜像仓库 创建镜像仓库 绑定代码仓库，并设置海外构建 3.创建完成后，即可自动拉取git仓库代码，依据git仓库一级目录下Dockerfile文件自动构建镜像 方法简介。 内置规则中，只需要 tag 名字为 release-v$version ，即构建 tag 为 $version 的镜像出来。 4自动拉取脚本 参考github： github地址 # 1 方法1 bash...
k8s安装metrics server
XXisCC的博客
08-06 678
我的k8s版本是1.23.1，所以可以使用metrics-server的版本v0.6.4。2.打开刚刚下载的yaml文件。
k8s.gcr.io、gcr.io仓库的镜像下载
m0_60028455的博客
12-09 1505
镜像下载、域名解析、时间同步请点击阿里巴巴开源镜像站 获取这类镜像的方法一般有2种： 1.通过拉取国内镜像仓库的内容（操作简单直接拉取即可，缺点是镜像的版本更新可能较慢，可能无法获取最新的镜像） 2.通过git+镜像服务拉取镜像（操作略繁琐，但是可以拉取到最新的镜像） 本篇记录的是方法2，即通过git+镜像服务获取镜像（git保存Dockerfile，镜像服务根据Dockerfile构建）。 考虑到既然是用docker的技术人员，git、镜像服务的操作都是基础，略微繁琐的方案能获取最新的镜像服务是我所..
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.9.2
03-14
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.9.2 是一个容器镜像的名称，用于部署和运行 kube-state-metrics 组件。kube-state-metrics 是一个 Kubernetes 集群监控工具，它通过暴露 Kubernetes 集群...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值