Kubernetes prometheus services is forbidden: User \“system:serviceaccount:monitoring:prometheus-k8s\

ts=2024-05-15T03:14:22.131Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg=“pkg/mod/k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169: failed to list *v1.Pod: pods is forbidden: User “system:serviceaccount:monitoring:prometheus-k8s” cannot list resource “pods” in API group “” in the namespace “middle””
ts=2024-05-15T03:14:22.132Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg=“pkg/mod/k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User “system:serviceaccount:monitoring:prometheus-k8s” cannot list resource “pods” in API group “” in the namespace “middle””
ts=2024-05-15T03:14:22.273Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg=“pkg/mod/k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169: failed to list *v1.Service: services is forbidden: User “system:serviceaccount:monitoring:prometheus-k8s” cannot list resource “services” in API group “” in the namespace “middle””
ts=2024-05-15T03:14:22.274Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg=“pkg/mod/k8s.io/client-go@v0.26.2/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User “system:serviceaccount:monitoring:prometheus-k8s” cannot list resource “services” in API group “” in the namespace “middle””

如果k8s部署服务，开启了/metrics：
metrics:
enabled: true
image:
registry: harbor.zdwk8s.com
repository: middle/redis-exporter
tag: 1.50.0-debian-11-r13
digest: “”
pullPolicy: IfNotPresent
pullSecrets: []
startupProbe:
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
livenessProbe:
enabled: true
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
enabled: true
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 1
successThreshold: 1
failureThreshold: 3
customStartupProbe: {}
customLivenessProbe: {}
customReadinessProbe: {}
command: []
redisTargetHost: “localhost”
extraArgs: {}
extraEnvVars: []
containerSecurityContext:
enabled: true
runAsUser: 1001
extraVolumes: []
extraVolumeMounts: []
resources:
limits: {}
requests: {}
podLabels: {}
podAnnotations:
prometheus.io/scrape: “true”
prometheus.io/port: “9121”
service:
type: ClusterIP
port: 9121
externalTrafficPolicy: Cluster
extraPorts: []
loadBalancerIP: “”
loadBalancerSourceRanges: []
annotations: {}
clusterIP: “”
serviceMonitor:
enabled: true
namespace: “monitoring”
interval: 30s
scrapeTimeout: “”
relabellings: []
metricRelabelings: []
honorLabels: false
additionalLabels: {}
podTargetLabels: []
prometheusRule:
enabled: true
namespace: “monitoring”
additionalLabels: {}
rules: []

先确认可以获取数据：
[root@china ~]# kubectl get svc -n middle
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redis-headless ClusterIP None 6379/TCP 81m
redis-master NodePort 10.227.93.201 6379:32379/TCP 81m
redis-metrics ClusterIP 10.227.183.204 9121/TCP 81m

[root@china ~]# curl 10.227.183.204:9121/metrics
…
go_gc_duration_seconds{quantile=“0”} 3.9832e-05
go_gc_duration_seconds{quantile=“0.25”} 0.000139377
go_gc_duration_seconds{quantile=“0.5”} 0.000236065
go_gc_duration_seconds{quantile=“0.75”} 0.000308893
go_gc_duration_seconds{quantile=“1”} 0.001813312
go_gc_duration_seconds_sum 0.01587861
go_gc_duration_seconds_count 61

但是prometheus无法获取数据，看日志：
kubectl logs prometheus-k8s-0 -n monitoring --tail 200

报错误：forbidden: User "system:serviceaccount:monitoring

在部署prometheus的文件中添加权限，在/kube-prometheus/manifests/prometheus-clusterRole.yaml

添加：
[root@china manifests]# cat prometheus-clusterRole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.44.0
name: prometheus-k8s
rules:

• apiGroups:
  • “”
    resources:
  • nodes
  • services
  • endpoints
  • pods
  • nodes/proxy
  • nodes/metrics
    verbs:
  • get
  • list
  • watch
• nonResourceURLs:
  • /metrics
    verbs:
  • get
    [root@china manifests]#

主要添加：

重新生效：
[root@china manifests]# kubectl apply -f prometheus-clusterRole.yaml