k8s环境中搭建prometheus监控平台及自动发现集群中的节点

k8s环境中搭建prometheus监控平台及自动发现集群中的节点

本文在已安装完成k8s环境，且使用了docker作为k8s的运行时。在以上的基础上搭建了prometheus，若按照本文搭建prometheus需要完成以上前提。

node_exporter

node_exporter是prometheus指标收集器，用于监控主机系统。

配置

新建配置文件node_exporter_k8s.yml

apiVersion: apps/v1
# 配置DaemonSet方式运行,当有新的k8s节点加入集群时就会自动创建及启动node_exporter容器
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    app: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      #我这里使用的是测试环境,为了方便测试使用了主机地址,如若不需要可以修改走k8s网络
      hostNetwork: true
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: node-exporter
        image: quay.io/prometheus/node-exporter:latest
        #因为是以启动容器的方式进行主机监控,所以需要指定path.rootfs参数,值必须要与主机根目录的路径进行绑定
        args:
        - --path.rootfs=/host
        volumeMounts:
        - name: rootfs
          mountPath: /host
      #映射主机根目录
      volumes:
      - name: rootfs
        hostPath:
          path: /

应用配置及启动

kubectl apply -f node_exporter_k8s.yml

prometheus

配置

新建配置文件prometheus.yml，这段配置用于配置prometheus能够自动发现node，并且抓取数据

# prometheus.yml
global:
  scrape_interval: 15s  # 默认抓取间隔

scrape_configs:
  - job_name: 'node-exporter'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - kube-system  # 指定命名空间，例如 kube-system
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: node-exporter
      - source_labels: [__meta_kubernetes_pod_ip]
        action: replace
        target_label: __address__
        regex: (.*)
        replacement: $1:9100  # 这里指定 node-exporter 的端口
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: instance

创建k8s Deployment

配置

新建配置文件prometheus_k8s.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: kube-system
  labels:
    app: prometheus
spec:
  selector:
    matchLabels:
      name: prometheus
  template:
    metadata:
      labels:
        name: prometheus
    spec:
      hostNetwork: true
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      containers:
      - name: prometheus
        image: prom/prometheus
        volumeMounts:
        - name: config
          mountPath: /etc/prometheus/prometheus.yml
      volumes:
      - name: config
        hostPath:
          #这个路径是上一步操作中创建的配置文件
          path: /k8s/prometheus/config/prometheus.yml

应用配置及启动

kubectl apply -f prometheus_k8s.yml

成功后访问主机9090端口即可访问prometheus界面

问题

如果启动prometheus出现以下错误

caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list resource \"pods\" in API group \"\" at the cluster scope"

则是因为权限问题导致，具体解决方法如下

1. 创建 ClusterRole

创建一个新的 ClusterRole，赋予它列出、获取和监视 Pods 的权限。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
    - pods
    - nodes
    - services
    - endpoints
  verbs:
    - get
    - list
    - watch
- apiGroups: [""]
  resources:
    - configmaps
  verbs:
    - get
- nonResourceURLs: ["/metrics"]
  verbs:
    - get

2. 创建 ClusterRoleBinding

创建一个 ClusterRoleBinding，将上述 ClusterRole 绑定到 Prometheus 使用的 ServiceAccount 上。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system

3. 应用 RBAC 配置

将上述两个 YAML 文件应用到 Kubernetes 集群中：

kubectl apply -f prometheus-clusterrole.yaml
kubectl apply -f prometheus-clusterrolebinding.yaml

4. 重新启动 Prometheus

kubectl rollout restart deployment prometheus -n <prometheus-namespace>