使用kube-state-metrics监控kubernetes并告警

最新推荐文章于 2024-08-20 23:54:31 发布

水中加点糖

最新推荐文章于 2024-08-20 23:54:31 发布

阅读量3.3k

点赞数

分类专栏： kubernetes 文章标签： kubernetes promethues grafana 告警监控

本文链接：https://blog.csdn.net/puhaiyang/article/details/115285128

版权

kubernetes 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

对于自己搭建的kubernetes集群，能做到实时监控pod的状态并及时发送告警是非常有必要的

通过翻阅kubernetes官方的各种文档后，我决定采用kube-state-metrics来实现对kubenetes进行监控，并配合promethues进行数据存储、grafana进行数据渲染与告警，从而实现一套比较完整的监控方案

安装过程如下：

数据收集

部署kube-state-metrics

将kube-state-metrics组件的yaml克隆下来

git clone https://github.com/devopscube/kube-state-metrics-configs.git

clone成功后会有如下几个文件：

root@k8s-master:~/kube-state-metrics/kube-state-metrics-configs# ll
total 36
drwxr-xr-x 3 root root 4096 Mar 27 10:06 ./
drwxr-xr-x 3 root root 4096 Mar 27 10:06 ../
-rw-r--r-- 1 root root  377 Mar 27 10:06 cluster-role-binding.yaml
-rw-r--r-- 1 root root 1651 Mar 27 10:06 cluster-role.yaml
-rw-r--r-- 1 root root 1069 Mar 27 10:06 deployment.yaml
drwxr-xr-x 8 root root 4096 Mar 27 10:06 .git/
-rw-r--r-- 1 root root   78 Mar 27 10:06 README.md
-rw-r--r-- 1 root root  193 Mar 27 10:06 service-account.yaml
-rw-r--r-- 1 root root  406 Mar 27 10:06 service.yaml

默认deployment.yaml是没有指定时区的，为了方便观察kubernetes中的数据时区再配置一下，指定为东八区的时间

        env:
          - name: TZ
            value: Asia/Shanghai

完整的deployment.yaml文件如下:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: v1.8.0
  name: kube-state-metrics
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.8.0
    spec:
      containers:
      - image: quay.io/coreos/kube-state-metrics:v1.8.0
        env:
          - name: TZ
            value: Asia/Shanghai
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          name: http-metrics
        - containerPort: 8081
          name: telemetry
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: kube-state-metrics

修改好后到clone的目录apply一下即可:

kubectl apply -f kube-state-metrics-configs/

如下:

root@k8s-master:~/kube-state-metrics# kubectl apply -f kube-state-metrics-configs/
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
serviceaccount/kube-state-metrics created
service/kube-state-metrics created

如果执行的是上面的命令的话，默认会在kube-system命名空间下创建对应的资源，查看kube-state-metrics是否部署成功，执行get命令查看一下

kubectl get deployments kube-state-metrics -n kube-system

如下：

root@k8s-master:~/kube-state-metrics# kubectl get deployments kube-state-metrics -n kube-system
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
kube-state-metrics   1/1     1            1           4m49s

全ready好了就说明已部署成功了

再看一下kube-state-metrics的日志:

如下图

在这里插入图片描述

数据展示

kube-state-metrics关联promethues

对于promethues与grafana的部署这里不再演示，感兴趣的可在上一篇文章《介绍一下,spring cloud下的另一种定时任务解决方案》中找到，或直接执行下面的两个地址的部署文件

https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/src/kubernetes/prometheus
https://github.com/spring-cloud/spring-cloud-dataflow/tree/master/src/kubernetes/grafana

当promethues准备好后，添加对应的采集job即可:

    - job_name: 'kube-state-metrics'
      static_configs:
        - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

如果是用的上面spring cloud data flow官方的promethues来采集数据的话，配置在promethues对应的configMap里即可

其prometheus-configmap.yaml文件内容如下：

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
  labels:
    app: prometheus
data:
  prometheus.yml: |-
    global:
      scrape_interval: 10s
      scrape_timeout: 9s
      evaluation_interval: 10s

    scrape_configs:
    - job_name: 'proxied-applications'
      metrics_path: '/metrics/connected'
      kubernetes_sd_configs:
        - role: pod
          namespaces:
            names:
              - default
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: prometheus-proxy
        - source_labels: [__meta_kubernetes_pod_container_port_number]
          action: keep
          regex: 8080
    - job_name: 'kube-state-metrics'
      static_configs:
        - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
    - job_name: 'proxies'
      metrics_path: '/metrics/proxy'
      kubernetes_sd_configs:
        - role: pod
          namespaces:
            names:
              - default
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: prometheus-proxy
        - source_labels: [__meta_kubernetes_pod_container_port_number]
          action: keep
          regex: 8080
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

之后再重启promethues让配置生效,再打开promethues的web看一下target里的配置是否生效，如下图

在这里插入图片描述