Kubernetes精讲之prometheus

目录

一 Prometheus简介

1.1 Prometheus架构

二 在k8s中部署Prometheus

2.1 下载部署Prometheus所需资源

2.3 登陆grafana

2.4 导入面板

三 监控使用示例

3.1 建立监控项目


一 Prometheus简介

Prometheus是一个开源的服务监控系统和时序数据库

其提供了通用的数据模型和快捷数据采集、存储和查询接口

它的核心组件Prometheus服务器定期从静态配置的监控目标或者基于服务发现自动配置的目标中进行拉取数据

新拉取到啊的 数据大于配置的内存缓存区时,数据就会持久化到存储设备当中

1.1 Prometheus架构

1.1.1 组件功能:

  • 监控代理程序:如node_exporter:收集主机的指标数据,如平均负载、CPU、内存、磁盘、网络等等多个维度的指标数据。
  • kubelet(cAdvisor):收集容器指标数据,也是K8S的核心指标收集,每个容器的相关指标数据包括:CPU使用率、限额、文件系统读写限额、内存使用率和限额、网络报文发送、接收、丢弃速率等等。
  • API Server:收集API Server的性能指标数据,包括控制队列的性能、请求速率和延迟时长等等
  • etcd:收集etcd存储集群的相关指标数据
  • kube-state-metrics:该组件可以派生出k8s相关的多个指标数据,主要是资源类型相关的计数器和元数据信息,包括制定类型的对象总数、资源限额、容器状态以及Pod资源标签系列等。
  • 每个被监控的主机都可以通过专用的exporter程序提供输出监控数据的接口,并等待Prometheus服务器周期性的进行数据抓取
  • 如果存在告警规则,则抓取到数据之后会根据规则进行计算,满足告警条件则会生成告警,并发送到Alertmanager完成告警的汇总和分发
  • 当被监控的目标有主动推送数据的需求时,可以以Pushgateway组件进行接收并临时存储数据,然后等待Prometheus服务器完成数据的采集
  • 任何被监控的目标都需要事先纳入到监控系统中才能进行时序数据采集、存储、告警和展示
  • 监控目标可以通过配置信息以静态形式指定,也可以让Prometheus通过服务发现的机制进行动态管理

二 在k8s中部署Prometheus

2.1 下载部署Prometheus所需资源

有网的方法

#在helm中添加Prometheus仓库
[root@k8s-master helm]# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

#下载Prometheus项目
[root@k8s-master helm]# helm pull prometheus-community/kube-prometheus-stack
[root@k8s-master helm]# ls
kube-prometheus-stack-62.6.0.tgz

#解压项目包
[root@k8s-master helm]# tar zxf kube-prometheus-stack-62.6.0.tgz
[root@k8s-master helm]# ls
kube-prometheus-stack  kube-prometheus-stack-62.6.0.tgz
[root@k8s-master helm]# cd kube-prometheus-stack/
[root@k8s-master kube-prometheus-stack]# ls
Chart.lock  charts  Chart.yaml  CONTRIBUTING.md  README.md  templates  values.yaml

没网的方法

[root@K8s-master ~]# mkdir prometheus
[root@K8s-master ~]# cd prometheus/
[root@K8s-master prometheus]# ls
grafana-11.2.0.tar                nginx-exporter-1.3.0-debian-12-r2.tar
kube-prometheus-stack-62.6.0.tgz  node-exporter-1.8.2.tar
kube-state-metrics-2.13.0.tar     prometheus-62.6.0.tar
nginx-18.1.11.tgz

[root@K8s-master prometheus]# tar zxf kube-prometheus-stack-62.6.0.tgz 
[root@K8s-master prometheus]# cd kube-prometheus-stack/
[root@K8s-master kube-prometheus-stack]# ls
Chart.lock  charts  Chart.yaml  CONTRIBUTING.md  README.md  templates  values.yaml


[root@K8s-master kube-prometheus-stack]# vim values.yaml 
  imageRegistry: "reg.harbor.org"

#根据所有项目中的values.yaml中指定的image路径下载容器镜像并上传至harbor仓库

#上传镜像
[root@K8s-master kube-prometheus-stack]# cd ..
[root@K8s-master prometheus]# docker load -i prometheus-62.6.0.tar 

打标签

[root@K8s-master prometheus]# docker tag quay.io/prometheus/prometheus:v2.54.1 reg.harbor.org/prometheus/prometheus:v2.54.1
[root@K8s-master prometheus]# docker tag quay.io/thanos/thanos:v0.36.1 reg.harbor.org/thanos/thanos:v0.36.1
[root@K8s-master prometheus]# docker tag quay.io/prometheus/alertmanager:v0.27.0 reg.harbor.org/prometheus/alertmanager:v0.27.0
[root@K8s-master prometheus]# docker tag quay.io/prometheus-operator/admission-webhook:v0.76.1 reg.harbor.org/prometheus-operator/admission-webhook:v0.76.1
[root@K8s-master prometheus]# docker tag registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6 reg.harbor.org/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6
[root@K8s-master prometheus]# docker tag quay.io/prometheus-operator/prometheus-operator:v0.76.1 reg.harbor.org/prometheus-operator/prometheus-operator:v0.76.1
[root@K8s-master prometheus]# docker tag quay.io/prometheus-operator/prometheus-config-reloader:v0.76.1 reg.harbor.org/prometheus-operator/prometheus-config-reloader:v0.76.1

上传

[root@K8s-master prometheus]# docker push reg.harbor.org/prometheus/prometheus:v2.54.1
[root@K8s-master prometheus]# docker push reg.harbor.org/thanos/thanos:v0.36.1
[root@K8s-master prometheus]# docker push reg.harbor.org/prometheus/alertmanager:v0.27.0
[root@K8s-master prometheus]# docker push reg.harbor.org/prometheus-operator/admission-webhook:v0.76.1
[root@K8s-master prometheus]# docker push reg.harbor.org/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6
[root@K8s-master prometheus]# docker push reg.harbor.org/prometheus-operator/prometheus-operator:v0.76.1
[root@K8s-master prometheus]# docker push reg.harbor.org/prometheus-operator/prometheus-config-reloader:v0.76.1

修改配置

[root@K8s-master kube-prometheus-stack]# cd charts/
[root@K8s-master charts]# ls
crds  grafana  kube-state-metrics  prometheus-node-exporter  prometheus-windows-exporter
[root@K8s-master charts]# cd grafana/
[root@K8s-master grafana]# ls
Chart.yaml  ci  dashboards  README.md  templates  values.yaml
[root@K8s-master grafana]# vim values.yaml 
  imageRegistry: "reg.harbor.org"
....
 image:
    # -- The Docker registry
    registry: docker.io
    repository: library/busybox
    tag: "latest"
    sha: ""
    pullPolicy: IfNotPresent
......

上传镜像

[root@K8s-master prometheus]# docker load -i grafana-11.2.0.tar
#打标签
[root@K8s-master prometheus]# docker tag grafana/grafana:11.2.0 reg.harbor.org/grafana/grafana:11.2.0
[root@K8s-master prometheus]# docker tag quay.io/kiwigrid/k8s-sidecar:1.27.4 reg.harbor.org/kiwigrid/k8s-sidecar:1.27.4
[root@K8s-master prometheus]# docker tag grafana/grafana-image-renderer:latest reg.harbor.org/grafana/grafana-image-renderer:latest
[root@K8s-master prometheus]# docker tag bats/bats:v1.4.1 reg.harbor.org/bats/bats:v1.4.1


#上传
[root@K8s-master prometheus]# docker push reg.harbor.org/grafana/grafana:11.2.0
[root@K8s-master prometheus]# docker push reg.harbor.org/kiwigrid/k8s-sidecar:1.27.4
[root@K8s-master prometheus]# docker push reg.harbor.org/grafana/grafana-image-renderer:latest
[root@K8s-master prometheus]# docker push reg.harbor.org/bats/bats:v1.4.1

修改配置

[root@K8s-master prometheus]# cd kube-prometheus-stack/
[root@K8s-master kube-prometheus-stack]# cd charts/
[root@K8s-master charts]# cd kube-state-metrics/
[root@K8s-master kube-state-metrics]# ls
Chart.yaml  README.md  templates  values.yaml
[root@K8s-master kube-state-metrics]# vim values.yaml
  registry: reg.harbor.org
....
  imageRegistry: "reg.harbor.org"
.....

上传镜像

[root@K8s-master prometheus]# docker load -i nginx-exporter-1.3.0-debian-12-r2.tar

#打标签
[root@K8s-master prometheus]# docker tag bitnami/nginx-exporter:1.3.0-debian-12-r2 reg.harbor.org/bitnami/nginx-exporter:1.3.0-debian-12-r2

#上传
[root@K8s-master prometheus]# docker push reg.harbor.org/bitnami/nginx-exporter:1.3.0-debian-12-r2

修改配置

[root@K8s-master prometheus]# cd kube-prometheus-stack/
[root@K8s-master kube-prometheus-stack]# cd charts/
[root@K8s-master charts]# cd prometheus-node-exporter/
[root@K8s-master prometheus-node-exporter]# ls
Chart.yaml  ci  README.md  templates  values.yaml
[root@K8s-master prometheus-node-exporter]# vim values.yaml
  registry: reg.harbor.org
......
  imageRegistry: "reg.harbor.org"
......

上传镜像

[root@K8s-master prometheus]# docker load -i node-exporter-1.8.2.tar 

#打标签
[root@K8s-master prometheus]# docker tag quay.io/prometheus/node-exporter:v1.8.2 reg.harbor.org/prometheus/node-exporter:v1.8.2
[root@K8s-master prometheus]# docker tag quay.io/brancz/kube-rbac-proxy:v0.18.0 reg.harbor.org/brancz/kube-rbac-proxy:v0.18.0

#上传
[root@K8s-master prometheus]# docker push reg.harbor.org/prometheus/node-exporter:v1.8.2
[root@K8s-master prometheus]# docker push reg.harbor.org/brancz/kube-rbac-proxy:v0.18.0

上传镜像

[root@K8s-master prometheus]# docker load -i kube-state-metrics-2.13.0.tar

#打标签
[root@K8s-master prometheus]# docker tag registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0 reg.harbor.org/kube-state-metrics/kube-state-metrics:v2.13.0
[root@K8s-master prometheus]# docker tag quay.io/brancz/kube-rbac-proxy:v0.18.0 reg.harbor.org/brancz/kube-rbac-proxy:v0.18.0

#上传
[root@K8s-master prometheus]# docker push reg.harbor.org/kube-state-metrics/kube-state-metrics:v2.13.0
[root@K8s-master prometheus]# docker push reg.harbor.org/brancz/kube-rbac-proxy:v0.18.0

安装Prometheus

#利用helm安装Prometheus
[root@k8s-master kube-prometheus-stack]# kubectl create namespace kube-prometheus-stack

#注意,在安装过程中千万别ctrl+c
[root@K8s-master kube-prometheus-stack]# helm  -n kube-prometheus-stack install kube-prometheus-stack  .
NAME: kube-prometheus-stack
LAST DEPLOYED: Thu Sep 12 00:48:16 2024
NAMESPACE: kube-prometheus-stack
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace kube-prometheus-stack get pods -l "release=kube-prometheus-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.


#查看所有pod是否运行
[root@K8s-master kube-prometheus-stack]# kubectl --namespace kube-prometheus-stack get pods
NAME                                                        READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-stack-alertmanager-0           2/2     Running   0          16m
kube-prometheus-stack-grafana-5454bc564-brzss               3/3     Running   0          17m
kube-prometheus-stack-kube-state-metrics-7f5b74ff59-cfxqq   1/1     Running   0          48s
kube-prometheus-stack-operator-54659dc7c4-dmqp9             1/1     Running   0          17m
kube-prometheus-stack-prometheus-node-exporter-f9qqj        1/1     Running   0          17m
kube-prometheus-stack-prometheus-node-exporter-jbx9x        1/1     Running   0          17m
kube-prometheus-stack-prometheus-node-exporter-m25bq        1/1     Running   0          17m
prometheus-kube-prometheus-stack-prometheus-0               2/2     Running   0          16m

#查看svc
[root@K8s-master kube-prometheus-stack]# kubectl -n kube-prometheus-stack get svc
NAME                                             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                            ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   17m
kube-prometheus-stack-alertmanager               ClusterIP   10.97.56.49      <none>        9093/TCP,8080/TCP            17m
kube-prometheus-stack-grafana                    ClusterIP   10.106.35.232    <none>        80/TCP                       17m
kube-prometheus-stack-kube-state-metrics         ClusterIP   10.98.215.227    <none>        8080/TCP                     17m
kube-prometheus-stack-operator                   ClusterIP   10.109.176.94    <none>        443/TCP                      17m
kube-prometheus-stack-prometheus                 ClusterIP   10.102.125.114   <none>        9090/TCP,8080/TCP            17m
kube-prometheus-stack-prometheus-node-exporter   ClusterIP   10.102.42.218    <none>        9100/TCP                     17m
prometheus-operated                              ClusterIP   None             <none>        9090/TCP                     17m

#修改暴漏方式
[root@K8s-master kube-prometheus-stack]# kubectl -n kube-prometheus-stack edit svc kube-prometheus-stack-grafana
  type: LoadBalancer

[root@K8s-master kube-prometheus-stack]# kubectl -n kube-prometheus-stack edit svc kube-prometheus-stack-prometheus
  type: LoadBalancer

[root@K8s-master kube-prometheus-stack]# kubectl -n kube-prometheus-stack get svc
NAME                                             TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                         AGE
alertmanager-operated                            ClusterIP      None             <none>          9093/TCP,9094/TCP,9094/UDP      21m
kube-prometheus-stack-alertmanager               ClusterIP      10.97.56.49      <none>          9093/TCP,8080/TCP               21m
kube-prometheus-stack-grafana                    LoadBalancer   10.106.35.232    172.25.254.60   80:34292/TCP                    21m
kube-prometheus-stack-kube-state-metrics         ClusterIP      10.98.215.227    <none>          8080/TCP                        21m
kube-prometheus-stack-operator                   ClusterIP      10.109.176.94    <none>          443/TCP                         21m
kube-prometheus-stack-prometheus                 LoadBalancer   10.102.125.114   172.25.254.61   9090:39249/TCP,8080:35449/TCP   21m
kube-prometheus-stack-prometheus-node-exporter   ClusterIP      10.102.42.218    <none>          9100/TCP                        21m
prometheus-operated                              ClusterIP      None             <none>          9090/TCP                        21m

各个svc的作用

alertmanager-operated 告警管理

kube-prometheus-stack-grafana 展示prometheus采集到的指标

kube-prometheus-stack-prometheus-node-exporter 收集节点级别的指标的工具

kube-prometheus-stack-prometheus 主程序

2.3 登陆grafana

#查看grafana密码
[root@K8s-master ~]# cd helm/
[root@K8s-master helm]# kubectl -n kube-prometheus-stack get secrets kube-prometheus-stack-grafana -o yaml
apiVersion: v1
data:
  admin-password: cHJvbS1vcGVyYXRvcg==
  admin-user: YWRtaW4=
  ldap-toml: ""
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: kube-prometheus-stack
    meta.helm.sh/release-namespace: kube-prometheus-stack
  creationTimestamp: "2024-09-11T16:48:31Z"
  labels:
    app.kubernetes.io/instance: kube-prometheus-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 11.2.0
    helm.sh/chart: grafana-8.5.1
  name: kube-prometheus-stack-grafana
  namespace: kube-prometheus-stack
  resourceVersion: "326952"
  uid: fcef5431-7838-49da-ad1f-4b7892ffa986
type: Opaque

#查看密码
[root@K8s-master helm]# echo -n "cHJvbS1vcGVyYXRvcg==" | base64 -d
prom-operator     #密码
[root@K8s-master helm]# echo -n "YWRtaW4=" | base64 -d
admin            #用户

浏览器访问172.25.254.60

2.4 导入面板

官方监控模板:Grafana dashboards | Grafana Labs

导入步骤如下:

导入后效果如下:

2.5 访问Prometheus 主程序

[root@k8s-master helm]# kubectl -n kube-prometheus-stack edit svc kube-prometheus-stack-prometheus
type: LoadBalancer

[root@K8s-master ~]# kubectl -n kube-prometheus-stack get svc kube-prometheus-stack-prometheus
NAME                               TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                         AGE
kube-prometheus-stack-prometheus   LoadBalancer   10.102.125.114   172.25.254.61   9090:39249/TCP,8080:35449/TCP   16h

三 监控使用示例

3.1 建立监控项目

打开监控功能

#下载示例所需helm项目
[root@k8s-master test]# helm  pull  bitnami/nginx --version 18.1.11
[root@k8s-master test]# ls
nginx-18.1.11.tgz  nginx-exporter-1.3.0-debian-12-r2.tar

[root@k8s-master test]# tar zxf nginx-18.1.11.tgz
[root@k8s-master test]# cd nginx/

#修改项目开启监控
[root@k8s-master nginx]# vim values.yaml
*** ***
metrics:
  ## @param metrics.enabled Start a Prometheus exporter sidecar container
  ##
  enabled: true
  ## Bitnami NGINX Prometheus Exporter image
  ## ref: https://hub.docker.com/r/bitnami/nginx-exporter/tags/
*** ***
  serviceMonitor:
    ## @param metrics.serviceMonitor.enabled Creates a Prometheus Operator ServiceMonitor (also requires `metrics.enabled` to be `true`)
    ##
    enabled: true
    ## @param metrics.serviceMonitor.namespace Namespace in which Prometheus is running
    ##
namespace: "kube-prometheus-stack"  #1021行
*** ***
    labels:          #1046行
      release: kube-prometheus-stack    #1047行          #指定监控标签



[root@k8s-master ~]# kubectl -n kube-prometheus-stack get servicemonitors.monitoring.coreos.com --show-labels

#安装项目,在安装之前一定要上传镜像到仓库中
[root@k8s-master nginx]# helm install  zhang .
NAME: zhang
LAST DEPLOYED: Tue Sep 10 23:18:19 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: nginx
CHART VERSION: 18.1.11
APP VERSION: 1.27.1

[root@k8s-master nginx]# kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
zhang-nginx-68dfdd6c-6gc69   2/2     Running   0          15s

root@k8s-master nginx]# kubectl get svc
NAME              TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                     AGE
kubernetes        ClusterIP      10.96.0.1       <none>          443/TCP                                     7d5h
zhang-nginx      LoadBalancer   10.99.134.151   172.25.254.63   80:34867/TCP,443:31734/TCP,9113:34171/TCP    61s

[root@k8s-master nginx]# curl  172.25.254.63
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

#压力测试
[root@k8s-master nginx]# ab -c 5 -n 100 http://172.25.254.63/index.html
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值