Kubernetes&Monitoring

于 2024-08-16 13:20:55 发布
阅读量230 收藏 10
点赞数 5
分类专栏： 云计算 # Monitoring 文章标签： kubernetes 容器 云原生 监控
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_51882166/article/details/141226828
版权

  • 监控集群中应用

  • 监控集群本身

    • Control-Plane Components(api-server,coredns,kube-scheduler)
    • Kubelet(cAdvisor)-暴露容器metrics
    • Kube-state-metrics-集群层面metrics（deployments，pods metrics）
    • Node-exporter-Host相关metrics(cpu,mem,network)

部署

  • helm是k8s的包管理工具

prometheus-operator
Install helm

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Install Prometheus Chart

kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Service Monitors

controlplane ~ ➜  kubectl get crd
NAME                                        CREATED AT
addons.k3s.cattle.io                        2024-08-15T00:00:11Z
helmcharts.helm.cattle.io                   2024-08-15T00:00:11Z
helmchartconfigs.helm.cattle.io             2024-08-15T00:00:11Z
traefikservices.traefik.containo.us         2024-08-15T00:01:07Z
ingressroutes.traefik.containo.us           2024-08-15T00:01:07Z
middlewaretcps.traefik.containo.us          2024-08-15T00:01:07Z
ingressrouteudps.traefik.containo.us        2024-08-15T00:01:07Z
serverstransports.traefik.containo.us       2024-08-15T00:01:07Z
tlsoptions.traefik.containo.us              2024-08-15T00:01:07Z
tlsstores.traefik.containo.us               2024-08-15T00:01:07Z
middlewares.traefik.containo.us             2024-08-15T00:01:07Z
ingressroutetcps.traefik.containo.us        2024-08-15T00:01:07Z
alertmanagerconfigs.monitoring.coreos.com   2024-08-15T09:10:15Z
alertmanagers.monitoring.coreos.com         2024-08-15T09:10:16Z
podmonitors.monitoring.coreos.com           2024-08-15T09:10:16Z
probes.monitoring.coreos.com                2024-08-15T09:10:16Z
prometheuses.monitoring.coreos.com          2024-08-15T09:10:16Z #创建prometheuses instance
prometheusrules.monitoring.coreos.com       2024-08-15T09:10:17Z
servicemonitors.monitoring.coreos.com       2024-08-15T09:10:17Z #添加targets用来让prometheus抓取
thanosrulers.monitoring.coreos.com          2024-08-15T09:10:17Z
  • Service monitors定义prometheus用于监控和抓取的targets集合
    Pod
# 创建一个持久化的rocky pod
kubectl run my-host --image=rockylinux/rockylinux --command -- /bin/bash -c "while true; do sleep 3600; done"
pod/my-host created
# 进入container
controlplane ~ ➜  kubectl exec my-host -it -- /bin/bash
[root@my-host /]# yum install wget -y
[root@my-host /]# wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
[root@my-host /]# tar -xzf node_exporter-1.8.2.linux-amd64.tar.gz 
[root@my-host /]# cd node_exporter-1.8.2.linux-amd64

 nohup ./node_exporter & 
# 后台启动nohup ./node_exporter

# 测试
controlplane ~ ➜  kubectl get pod my-host -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
my-host   1/1     Running   0          13m   10.42.0.12   controlplane   <none>           <none>
# 尝试访问服务
controlplane ~ ✖ curl 10.42.0.12:9100
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Node Exporter</title>
    <style>body {
  font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,Liberation Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;
  margin: 0;
}
header {
  background-color: #e6522c;
  color: #fff;
  font-size: 1rem;
  padding: 1rem;
}
main {
  padding: 1rem;
}
label {
  display: inline-block;
  width: 0.5em;
}

</style>
  </head>
  <body>
    <header>
      <h1>Node Exporter</h1>
    </header>
    <main>
      <h2>Prometheus Node Exporter</h2>
      <div>Version: (version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)</div>
      <div>
        <ul>
          
          <li><a href="/metrics">Metrics</a></li>
          
        </ul>
      </div>
      
      
    </main>
  </body>
</html>
controlplane ~ ➜  curl 10.42.0.12:9100/metrics 
···
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
···

创建Service

# 查看Prometheus instance配置
controlplane ~ ➜  kubectl get prometheuses.monitoring.coreos.com -o yaml 
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    annotations:
      meta.helm.sh/release-name: prometheus-stack
      meta.helm.sh/release-namespace: default
    creationTimestamp: "2024-08-16T02:52:40Z"
    generation: 1
    labels:
      app: kube-prometheus-stack-prometheus
      app.kubernetes.io/instance: prometheus-stack
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/part-of: kube-prometheus-stack
      app.kubernetes.io/version: 45.6.0
      chart: kube-prometheus-stack-45.6.0
      heritage: Helm
      release: prometheus-stack
    name: prometheus-stack-kube-prom-prometheus
    namespace: default
    resourceVersion: "4393"
    uid: b1e782a0-c7d6-4f05-b5b3-20502623a9df
  spec:
    alerting:
      alertmanagers:
      - apiVersion: v2
        name: prometheus-stack-kube-prom-alertmanager
        namespace: default
        pathPrefix: /
        port: http-web
    enableAdminAPI: false
    evaluationInterval: 30s
    externalUrl: http://prometheus-stack-kube-prom-prometheus.default:9090
    hostNetwork: false
    image: quay.io/prometheus/prometheus:v2.42.0
    listenLocal: false
    logFormat: logfmt
    logLevel: info
    paused: false
    podMonitorNamespaceSelector: {}
    podMonitorSelector:
      matchLabels:
        release: prometheus-stack
    portName: http-web
    probeNamespaceSelector: {}
    probeSelector:
      matchLabels:
        release: prometheus-stack
    replicas: 1
    retention: 10d
    routePrefix: /
    ruleNamespaceSelector: {}
    ruleSelector:
      matchLabels:
        release: prometheus-stack # 创建的rule包含这个标签，才能被prometheus instance发现
    scrapeInterval: 30s
    securityContext:
      fsGroup: 2000
      runAsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
    serviceAccountName: prometheus-stack-kube-prom-prometheus
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-stack# 创建的serviceMonitor也要包含这个标签，才能被prometheus instance发现
    shards: 1
    version: v2.42.0
    walCompression: true
  status:
    availableReplicas: 1
    conditions:
    - lastTransitionTime: "2024-08-16T02:53:21Z"
      observedGeneration: 1
      status: "True"
      type: Available
    - lastTransitionTime: "2024-08-16T02:52:50Z"
      observedGeneration: 1
      status: "True"
      type: Reconciled
    paused: false
    replicas: 1
    shardStatuses:
    - availableReplicas: 1
      replicas: 1
      shardID: "0"
      unavailableReplicas: 0
      updatedReplicas: 1
    unavailableReplicas: 0
    updatedReplicas: 1
kind: List
metadata:
  resourceVersion: ""

# 获取pod标签
controlplane ~ ➜  kubectl get pod my-host --show-labels NAME      READY   STATUS    RESTARTS   AGE   LABELS
my-host   1/1     Running   0          16m   run=my-host
#### svc.yml
apiVersion: v1
kind: Service
metadata:
  name: my-host-exporter-svc
  labels:
    job: my-host-exporter
    app: my-host-exporter-svc
spec:
  selector:
    run: my-host
  ports:
    - name: exporter
      protocol: TCP
      port: 9100
      targetPort: 9100
### 测试
controlplane ~ ➜  kubectl get svc my-host-exporter-svc 
NAME                   TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
my-host-exporter-svc   ClusterIP   10.43.32.68   <none>        9100/TCP   9s
controlplane ~ ✖ curl 10.43.32.68:9100
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Node Exporter</title>
    <style>body {
  font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,Liberation Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;
  margin: 0;
}
header {
  background-color: #e6522c;
  color: #fff;
  font-size: 1rem;
  padding: 1rem;
}
main {
  padding: 1rem;
}
label {
  display: inline-block;
  width: 0.5em;
}

</style>
  </head>
  <body>
    <header>
      <h1>Node Exporter</h1>
    </header>
    <main>
      <h2>Prometheus Node Exporter</h2>
      <div>Version: (version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)</div>
      <div>
        <ul>
          
          <li><a href="/metrics">Metrics</a></li>
          
        </ul>
      </div>
      
      
    </main>
  </body>
</html>

ServiceMonitor Template

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-host-svc-mon
  labels:
    release: prometheus-stack  
spec:
  jobLabel: job
  endpoints:
  - port: exporter # 同svc
    interval: 30s
    path: /metrics # metrics路径
  selector:
    matchLabels:
      app: my-host-exporter-svc

创建之后发现我们定义的ServiceMonitor已经被prometheus instance发现并可以抓取metrics
在这里插入图片描述
我们也可以执行查询，对my-host Pod的一些指标进行观察
在这里插入图片描述

Rules

为了添加规则，Operator拥有一个prometheusrule
的CRD，用来向prometheus instance注册新规则

Template examples

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: prometheus-stack  # 保证可以被prometheus instance找到并注册
  name: my-host-rules
spec:
  groups:
    - name: api
      rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{$labels.instance}} down"

在这里插入图片描述

Dean.song
关注
  • 5
    点赞
  • 踩
  • 10
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
在Linux上以all in one模式安装kubernetes & kubesphere
每天都要开心呀(#^.^#)
07-27 949
通过 KubeSphere 可以快速管理 Kubernetes 集群、部署应用、服务发现、CI/CD 流水线、集群扩容、微服务治理、日志查询和监控告警。
CC00325.CloudKubernetes——|KuberNetes&运维.V43|——|白盒监控.v05|kafka.v02|
yanqi_vip
04-01 548
一、监控没有metrics接口的业务应用：以kafka为例 ### --- 创建一个kafka-exporter的deployment ~~~ 创建kafka-exporter的yaml文件 [root@k8s-master01 kafka-exporter]# vim kafka-exporter-deployment.yaml apiVersion: apps/v1 ki...
第23章：Kubernetes Metrics & Monitoring
Davidwatt的博客
12-04 814
Kubernetes Metrics & Monitoring
Kubernetes&&容器编排调度&&K3s
最新发布
dfq737211338的博客
06-03 44
官网RKE容器奇谈：探秘 K3s 前世今生，轻舟启航的轻量级 KubernetesK3s vs K8s：轻量级和全功能的对决轻量级 Kubernetes 集群发行版 K3s 完全进阶指南。
Kubernetes: Monitoring with Prometheus Kubernetes: Monitoring with Prometheus Lynda课程中文字幕
lyndacn的博客
03-17 206
Kubernetes: Monitoring with Prometheus 中文字幕 Kubernetes: Monitoring with Prometheus 中文字幕Kubernetes: Monitoring with Prometheus 为了防止中断，您必须利用Kubernetes环境中的监控和警报工具 Prometheus是一个开源系统监控和警报工具包，特别适合Kubernete...
CC00328.CloudKubernetes——|KuberNetes&运维.V46|——|黑盒监控.v01|blackbox.v01|创建|
yanqi_vip
04-01 195
一、黑盒监控 ### --- 黑盒监控和白盒监控模板 ~~~ # 白盒监控： ~~~ 监控一些内部的数据，topic的监控数据，Redis key的大小。 ~~~ 内部暴露的指标被称为白盒监控。#比较关注的是原因。 ~~~ # 黑盒监控： ~~~ 站在用户的角度看到的东西。网站不能打开，网站打开的比较慢。比较#关注现象， ~~~ 表示正...
CC00340.CloudKubernetes——|KuberNetes&运维.V57|——|Prometheus.v02|自动发现.v02|
yanqi_vip
04-01 483
一、创建ClusterRoleBinding查看Prometheus是否正确获取数据 ### --- 报错现象：解决failed to list *v1beta1.Ingress ~~~ 查看Prometheus日志是否有报错信息 [root@k8s-master01 discovery]# kubectl get po -n monitoring NAME ...
Kubernetes monitoring architecture
chengyuming123的博客
01-21 286
参考文献：Kubernetes monitoring architecture 概要 监控分为两个管道（pipelines）: Core metrics pipeline 包括 Kubelet, 一个资源评估器，精简化的Heapster（也称作metric-server）和服务于master metrics API的API服务器。这些指标用于核心系统组件，例如调度逻辑(例如调度程序和基于系统...
CC00361.CloudKubernetes——|KuberNetes&运维.V77|——|IngressNginx.v13|监控|
yanqi_vip
04-01 1036
一、IngressNginx监控 ### --- IngressNginx监控 ~~~ # IngressNginx监控官网地址： ~~~ https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/ 二、ingress nginx 监控配置：https://github.com/kub...
terraform-kubernetes-monitoring-prometheus-operator:Prometheus运算符由Terraform进行检测
04-04
Prometheus运算符由Terraform进行检测设置AWS提供程序（可选）## Retrieve authentication for kubernetes from aws.#provider "aws" { profile = var.aws_profile region = var.aws_region}## Get kubernetes ...
terraform-kubernetes-monitoring-thanos-querier:Thanos查询器服务，用于聚合普罗米修斯实例
04-06
Thanos查询层 查询层负责对您的thanos商店（运行商店api的远程实例）运行PromQL查询。 ┌──────────────────┐ ┌────────────┬─────────┐ ┌────────────┐...
kube-slack：Kubernetes松弛监控
02-03
kube-slack是Kubernetes的监视服务。 当pod失败时，它将在Slack频道中发布消息。 安装 创建一个传入的Webhook： 在[松弛]介面中，按一下搜寻框旁边的齿轮按钮（[频道设定]）。 选择“添加应用或集成” 搜索“传入...
kubernetes-kargo-logging-monitoring：使用kargo部署kubernetes集群
02-03
通过具有日志记录（EFK）和监视（Prometheus）支持的kargo部署kubernetes 您会得到什么： Kargo ：一种强大，灵活的方式来构建，热升级/迁移和扩展kubernetes（k8s） 正在运行的主节点的X数量：k8（用于容器编排）...
wavefront-collector-for-kubernetes:使用Wavefront监视Kubernetes集群
03-16
Kubernetes的Wavefront收集器 是一个高性能的流分析平台，用于监视和优化您的环境和应用程序。 Kubernetes的Wavefront收集器是一种代理，可在Kubernetes集群中的每个节点上作为运行。 它收集有关群集的指标和事件...
kubenurse：Kubernetes网络监控
02-04
库本努尔kubenurse是一项小服务，它监视kubernetes群集中的所有网络连接，并将采用的度量标准导出为prometheus端点。部署方式您可以从获取Docker映像。 目录包含将kubenurse部署到kube-system名称空间的yaml。 一切...
kube-top:您的kubernetes集群的顶部
04-27
kube-top 聚集并提供以下kubectl命令的直观表示： kubectl top pods kubectl top nodes kubectl get pods kubectl get events用法-namespace覆盖默认名称空间。 对于所有名称空间，请使用“”。...
ppt what is kubernetes
06-14
apiVersion: monitoring/v1 kind: ServiceMonitor metadata: name: myapp-monitor spec: selector: matchLabels: app: myapp endpoints: - port: web ``` ```yaml apiVersion: v1 kind: Pod metadata: name:...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值