Prometheus简介
部署Prometheus
一、Prometheus简介
官网介绍https://prometheus.io/docs/prometheus/latest/getting_started/
Prometheus架构图
时间序列数据：按照时间顺序记录系统状态变化的数据称为时序数据
时间序列数据引擎优势：①适用于大规模数据处理，性能 > NoSQL > 关系型数据库；②高效压缩算法，可以有效降低IO和存储空间
二、Prometheus部署
Prometheus部署可以采用StatefulSet和Deployment，以下将采用Deployment方式部署
- 创建namespace
kubectl create namespace monitor
2. 创建Prometheus使用的RBAC规则（定义了很多roles，Prometheus通过这些roles抓取到集群里面的数据指标）；
vim prometheus-rabc.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources: ["nodes","nodes/proxy","services","endpoints","pods"]
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources: ["ingress"]
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: prometheus
namespace: monitor
- 创建Prometheus的configmap来保存配置文件（配置文件、roles文件）；
配置文件
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitor
data:
prometheus.yml: |
global:
scrape_interval: 15s # 采集间隔15s
evaluation_interval: 15s
external_labels:
cluster: "kubernetes"
############ 数据采集job （采集Prometheus本身）###################
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['127.0.0.1:9090']
labels:
instance: prometheus
############ 指定告警规则文件路径位置 ###################
rule_files:
- /etc/prometheus/rules/*.rules
rules文件
此处分为两部分通用规则和节点规则，后续还可以添加svc、pod等
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-rules
namespace: monitor
data:
general.rules: |
groups:
- name: general.rules
rules:
- alert: InstanceDown
expr: |
up{job=~"k8s-nodes|prometheus"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.instance }} 停止工作"
description: "{{ $labels.instance }} 主机名：{{ $labels.hostname }} 已经停止1分钟以上."
node.rules: |
groups:
- name: node.rules
rules:
- alert: NodeFilesystemUsage
expr: |
100 - (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
for: 1m
labels:
severity: warning
annotations:
summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"
description: "{{ $labels.instance }} 主机名：{{ $labels.hostname }} : {{ $labels.mountpoint }} 分区使用大于85% (当前值: {{ $value }})"
4. 创建service将服务暴露（Prometheus默认端口9090）
apiVersion: v1
kind: Service
metadata:
name: prometheus
namespace: monitor
labels:
k8s-app: prometheus
spec:
type: ClusterIP
ports:
- name: http
port: 9090 #prometheus主服务使用的端口是9090
targetPort: 9090
selector:
k8s-app: prometheus
5. 创建deployment部署Prometheus容器；
创建之前需要先创建sc用来做持久化存储
部署存储时一定注意pv、pvc、sc之间的关系：
PV是Kubernetes集群中的一块持久化存储资源，它由管理员创建并配置。每个PV都有一个特定的容量和访问模式（如读写权限等），并且可以被绑定到一个PVC上。PV是实际的存储资源抽象，比如云提供商的云盘、NFS共享、本地磁盘等。
PVC是用户（或者说是Pod）对存储资源的一种请求或声明。用户通过创建PVC来请求特定大小和访问模式的存储资源，而不是直接请求某个特定的PV。Kubernetes会自动匹配合适的PV给PVC，一旦绑定，该PVC就代表了用户对那部分PV的使用权。这样设计的好处是实现了存储资源的动态分配与绑定，同时使得应用（Pods）与底层存储细节解耦。
StorageClass是用来描述存储类别的对象，定义了一组PV的参数，包括类型、性能特征、备份策略等。当用户创建PVC时，可以指定一个StorageClass，Kubernetes会根据这个类别自动创建或绑定相应的PV给PVC。StorageClass引入了存储资源的动态供给机制，使得存储的配置和分配更加灵活和自动化。如果没有指定StorageClass，Kubernetes可能会使用默认的StorageClass（如果设置有）来满足PVC的需求。
PV与PVC的关系：PV提供存储资源，PVC是对这些资源的请求。PVC的创建触发了PV的分配过程，当一个PV与PVC成功绑定后，该PV的存储资源就可以被PVC所引用的Pod使用。
StorageClass与PV/PVC的关系：StorageClass为PV的创建提供了模板和策略，当PVC指定某个StorageClass时，Kubernetes会根据该类别自动创建PV（如果不存在合适的已存在PV）并与PVC绑定。这简化了存储资源的管理和分配过程，使得存储资源的配置更加标准化和自动化。
总结来说，PV、PVC和StorageClass共同构成了Kubernetes中持久化存储的管理体系，其中PV代表实际存储资源，PVC是用户对存储资源的请求，而StorageClass则定义了如何动态供应这些存储资源的规则，三者相互配合，确保了存储资源的有效管理和分配。
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: prometheus-data-pvc
namespace: monitor
spec:
accessModes:
- ReadWriteMany
storageClassName: "nfs-storage"
resources:
requests:
storage: 10Gi
部署deployment时注意Prometheus容器的几个参数
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitor
labels:
k8s-app: prometheus
spec:
replicas: 1
selector:
matchLabels:
k8s-app: prometheus
template:
metadata:
labels:
k8s-app: prometheus
spec:
serviceAccountName: prometheus
containers:
- name: prometheus
image: prom/prometheus:v2.36.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 9090
securityContext:
runAsUser: 65534
privileged: true
command:
- "/bin/prometheus"
args:
- "--config.file=/etc/prometheus/prometheus.yml" #prometheus配置文件所在的目录
- "--web.enable-lifecycle" #是否开启热加载
- "--storage.tsdb.path=/prometheus" #Prometheus数据存放的目录
- "--storage.tsdb.retention.time=10d" #Prometheus数据周期，只能查看10天的数据
- "--web.console.libraries=/etc/prometheus/console_libraries" #指定控制台依赖的存储路径
- "--web.console.templates=/etc/prometheus/consoles" #控制台模板存储路径
resources:
limits:
cpu: 2000m
memory: 2048Mi
requests:
cpu: 1000m
memory: 512Mi
readinessProbe:
httpGet:
path: /-/ready
port: 9090
initialDelaySeconds: 5
timeoutSeconds: 10
livenessProbe:
httpGet:
path: /-/healthy
port: 9090
initialDelaySeconds: 30
timeoutSeconds: 30
volumeMounts:
- name: data
mountPath: /prometheus
subPath: prometheus
- name: config
mountPath: /etc/prometheus
- name: prometheus-rules
mountPath: /etc/prometheus/rules
- name: configmap-reload
image: jimmidyson/configmap-reload:v0.5.0 #用于监听configmap文件中的内容，如果发生改变会自动的调用Prometheus接口自动热加载
imagePullPolicy: IfNotPresent
args:
- "--volume-dir=/etc/config"
- "--webhook-url=http://localhost:9090/-/reload"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 10Mi
volumeMounts:
- name: config
mountPath: /etc/config
readOnly: true
volumes:
- name: data
persistentVolumeClaim:
claimName: prometheus-data-pvc
- name: prometheus-rules
configMap:
name: prometheus-rules
- name: config
configMap:
name: prometheus-config
6. 创建ingress实现外部访问
由于没有安装ingress组件，故未通过ingress实现访问，而是通过svc NodePort方式实现
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: monitor
name: prometheus-ingress
spec:
ingressClassName: nginx
rules:
- host: prometheus.kubernets.cn
http:
paths:
- pathType: Prefix
backend:
service:
name: prometheus
port:
number: 9090
path: /
Prometheus监控总结
- 监控全面：可以监控集群组件、容器、各种服务以及底层服务器资源
- 支持动态服务发现：configmap-reload用于监听指定的 ConfigMap 文件中的内容，如果内容发生更改会动态加载
- 灵活的告警机制：结合AlterManager可以灵活配置各种告警规则以及通过多渠道（短信、邮件、钉钉等）进行告警
- 高效存储：Prometheus存储时时间序列数据存储，高效压缩算法可以降低IO和存储空间，同时具备大规模数据查询的优势
- 多维数据模型：支持多维度数据并使用标准的PromQL进行分析和展示
三、k8s集群层面监控
kubeStateMetrics
主要功能
- 节点状态信息，如节点 CPU 和内存的使用情况、节点状态、节点标签等。
- Pod 的状态信息，如 Pod 状态、Pod 的标签和注释、容器状态、容器镜像信息等
- 控制器（Deployment、Daemonset、Statefulset 和 ReplicaSet ）的状态信息，如副本数、副本状态、创建时间等。
- Service 的状态信息，如服务类型、服务 IP 和端口等。
- 存储卷的状态信息，如存储卷类型、存储卷容量等。
- Kubernetes 的 API 服务器状态信息，如 API 服务器的状态、请求次数、响应时间等。
yaml文件中包含六类信息
ServiceAccount、ClusterRole、ClusterRoleBinding、Deployment、ConfigMap、Service
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-state-metrics
namespace: monitor
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-state-metrics
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
resources:
- configmaps
- secrets
- nodes
- pods
- services
- resourcequotas
- replicationcontrollers
- limitranges
- persistentvolumeclaims
- persistentvolumes
- namespaces
- endpoints
verbs: ["list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- daemonsets
- deployments
- replicasets
verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources:
- cronjobs
- jobs
verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
resources:
- horizontalpodautoscalers
verbs: ["list", "watch"]
- apiGroups: ["networking.k8s.io", "extensions"]
resources:
- ingresses
verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources:
- storageclasses
verbs: ["list", "watch"]
- apiGroups: ["certificates.k8s.io"]
resources:
- certificatesigningrequests
verbs: ["list", "watch"]
- apiGroups: ["policy"]
resources:
- poddisruptionbudgets
verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kube-state-metrics-resizer
namespace: monitor
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
resources:
- pods
verbs: ["get"]
- apiGroups: ["extensions","apps"]
resources:
- deployments
resourceNames: ["kube-state-metrics"]
verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kube-state-metrics
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-state-metrics
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kube-state-metrics
namespace: monitor
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
name: kube-state-metrics
namespace: monitor
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kube-state-metrics
namespace: monitor
labels:
k8s-app: kube-state-metrics
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.3.0
spec:
selector:
matchLabels:
k8s-app: kube-state-metrics
version: v1.3.0
replicas: 1
template:
metadata:
labels:
k8s-app: kube-state-metrics
version: v1.3.0
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
priorityClassName: system-cluster-critical
serviceAccountName: kube-state-metrics
containers:
- name: kube-state-metrics
image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.4.2
ports:
- name: http-metrics ## 用于公开kubernetes的指标数据的端口
containerPort: 8080
- name: telemetry ##用于公开自身kube-state-metrics的指标数据的端口
containerPort: 8081
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 5
- name: addon-resizer ##addon-resizer 用来伸缩部署在集群内的 metrics-server, kube-state-metrics等监控组件
image: mirrorgooglecontainers/addon-resizer:1.8.6
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 100m
memory: 30Mi
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: config-volume
mountPath: /etc/config
command:
- /pod_nanny
- --config-dir=/etc/config
- --container=kube-state-metrics
- --cpu=100m
- --extra-cpu=1m
- --memory=100Mi
- --extra-memory=2Mi
- --threshold=5
- --deployment=kube-state-metrics
volumes:
- name: config-volume
configMap:
name: kube-state-metrics-config
---
# Config map for resource configuration.
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-state-metrics-config
namespace: monitor
labels:
k8s-app: kube-state-metrics
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
data:
NannyConfiguration: |-
apiVersion: nannyconfig/v1alpha1
kind: NannyConfiguration
---
apiVersion: v1
kind: Service
metadata:
name: kube-state-metrics
namespace: monitor
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "kube-state-metrics"
annotations:
prometheus.io/scrape: 'true'
spec:
ports:
- name: http-metrics
port: 8080
targetPort: http-metrics
protocol: TCP
- name: telemetry
port: 8081
targetPort: telemetry
protocol: TCP
selector:
k8s-app: kube-state-metrics
部署成功
验证指标是否被收集
curl -kL $(kubectl get service -n monitor | grep kube-state-metrics |awk ‘{ print $3 }’):8080/metrics
核心组件信息收集任务配置
所有的任务采集需要在prometheus-config.yaml文件中配置
集群是通过kubeadm部署的，核心组件都是静态pod，基于配置文件生成pod的信息，依托于kubelet做生命周期的管理
job中
apiserver
注意事项：使用https访问时，需要tls相关配置，可以指定ca证书路径或者 insecure_skip_verify: true跳过证书验证。
除此之外，还要指定 bearer_token_file，否则会提示 server returned HTTP status 400 Bad Request;
- job_name: kube-apiserver
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
action: keep
regex: default;kubernetes
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
controller-manager
修改配置文件两处
- --bind-address=0.0.0.0
注释- --port=0
文件位置/etc/kubernetes/manifests/
查看controller-manager信息
[root@aminglinux01-master ~]# kubectl describe pod -n kube-system kube-controller-manager-aminglinux01-master
默认绑定的地址是127.0.0.1，数据采集的时候不一定能通，所以需要修改为0.0.0.0
有几个Master节点需要在几个Master节点上修改
- job_name: kube-controller-manager
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_component]
regex: kube-controller-manager
action: keep
- source_labels: [__meta_kubernetes_pod_ip]
regex: (.+)
target_label: __address__
replacement: ${1}:10252
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: service
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
疑问：controller端口10252不同，看监听的是10257端口，待确认
同理配置scheduler、kube-state-metrics、coredns、etcd
注意配置etcd的时候两个节点修改时间需要拉长，避免两个节点同时异常集群崩掉
重要参数
kubernetes_sd_configs: 设置发现模式为 Kubernetes 动态服务发现
kubernetes_sd_configs.role: 指定 Kubernetes 的服务发现模式，这里设置为 endpoints 的服务发现模式，该模式下会调用 kube-apiserver 中的接口获取指标数据。并且还限定只获取 kube-state-metrics 所在 - Namespace 的空间 kube-system 中的 Endpoints 信息
kubernetes_sd_configs.namespace: 指定只在配置的 Namespace 中进行 endpoints 服务发现
relabel_configs: 用于对采集的标签进行重新标记
热加载
curl -XPOST http://192.168.222.101:31611/-/reload
cAdvisor
特点：
①与kubelet融合无需单独部署cAdvisor组件，采集地址/metrics/cadvisor，需要配置https访问，可以设置 insecure_skip_verify: true 跳过证书验证;
②以守护进程的方式运行，监控容器资源和性能
- job_name: kubelet
metrics_path: /metrics/cadvisor
scheme: https
tls_config:
insecure_skip_verify: true #需要配置https访问，可以设置 insecure_skip_verify: true 跳过证书验证;
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
nodeexporter
采集底层资源信息：服务器CPU频率信息、磁盘IO统计、剩余可用内存等等。
以守护进程运行，每个节点一个
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: monitor
labels:
name: node-exporter
spec:
selector:
matchLabels:
name: node-exporter
template:
metadata:
labels:
name: node-exporter
spec:
hostPID: true
hostIPC: true
hostNetwork: true
containers:
- name: node-exporter
image: prom/node-exporter:latest
ports:
- containerPort: 9100
resources:
requests:
cpu: 0.15
securityContext:
privileged: true
args:
- --path.procfs
- /host/proc
- --path.sysfs
- /host/sys
- --collector.filesystem.ignored-mount-points
- '"^/(sys|proc|dev|host|etc)($|/)"'
volumeMounts:
- name: dev
mountPath: /host/dev
- name: proc
mountPath: /host/proc
- name: sys
mountPath: /host/sys
- name: rootfs
mountPath: /rootfs
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
volumes:
- name: proc
hostPath:
path: /proc
- name: dev
hostPath:
path: /dev
- name: sys
hostPath:
path: /sys
- name: rootfs
hostPath:
path: /
重要参数
hostPID：指定是否允许Node Exporter进程绑定到主机的PID命名空间。若值为true，则可以访问宿主机中的PID信息。
hostIPC：指定是否允许Node Exporter进程绑定到主机的IPC命名空间。若值为true，则可以访问宿主机中的IPC信息。
hostNetwork：指定是否允许Node Exporter进程绑定到主机的网络命名空间。若值为true，则可以访问宿主机中的网络信息。
验证
curl localhost:9100/metrics |grep cpu
配置采集任务
node_exporter也是每个node节点都运行，因此role使用node即可，默认address端口为10250，替换为9100即可
- job_name: k8s-nodes
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__address__]
regex: '(.*):10250'
replacement: '${1}:9100'
target_label: __address__
action: replace
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- source_labels: [__meta_kubernetes_endpoints_name]
action: replace
target_label: endpoint
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
三种工具各司其职，kube-state-metrics监控k8s中的各种资源对象；cAdvisor监控容器资源；node-exporter对底层服务器信息进行采集。
扫一扫
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
