Prometheus是一个集数据收集存储、数据查询和数据图表显示于一身的开源监控组件。本文主要讲解如何搭建Prometheus,并使用它监控Kubernetes集群。
准备工作
- Kubernetes集群,如果你还没有搭建好Kubernetes集群,可以参考这篇文章-Kubernetes-离线部署Kubernetes 1.9.0
- Prometheus的docker镜像,截至笔者写这篇文章之时,最新的版本是2.3.2,如果你的Kubernetes集群不能访问Docker Hub,请将镜像pull下来放到自己的私有镜像仓库。
![img_5ac923fae4a1263fb63488f0dfe61054.png](https://i-blog.csdnimg.cn/blog_migrate/a8ff676528fc0bb1496c7dc5ac934ccf.png)
部署Prometheus
-
创建命名空间
新建一个yaml文件命名为monitor-namespace.yaml,写入如下内容
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
执行如下命令创建monitoring命名空间
kubectl create -f monitor-namespace.yaml
![img_b6cba0b57797e2178451c3e58b5b4328.png](https://i-blog.csdnimg.cn/blog_migrate/43c885c3b64827bde408eb1eab9db5b3.png)
-
创建ClusterRole
你需要对上面创建的命名空间分配集群的读取权限,以便Prometheus可以通过Kubernetes的API获取集群的资源指标。
新建一个yaml文件命名为cluster-role.yaml,写入如下内容:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/proxy
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups:
- extensions
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: default
namespace: monitoring
执行如下命令创建
kubectl create -f cluster-role.yaml
![img_e7b53a79e4e62974b157c625ff0e4cac.png](https://i-blog.csdnimg.cn/blog_migrate/a63647452431cacd14c4c604e77c0b39.png)
-
创建Config Map
我们需要创建一个Config Map保存后面创建Prometheus容器用到的一些配置,这些配置包含了从Kubernetes集群中动态发现pods和运行中的服务。
新建一个yaml文件命名为config-map.yaml,写入如下内容:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-conf
labels:
name: prometheus-server-conf
namespace: monitoring
data:
prometheus.yml: |-
global:
scrape_interval: 5s
evaluation_interval: 5s
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
执行如下命令进行创建
kubectl create -f config-map.yaml -n monitoring
![img_3d0708335487d3ee73ffb2d41575331d.png](https://i-blog.csdnimg.cn/blog_migrate/cd01d27d5e6e637ae3962ef601550490.png)
-
创建Deployment模式的Prometheus
新建一个yaml文件命名为prometheus-deployment.yaml,写入如下内容,镜像那里注意一下,我写的是我私有仓库的路径,如果kubernetes集群能访问Docker Hub请修改为【prom/prometheus:v2.3.2】:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: monitoring
spec:
replicas: 1
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: registry.docker.uih/library/prometheus:2.3.2
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
emptyDir: {}
使用如下命令部署
kubectl create -f prometheus-deployment.yaml --namespace=monitoring
部署完成后通过dashboard能够看到如下的界面:
![img_254ce2cce549aa176ba5169afdff718f.png](https://i-blog.csdnimg.cn/blog_migrate/ef3b15e3b4974e1d1a1b71dcaa201edb.png)
连接Prometheus
这里有两种方式
- 通过kubectl命令进行端口代理
- 针对Prometheus的POD暴露一个服务,推荐此种方式
首先新建一个yaml文件命名为prometheus-service.yaml,写入如下内容:
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
spec:
selector:
app: prometheus-server
type: NodePort
ports:
- port: 9090
targetPort: 9090
nodePort: 30909
执行如下命令创建服务
kubectl create -f prometheus-service.yaml --namespace=monitoring
![img_5e5e0c348cd93fb4751cdcd38693d4c9.png](https://i-blog.csdnimg.cn/blog_migrate/d182fc7f2a6ddbb575017f939debe403.png)
通过dashboard可以查看到部署成功的服务
![img_0ea4820198b727d27bb3d42db70be191.png](https://i-blog.csdnimg.cn/blog_migrate/d206c80e73da19ff2fdfd700ec8c5775.png)
现在可以通过浏览器访问【http://10.3.14.193:30909】,看到如下界面
![img_98047cb289f138b56b963b69541c3fc4.png](https://i-blog.csdnimg.cn/blog_migrate/6a9919219c4fb97b16bc5bbb9d9687ea.png)
现在可以点击 status –> Targets,马上就可以看到所有Kubernetes集群上的Endpoint通过服务发现的方式自动连接到了Prometheus。
![img_294412ffd64d760b39e887df914706b4.png](https://i-blog.csdnimg.cn/blog_migrate/aef8c8de066c77452247054414ed3b81.png)
![img_febac73497f10999541e4742c29b7e52.png](https://i-blog.csdnimg.cn/blog_migrate/83cac81d891d96196a3f92322d361f6a.png)
我们还可以通过图形化界面查看内存
![img_e6242d05ac60ff94a7d90a180ff6de2f.png](https://i-blog.csdnimg.cn/blog_migrate/c0eb4411a3f7e609832a7e41f17370fc.png)