容器云平台No.7~kubernetes监控系统prometheus-operator

简介

prometheus-operator
Prometheus：一个非常优秀的监控工具或者说是监控方案。它提供了数据搜集、存储、处理、可视化和告警一套完整的解决方案。作为kubernetes官方推荐的监控系统，用Prometheus来监控kubernetes集群的状况和运行在集群上的应用运行状况。

Prometheus架构图

那Prometheus Operator是做什么的呢？
Operator是由CoreOS公司开发的，用来扩展 Kubernetes API，特定的应用程序控制器，它用来创建、配置和管理复杂的有状态应用，如数据库、缓存和监控系统。
可以理解为，Prometheus Operator就是用于管理部署Prometheus到kubernetes的工具，其目的是简化和自动化对Prometheus组件的维护。

Prometheus Operator架构

部署前准备

1、克隆kube-prometheus项目

[root@k8s-master001 opt]# git clone https://github.com/prometheus-operator/kube-prometheus.git

2、进入kube-prometheus/manifests目录，可以看到一堆yaml文件，文件太多，我们按用组件分类

[root@k8s-master001 manifests]# ls -al
total 20
drwxr-xr-x. 10 root root  140 Sep 14 21:25 .
drwxr-xr-x. 12 root root 4096 Sep 14 21:11 ..
drwxr-xr-x.  2 root root 4096 Sep 14 21:23 adapter
drwxr-xr-x.  2 root root  189 Sep 14 21:22 alertmanager
drwxr-xr-x.  2 root root  241 Sep 14 21:22 exporter
drwxr-xr-x.  2 root root  254 Sep 14 21:23 grafana
drwxr-xr-x.  2 root root  272 Sep 14 21:22 metrics
drwxr-xr-x.  2 root root 4096 Sep 14 21:25 prometheus
drwxr-xr-x.  2 root root 4096 Sep 14 21:23 serviceMonitor
drwxr-xr-x.  2 root root 4096 Sep 14 21:11 setup

3、修改yaml文件中的nodeSelector
首先查看下现在Node节点的标签

[root@k8s-master001 manifests]# kubectl get node --show-labels=true
NAME            STATUS   ROLES    AGE     VERSION   LABELS
k8s-master001   Ready    master   4d16h   v1.19.0   app.storage=rook-ceph,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master001,kubernetes.io/os=linux,node-role.kubernetes.io/master=
k8s-master002   Ready    master   4d16h   v1.19.0   app.storage=rook-ceph,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master002,kubernetes.io/os=linux,node-role.kubernetes.io/master=
k8s-master003   Ready    master   4d16h   v1.19.0   app.storage=rook-ceph,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master003,kubernetes.io/os=linux,node-role.kubernetes.io/master=,role=ingress-controller

并把manifests目录的yaml文件中nodeSelector改为kubernetes.io/os=linux
例如：vim setup/prometheus-operator-deployment.yaml，

      nodeSelector:
        kubernetes.io/os: linux

其他的自行修改,可以如下命令过滤并查看是否需要修改

[root@k8s-master001 manifests]# grep -A1 nodeSelector  prometheus/*
prometheus/prometheus-prometheus.yaml:  nodeSelector:
prometheus/prometheus-prometheus.yaml:  nodeSelector:
prometheus/prometheus-prometheus.yaml-    kubernetes.io/os: linux

部署kube-prometheus

1、安装operator

[root@k8s-master001 manifests]# kubectl  apply -f setup/
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created

[root@k8s-master001 manifests]# kubectl  get po -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-operator-74d54b5cfc-xgqg7   2/2     Running   0          2m40s

2、安装adapter

[root@k8s-master001 manifests]# kubectl  apply -f adapter/
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
servicemonitor.monitoring.coreos.com/prometheus-adapter created

[root@k8s-master001 manifests]# kubectl  get po -n monitoring</