prometheus+grafana 监控 Kubernetes

前言

Prometheus官方网址，或者百度自己了解脑补：https://prometheus.io/

官方文档说明链接

Prometheus是一个开源的系统监控工具。
根据配置的任务（job）以http/s周期性的收刮（scrape/pull）
指定目标（target）上的指标（metric）。目标（target）
可以以静态方式或者自动发现方式指定。
Prometheus将收刮（scrape）的指标（metric）保存在本地或者远程存储上。

Prometheus以pull方式来收集指标。对比push方式，
pull可以集中配置、针对不同的视角搭建不同的监控系统

Prometheus Server：核心组件，负责收刮和存储时序数据（time series data），并且提供查询接口；

Jobs/Exporters：客户端，监控并采集指标，对外暴露HTTP服务（/metrics）；
目前已经有很多的软件原生就支持Prometjeus，提供/metrics，可以直接使用；
对于像操作系统已经不提供/metrics的应用，可以使用现有的exporters
或者开发自己的exporters来提供/metrics服务；

Pushgateway：针对push系统设计，Short-lived jobs定时将指标push到Pushgateway，再由Prometheus Server从Pushgateway上pull；

Alertmanager：报警组件，根据实现配置的规则（rule）进行响应，例如发送邮件；

Web UI：Prometheus内置一个简单的Web控制台，可以查询指标，查看配置信息或者Service Discovery等，实际工作中，查看指标或者创建仪表盘通常使用Grafana，Prometheus作为Grafana的数据源；

数据结构
Prometheus按照时间序列存储指标，每一个指标都由Notation + Samples组成：
Notation：通常有指标名称与一组label组成：
{=, …}
Samples：样品，通常包含一个64位的浮点值和一个毫秒级的时间戳

Prometheus： 监控，报警，时间序列，http 协议进行系统监控，http接口方式

Grafana : 进行数据展现

一 . 安装部署普罗米修斯 yaml

[root@master k8s-prometheus-grafana-master]# ls
grafana  node-exporter.yaml  prometheus
[root@master k8s-prometheus-grafana-master]# cd prometheus/
[root@master prometheus]# ls
configmap.yaml  prometheus.deploy.yml  prometheus.svc.yml  rbac-setup.yaml
[root@master prometheus]# cd ../
[root@master k8s-prometheus-grafana-master]# ls
grafana  node-exporter.yaml  prometheus
[root@master k8s-prometheus-grafana-master]# kubectl create -f node-exporter.yaml
daemonset.apps/node-exporter created
service/node-exporter created
[root@master k8s-prometheus-grafana-master]#

[root@master k8s-prometheus-grafana-master]# cat node-exporter.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: kube-system
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
      k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      containers:
      - image: prom/node-exporter
        name: node-exporter
        ports:
        - containerPort: 9100
          protocol: TCP
          name: http
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: node-exporter
  name: node-exporter
  namespace: kube-system
spec:
  ports:
  - name: http
    port: 9100
    nodePort: 31672
    protocol: TCP
  type: NodePort
  selector:
    k8s-app: node-exporter

安装其他相关资源

[root@master k8s-prometheus-grafana-master]#

[root@master k8s-prometheus-grafana-master]# ls
grafana  node-exporter.yaml  prometheus
[root@master k8s-prometheus-grafana-master]# cd prometheus/
[root@master prometheus]# ls
configmap.yaml  prometheus.deploy.yml  prometheus.svc.yml  rbac-setup.yaml
[root@master prometheus]# kubectl create -f configmap.yaml
configmap/prometheus-config created
[root@master prometheus]# vim prometheus.deploy.yml
[root@master prometheus]# kubectl create -f rbac-setup.yaml
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
[root@master prometheus]# kubectl create -f prometheus.deploy.yml
deployment.apps/prometheus created
[root@master prometheus]# kubectl create -f prometheus.svc.yml
service/prometheus created
[root@master prometheus]#

查看资源确认

[root@master prometheus]# kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-2n8rc          1/1     Running   0          21d
coredns-9d85f5447-pxx2r          1/1     Running   0          21d
etcd-master                      1/1     Running   5          21d
kube-apiserver-master            1/1     Running   5          21d
kube-controller-manager-master   1/1     Running   5          21d
kube-flannel-ds-amd64-5gkc2      1/1     Running   0          21d
kube-flannel-ds-amd64-l5vfb      1/1     Running   0          21d
kube-proxy-k7wbk                 1/1     Running   0          21d
kube-proxy-pzdnc                 1/1     Running   5          21d
kube-proxy-qbl9p                 1/1     Running   0          21d
kube-scheduler-master            1/1     Running   5          21d
node-exporter-6k82f              1/1     Running   0          7m57s
node-exporter-zqdjv              1/1     Running   0          7m57s
prometheus-759d85775b-nrb8s      1/1     Running   0          62s

二 . 部署 grafana

安装相关资源

[root@master k8s-prometheus-grafana-master]# ls
grafana  node-exporter.yaml  prometheus
[root@master k8s-prometheus-grafana-master]# cd grafana/
[root@master grafana]# ls
grafana-deploy.yaml  grafana-ing.yaml  grafana-svc.yaml
[root@master grafana]# vim grafana-deploy.yaml
[root@master grafana]# kubectl create -f grafana-deploy.yaml
deployment.apps/grafana-core created
[root@master grafana]# kubectl create -f grafana-svc.yaml
service/grafana created
[root@master grafana]# kubectl create -f grafana-ing.yaml
ingress.extensions/grafana created

查看确认

[root@master grafana]# kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-9d85f5447-2n8rc          1/1     Running   0          21d
coredns-9d85f5447-pxx2r          1/1     Running   0          21d
etcd-master                      1/1     Running   5          21d
grafana-core-7789756d87-ttcg7    1/1     Running   0          2m9s
kube-apiserver-master            1/1     Running   5          21d
kube-controller-manager-master   1/1     Running   5          21d
kube-flannel-ds-amd64-5gkc2      1/1     Running   0          21d
kube-flannel-ds-amd64-l5vfb      1/1     Running   0          21d
kube-proxy-k7wbk                 1/1     Running   0          21d
kube-proxy-pzdnc                 1/1     Running   5          21d
kube-proxy-qbl9p                 1/1     Running   0          21d
kube-scheduler-master            1/1     Running   5          21d
node-exporter-6k82f              1/1     Running   0          25m
node-exporter-zqdjv              1/1     Running   0          25m
prometheus-759d85775b-nrb8s      1/1     Running   0          18m

查看服务

[root@master grafana]# kubectl get svc -n kube-system
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
grafana         NodePort    10.96.101.204   <none>        3000:30129/TCP           2m17s
kube-dns        ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   21d
node-exporter   NodePort    10.96.226.158   <none>        9100:31672/TCP           27m
prometheus      NodePort    10.96.109.175   <none>        9090:30003/TCP           20m
[root@master grafana]#

[root@master grafana]# kubectl get svc -n kube-system -o  wide
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE    SELECTOR
grafana         NodePort    10.96.101.204   <none>        3000:30129/TCP           2m42s   app=grafana,component=core
kube-dns        ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   21d    k8s-app=kube-dns
node-exporter   NodePort    10.96.226.158   <none>        9100:31672/TCP           27m    k8s-app=node-exporter
prometheus      NodePort    10.96.109.175   <none>        9090:30003/TCP           20m    app=prometheus

三 . 通过查看端口号来访问

设置显示数据模板