构建Docker容器监控系统(Cadvisor +Prometheus+Grafana)

Cadvisor +Prometheus+Grafana

1.1、Cadvisor产品简介

        Cadvisor是Google开源的一款用于展示和分析容器运行状态的可视化工具。通过在主机上运行Cadvisor用户可以轻松的获取到当前主机上容器的运行统计信息，并以图表的形式向用户展示。

1.2、安装docker-ce

[root@localhost ~]# iptables -F
[root@localhost ~]# setenforce 0
setenforce: SELinux is disabled
[root@localhost ~]# systemctl stop firewalld

[root@localhost ~]# cd /etc/yum.repos.d/
[root@localhost ~]# wget http://mirrors.aliyun.com/repo/Centos-7.repo

[root@localhost ~]# yum -y install yum-utils device-mapper-persistent-data lvm2
[root@localhost ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@localhost ~]# yum -y install docker-ce
[root@localhost ~]# systemctl start docker
[root@localhost ~]# systemctl enable docker
[root@localhost ~]# docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:39:02 2023
 OS/Arch:           linux/amd64
 Context:           default
 
Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:38:05 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

阿里云镜像加速器

阿里云登录 - 欢迎登录阿里云，安全稳定的云计算服务平台欢迎登录阿里云，全球领先的云计算及人工智能科技公司，阿里云为200多个国家和地区的企业、开发者和政府机构提供云计算基础服务及解决方案。阿里云云计算、安全、大数据、人工智能、企业应用、物联网等云计算服务。https://cr.console.aliyun.com/

[root@localhost ~]# cat << END > /etc/docker/daemon.json
{
        "registry-mirrors":[ "https://nyakyfun.mirror.aliyuncs.com" ]
}
END

[root@localhost ~]# systemctl daemon-reload

[root@localhost ~]# systemctl restart docker

 下载组件镜像

[root@localhost ~]# docker pull tutum/influxdb
 
Using default tag: latest
latest: Pulling from tutum/influxdb
a3ed95caeb02: Pull complete 
23efb549476f: Pull complete 
aa2f8df21433: Pull complete 
ef072d3c9b41: Pull complete 
c9f371853f28: Pull complete 
a248b0871c3c: Pull complete 
749db6d368d0: Pull complete 
db2492acfcc3: Pull complete 
b7e7d2e12d53: Pull complete 
4272a53eef10: Pull complete 
9b2fefdb5321: Pull complete 
Digest: sha256:5b7c5e318303ad059f3d1a73d084c12cb39ae4f35f7391b79b0ff2c0ba45304b
Status: Downloaded newer image for tutum/influxdb:latest
docker.io/tutum/influxdb:latest
 
[root@localhost ~]# docker pull google/cadvisor
 
Using default tag: latest
latest: Pulling from google/cadvisor
ff3a5c916c92: Pull complete 
44a45bb65cdf: Pull complete 
0bbe1a2fe2a6: Pull complete 
Digest: sha256:815386ebbe9a3490f38785ab11bda34ec8dacf4634af77b8912832d4f85dca04
Status: Downloaded newer image for google/cadvisor:latest
docker.io/google/cadvisor:latest
 
[root@localhost ~]# docker pull grafana/grafana
 
Using default tag: latest
latest: Pulling from grafana/grafana
97518928ae5f: Pull complete 
5b58818b7f48: Pull complete 
d9a64d9fd162: Pull complete 
4e368e1b924c: Pull complete 
867f7fdd92d9: Pull complete 
387c55415012: Pull complete 
07f94c8f51cd: Pull complete 
ce8cf00ff6aa: Pull complete 
e44858b5f948: Pull complete 
4000fdbdd2a3: Pull complete 
Digest: sha256:18d94ae734accd66bccf22daed7bdb20c6b99aa0f2c687eea3ce4275fe275062
Status: Downloaded newer image for grafana/grafana:latest
docker.io/grafana/grafana:latest
 
[root@localhost ~]# docker images
 
REPOSITORY        TAG       IMAGE ID       CREATED         SIZE
grafana/grafana   latest    9b957e098315   20 months ago   275MB
google/cadvisor   latest    eb1210707573   4 years ago     69.6MB
tutum/influxdb    latest    c061e5808198   6 years ago     290MB

 创建自定义网络

[root@localhost ~]# docker network create monitor
 
bfe09a56d338b3173da008204dac13e46873ab8c46601683bba6457178e7dbcc
 
[root@localhost ~]# docker network ls
 
NETWORK ID     NAME      DRIVER    SCOPE
dbe5fb2353e8   bridge    bridge    local
fc4521838733   host      host      local
bfe09a56d338   monitor   bridge    local
4e2c57eec484   none      null      local

 

1.3、部署Cadvisor

被监控主机上部署Cadvisor容器

[root@localhost ~]# docker run -d \
> --volume=/:/rootfs:ro \
> --volume=/var/run:/var/run:ro \
> --volume=/sys:/sys:ro \
> --volume=/var/lib/docker/:/var/lib/docker:ro \
> --volume=/dev/disk/:/dev/disk:ro \
> --publish=8080:8080 \
> --detach=true \
> --name=cadvisor \
> google/cadvisor:latest

39067a18d264733e884272d932cd1d2bf1ae279092729aca01cfc0f12fa2571d

1.4、访问cAdvisor页面

访问http://192.168.2.118:8080 cAdvisor页面可以看到收集到的数据

 

 

1.5、Prometheus产品简介

        Prometheus是一个最初在SoundCloud上构建的开源系统监视和警报工具包。自2012年成立以来，很多公司和组织都采用了Prometheus，该项目拥有非常活跃的开发者和用户社区。 它现在是一个独立的开源项目，可以独立于任何公司进行维护。 为了强调这一点，并阐明项目的治理结构，Prometheus于2016年加入Cloud Native Computing Foundation（云原生基金会），作为继Kubernetes之后的第二个托管项目。

Prometheus的主要特征有：

• 多维度数据模型-由指标键值对标识的时间序列数据组成
• PromQL，一种灵活的查询语言
• 不依赖分布式存储; 单个服务器节点是自治的
• 以HTTP方式，通过pull模型拉取时间序列数据
• 支持通过中间网关推送时间序列数据
• 通过服务发现或者静态配置，来发现目标服务对象
• 支持多种多样的图表和界面展示

1.6、部署Prometheus 

[root@localhost ~]# docker pull prom/prometheus

Using default tag: latest
latest: Pulling from prom/prometheus
3cb635b06aa2: Pull complete 
34f699df6fe0: Pull complete 
33d6c9635e0f: Pull complete 
f2af7323bed8: Pull complete 
c16675a6a294: Pull complete 
827843f6afe6: Pull complete 
3d272942eeaf: Pull complete 
7e785cfa34da: Pull complete 
05e324559e3b: Pull complete 
170620261a59: Pull complete 
ec35f5996032: Pull complete 
5509173eb708: Pull complete 
Digest: sha256:cb9817249c346d6cfadebe383ed3b3cd4c540f623db40c4ca00da2ada45259bb
Status: Downloaded newer image for prom/prometheus:latest
docker.io/prom/prometheus:latest

先准备配置

[root@localhost ~]# vim /tmp/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'docker'      ##定义一个叫docker的组
    static_configs:
- targets: ['192.168.2.118:8080']   ##填写一个或多个cadvisor的主机地址用逗号隔开

运行容器

[root@localhost ~]# docker run -d \
--name=prometheus  -p 9090:9090  \
-v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \
 -v /etc/localtime:/etc/localtime \
prom/prometheus

402b019bb5fd489911f925ae717e87994e68ec7b836f37bc106de8f3c2801cbc


#如果起不来就重启一下docker，再用docker restart $(docker ps -qa) 重启所有容器

1.7、访问prometheus页面

http://192.168.2.118:9090

 

 

 

看到docker组状态up为正常

查询项是可以查到数据的

 1.8、部署Granfana

[root@localhost ~]# docker run -d \
> --name=grafana \
>  -p 3000:3000 \
> grafana/grafana

42b131dbf96ac1b613c6b8ac305c94eb322b8cf6b320f48990ad059bba5ba0d6

1.9、配置Granfana

访问http://192.168.2.118:3000默认账户admin 密码 admin首次登陆需要修改密码

 

 配置数据源

 

 

 

 

 

 导入模板

 

 选择对应的数据源，点击导入，就可以看到被监控主机的数据

 准备测试容器

[root@localhost ~]# docker run --name=nginx -d -p 80:80 nginx
d2000b50c5a035fde255c5f488fde43653733806a36167c4b73460a0c74d27f6

 

 

 到此Cadvisor +Prometheus+Grafana基本架构部署完毕