Prometheus 容器化部署,配合Grafan画图工具监控节点
一、部署环境
主机名 | IP地址 | 服务 |
---|---|---|
prometheus | 192.168.85.131 | prometheus、grafana |
node-exporter | 192.168.85.132 | node_exporter |
二、部署 Prometheus
准备工作
修改主机名
[root@localhost ~]# hostnamectl set-hostname prometheus
[root@localhost ~]# bash
关闭防火墙
[root@prometheus ~]# systemctl disable --now firewalld
关闭selinux
[root@prometheus ~]# vim /etc/selinux/config
[root@prometheus ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# disabled - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of disabled.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@prometheus ~]# reboot
[root@prometheus ~]# getenforce
Disabled
首先配置yum仓库
[root@prometheus ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo
配置docker的yum源
[root@prometheus ~]# cd /etc/yum.repos.d/
[root@prometheus ~]# curl -o docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo
[root@prometheus ~]# sed -i 's@https://download.docker.com@https://mirrors.tuna.tsinghua.edu.cn/docker-ce@g' docker-ce.repo
安装docker-ce以及docker组件
[root@prometheus ~]# yum -y install docker-ce
启动docker
[root@prometheus ~]# systemctl start docker
[root@prometheus ~]# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-12-29 18:36:30 CST; 8min ago
Docs: https://docs.docker.com
Main PID: 1156 (dockerd)
Tasks: 8
Memory: 135.7M
CGroup: /system.slice/docker.service
└─1156 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
设置阿里云镜像加速
[root@prometheus ~]# vim /etc/docker/daemon.json
[root@prometheus ~]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://in3617d8.mirror.aliyuncs.com"]
}
[root@prometheus ~]# systemctl restart docker
[root@prometheus ~]# systemctl daemon-reload
[root@prometheus ~]# docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
scan: Docker Scan (Docker Inc., v0.12.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc version: v1.0.2-0-g52b36a2
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 4.18.0-257.el8.x86_64
Operating System: CentOS Stream 8
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.622GiB
Name: prometheus
ID: Z3JR:D5TL:NVZ7:DJWE:7FL3:EC74:6DKV:6HOB:IPXV:FCQ3:MF4B:R3BG
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://in3617d8.mirror.aliyuncs.com/
Live Restore Enabled: false
运行promethues容器
拉取官方prometheus镜像
[root@prometheus ~]# docker pull prom/prometheus
Using default tag: latest
latest: Pulling from prom/prometheus
3cb635b06aa2: Pull complete
34f699df6fe0: Pull complete
33d6c9635e0f: Pull complete
f2af7323bed8: Pull complete
c16675a6a294: Pull complete
827843f6afe6: Pull complete
3d272942eeaf: Pull complete
7e785cfa34da: Pull complete
05e324559e3b: Pull complete
170620261a59: Pull complete
ec35f5996032: Pull complete
5509173eb708: Pull complete
Digest: sha256:cb9817249c346d6cfadebe383ed3b3cd4c540f623db40c4ca00da2ada45259bb
Status: Downloaded newer image for prom/prometheus:latest
docker.io/prom/prometheus:latest
创建存放prometheus配置文件的目录,并提供默认配置文件
[root@prometheus ~]# mkdir -p /prometheus/config/
[root@prometheus ~]# cd /prometheus/config/
[root@prometheus config]# vim prometheus.yml
[root@prometheus config]# cat prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
使用官方promethrus镜像创建容器
[root@prometheus ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
prom/prometheus latest a3d385fc29f9 11 days ago 201MB
[root@prometheus ~]# docker run --name prometheus -d --restart always -p 9090:9090 -v /prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:latest
7c224924088bc66c18f11549225e92677646ee5bbf5010340d9f57ddbd407cfc
[root@prometheus ~]#
[root@prometheus ~]#
[root@prometheus ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c224924088b prom/prometheus:latest "/bin/prometheus --c…" 5 seconds ago Up 4 seconds 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
查看端口号
[root@prometheus ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:9090 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:9090 [::]:*
LISTEN 0 128 [::]:22 [::]:*
使用本机IP地址192.168.85.131 + 端口号9090/targets在浏览器中访问
三、部署 node_exporter
准备工作
修改主机名
[root@localhost ~]# hostnamectl set-hostname node-exporter
[root@localhost ~]# bash
关闭防火墙
[root@node-exporter ~]# systemctl disable --now firewalld
关闭selinux
[root@node-exporter ~]# vim /etc/selinux/config
[root@node-exporter ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# disabled - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of disabled.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@node-exporter ~]# reboot
[root@node-exporter ~]# getenforce
Disabled
部署node_exporter
下载node_exporter
[root@node-exporter ~]# cd /usr/src
[root@node-exporter src]# wget https://github.com/prometheus/node_exporter/releases/download/v1.3.0/node_exporter-1.3.0.linux-amd64.tar.gz
--2021-12-29 19:23:02-- https://github.com/prometheus/node_exporter/releases/download/v1.3.0/node_exporter-1.3.0.linux-amd64.tar.gz
解压并重命名
[root@node-exporter src]# ls
debug kernels node_exporter-1.3.0.linux-amd64.tar.gz
[root@node-exporter src]# tar -xf node_exporter-1.3.0.linux-amd64.tar.gz -C /usr/local/
[root@node-exporter src]# cd /usr/local/
[root@node-exporter local]# ls
bin etc games include lib lib64 libexec node_exporter-1.3.0.linux-amd64 sbin share src
[root@node-exporter local]# mv node_exporter-1.3.0.linux-amd64/ node_exporter
[root@node-exporter local]# ls
bin etc games include lib lib64 libexec node_exporter sbin share src
编写service文件
[root@node-exporter ~]# cat > /usr/lib/systemd/system/node_exporter.service <<EOF
[unit]
Description=The node_exporter Server
After=network.target
[Service]
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure
RestartSec=15s
SyslogIdentifier=node_exporter
[Install]
WantedBy=multi-user.target
EOF
启动服务
[root@node-exporter ~]# systemctl daemon-reload
[root@node-exporter ~]# systemctl enable --now node_exporter
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /usr/lib/systemd/system/node_exporter.service.
[root@node-exporter ~]# systemctl status node_exporter
● node_exporter.service
Loaded: loaded (/usr/lib/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-12-29 19:27:22 CST; 1min 2s ago
Main PID: 89713 (node_exporter)
Tasks: 4 (limit: 23485)
Memory: 8.8M
CGroup: /system.slice/node_exporter.service
└─89713 /usr/local/node_exporter/node_exporter
查看都口号
[root@node-exporter ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 *:9100 *:*
LISTEN 0 128 [::]:22 [::]:*
四、添加节点到prometheus中
修改/prometheus/config目录下的prometheus配置文件prometheus.yml
[root@prometheus ~]# cd /prometheus/config/
[root@prometheus config]# vim prometheus.yml
[root@prometheus config]# cat prometheus.yml
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090"]
- job_name: "node1" #添加节点工作名称
static_configs:
- targets: ["192.168.85.132:9100"] #添加node)exporter节点IP地址和端口号
重启prometheus容器
[root@prometheus ~]# docker restart prometheus
prometheus
[root@prometheus ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c224924088b prom/prometheus:latest "/bin/prometheus --c…" 55 minutes ago Up 7 seconds 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
使用prometheus主机IP地址192.168.85.131 + 端口号9090/targets在浏览器中访问
五、部署grafana画图工具
拉取grafan/grafan官方镜像
[root@prometheus ~]# docker pull grafana/grafana
Using default tag: latest
latest: Pulling from grafana/grafana
97518928ae5f: Already exists
5b58818b7f48: Already exists
d9a64d9fd162: Already exists
4e368e1b924c: Already exists
867f7fdd92d9: Already exists
387c55415012: Already exists
07f94c8f51cd: Pull complete
ce8cf00ff6aa: Pull complete
e44858b5f948: Pull complete
4000fdbdd2a3: Pull complete
Digest: sha256:18d94ae734accd66bccf22daed7bdb20c6b99aa0f2c687eea3ce4275fe275062
Status: Downloaded newer image for grafana/grafana:latest
docker.io/grafana/grafana:latest
[root@prometheus ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
prom/prometheus latest a3d385fc29f9 11 days ago 201MB
grafana/grafana latest 9b957e098315 2 weeks ago 275MB
使用官方grafana镜像运行容器
[root@prometheus ~]# docker run -d --name grafan -p 3000:3000 grafana/grafana
39f8ffa8c45b8b57a61e4526118384c15b56539bd878154a0a129edf60393849
[root@prometheus ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
39f8ffa8c45b grafana/grafana "/run.sh" 5 seconds ago Up 4 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp grafan
7c224924088b prom/prometheus:latest "/bin/prometheus --c…" About an hour ago Up 20 minutes 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp prometheus
[root@prometheus ~]# ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:3000 0.0.0.0:*
LISTEN 0 128 0.0.0.0:9090 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:3000 [::]:*
LISTEN 0 128 [::]:9090 [::]:*
LISTEN 0 128 [::]:22 [::]:*
使用prometheus主机IP地址192.168.85.131 + 端口号3000在浏览器中访问
登录用户和密码都是 admin,输入后提示修改密码
进入首页
配置数据源
导入仪表盘模板
仪表板模板下载地址