Prometheus 容器化部署，配合Grafan画图工具监控节点

Prometheus 容器化部署，配合Grafan画图工具监控节点

一、部署环境

主机名	IP地址	服务
prometheus	192.168.85.131	prometheus、grafana
node-exporter	192.168.85.132	node_exporter

二、部署 Prometheus

准备工作

修改主机名

[root@localhost ~]# hostnamectl  set-hostname prometheus
[root@localhost ~]# bash

关闭防火墙

[root@prometheus ~]# systemctl disable --now firewalld

关闭selinux

[root@prometheus ~]# vim /etc/selinux/config 
[root@prometheus ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     disabled - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of disabled.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@prometheus ~]# reboot
[root@prometheus ~]# getenforce 
Disabled

首先配置yum仓库

[root@prometheus ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo

配置docker的yum源

[root@prometheus ~]# cd /etc/yum.repos.d/
[root@prometheus ~]# curl -o docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo
[root@prometheus ~]# sed -i 's@https://download.docker.com@https://mirrors.tuna.tsinghua.edu.cn/docker-ce@g' docker-ce.repo

安装docker-ce以及docker组件

[root@prometheus ~]# yum -y install docker-ce    

启动docker

[root@prometheus ~]# systemctl start docker
[root@prometheus ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-12-29 18:36:30 CST; 8min ago
     Docs: https://docs.docker.com
 Main PID: 1156 (dockerd)
    Tasks: 8
   Memory: 135.7M
   CGroup: /system.slice/docker.service
           └─1156 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

设置阿里云镜像加速

[root@prometheus ~]# vim /etc/docker/daemon.json
[root@prometheus ~]# cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://in3617d8.mirror.aliyuncs.com"]
}
[root@prometheus ~]# systemctl restart docker
[root@prometheus ~]# systemctl daemon-reload
[root@prometheus ~]# docker info 
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-257.el8.x86_64
 Operating System: CentOS Stream 8
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.622GiB
 Name: prometheus
 ID: Z3JR:D5TL:NVZ7:DJWE:7FL3:EC74:6DKV:6HOB:IPXV:FCQ3:MF4B:R3BG
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://in3617d8.mirror.aliyuncs.com/
 Live Restore Enabled: false

运行promethues容器

拉取官方prometheus镜像

[root@prometheus ~]# docker pull prom/prometheus  
Using default tag: latest
latest: Pulling from prom/prometheus
3cb635b06aa2: Pull complete 
34f699df6fe0: Pull complete 
33d6c9635e0f: Pull complete 
f2af7323bed8: Pull complete 
c16675a6a294: Pull complete 
827843f6afe6: Pull complete 
3d272942eeaf: Pull complete 
7e785cfa34da: Pull complete 
05e324559e3b: Pull complete 
170620261a59: Pull complete 
ec35f5996032: Pull complete 
5509173eb708: Pull complete 
Digest: sha256:cb9817249c346d6cfadebe383ed3b3cd4c540f623db40c4ca00da2ada45259bb
Status: Downloaded newer image for prom/prometheus:latest
docker.io/prom/prometheus:latest

创建存放prometheus配置文件的目录，并提供默认配置文件

[root@prometheus ~]# mkdir -p  /prometheus/config/
[root@prometheus ~]# cd /prometheus/config/
[root@prometheus config]# vim prometheus.yml 
[root@prometheus config]# cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ["localhost:9090"]

使用官方promethrus镜像创建容器

[root@prometheus ~]# docker images
REPOSITORY        TAG       IMAGE ID       CREATED       SIZE
prom/prometheus   latest    a3d385fc29f9   11 days ago   201MB
[root@prometheus ~]# docker run --name prometheus -d --restart always  -p 9090:9090 -v /prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml   prom/prometheus:latest
7c224924088bc66c18f11549225e92677646ee5bbf5010340d9f57ddbd407cfc
[root@prometheus ~]# 
[root@prometheus ~]# 
[root@prometheus ~]# docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED         STATUS         PORTS                                       NAMES
7c224924088b   prom/prometheus:latest   "/bin/prometheus --c…"   5 seconds ago   Up 4 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus

查看端口号

[root@prometheus ~]# ss -antl
State         Recv-Q        Send-Q               Local Address:Port                Peer Address:Port        Process        
LISTEN        0             128                        0.0.0.0:9090                     0.0.0.0:*                          
LISTEN        0             128                        0.0.0.0:22                       0.0.0.0:*                          
LISTEN        0             128                           [::]:9090                        [::]:*                          
LISTEN        0             128                           [::]:22                          [::]:*                        

使用本机IP地址192.168.85.131 + 端口号9090/targets在浏览器中访问

三、部署 node_exporter

准备工作

修改主机名

[root@localhost ~]#  hostnamectl  set-hostname node-exporter
[root@localhost ~]# bash

关闭防火墙

[root@node-exporter ~]# systemctl disable --now firewalld

关闭selinux

[root@node-exporter ~]# vim /etc/selinux/config 
[root@node-exporter ~]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     disabled - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of disabled.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@node-exporter ~]# reboot
[root@node-exporter ~]# getenforce 
Disabled

部署node_exporter

下载node_exporter

[root@node-exporter ~]# cd /usr/src
[root@node-exporter src]# wget https://github.com/prometheus/node_exporter/releases/download/v1.3.0/node_exporter-1.3.0.linux-amd64.tar.gz
--2021-12-29 19:23:02--  https://github.com/prometheus/node_exporter/releases/download/v1.3.0/node_exporter-1.3.0.linux-amd64.tar.gz

解压并重命名

[root@node-exporter src]# ls
debug  kernels  node_exporter-1.3.0.linux-amd64.tar.gz
[root@node-exporter src]# tar -xf node_exporter-1.3.0.linux-amd64.tar.gz  -C /usr/local/
[root@node-exporter src]# cd /usr/local/
[root@node-exporter local]# ls
bin  etc  games  include  lib  lib64  libexec  node_exporter-1.3.0.linux-amd64  sbin  share  src
[root@node-exporter local]# mv node_exporter-1.3.0.linux-amd64/ node_exporter
[root@node-exporter local]# ls
bin  etc  games  include  lib  lib64  libexec  node_exporter  sbin  share  src

编写service文件

[root@node-exporter ~]# cat > /usr/lib/systemd/system/node_exporter.service   <<EOF
[unit]
Description=The node_exporter Server
After=network.target

[Service]
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure
RestartSec=15s
SyslogIdentifier=node_exporter

[Install]
WantedBy=multi-user.target

EOF

启动服务

[root@node-exporter ~]# systemctl daemon-reload
[root@node-exporter ~]# systemctl enable --now node_exporter
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /usr/lib/systemd/system/node_exporter.service.
[root@node-exporter ~]# systemctl status node_exporter
● node_exporter.service
   Loaded: loaded (/usr/lib/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-12-29 19:27:22 CST; 1min 2s ago
 Main PID: 89713 (node_exporter)
    Tasks: 4 (limit: 23485)
   Memory: 8.8M
   CGroup: /system.slice/node_exporter.service
           └─89713 /usr/local/node_exporter/node_exporter

查看都口号

[root@node-exporter ~]# ss -antl
State         Recv-Q        Send-Q               Local Address:Port                Peer Address:Port        Process        
LISTEN        0             128                        0.0.0.0:22                       0.0.0.0:*                          
LISTEN        0             128                              *:9100                           *:*                          
LISTEN        0             128                           [::]:22                          [::]:*                       

四、添加节点到prometheus中

修改/prometheus/config目录下的prometheus配置文件prometheus.yml

[root@prometheus ~]# cd /prometheus/config/
[root@prometheus config]# vim prometheus.yml 
[root@prometheus config]# cat prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ["localhost:9090"]

  - job_name: "node1"          #添加节点工作名称       
    static_configs:
    - targets: ["192.168.85.132:9100"]   #添加node）exporter节点IP地址和端口号

重启prometheus容器

[root@prometheus ~]# docker restart prometheus
prometheus
[root@prometheus ~]# docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS         PORTS                                       NAMES
7c224924088b   prom/prometheus:latest   "/bin/prometheus --c…"   55 minutes ago   Up 7 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus

使用prometheus主机IP地址192.168.85.131 + 端口号9090/targets在浏览器中访问

五、部署grafana画图工具

拉取grafan/grafan官方镜像

[root@prometheus ~]# docker pull grafana/grafana
Using default tag: latest
latest: Pulling from grafana/grafana
97518928ae5f: Already exists 
5b58818b7f48: Already exists 
d9a64d9fd162: Already exists 
4e368e1b924c: Already exists 
867f7fdd92d9: Already exists 
387c55415012: Already exists 
07f94c8f51cd: Pull complete 
ce8cf00ff6aa: Pull complete 
e44858b5f948: Pull complete 
4000fdbdd2a3: Pull complete 
Digest: sha256:18d94ae734accd66bccf22daed7bdb20c6b99aa0f2c687eea3ce4275fe275062
Status: Downloaded newer image for grafana/grafana:latest
docker.io/grafana/grafana:latest
[root@prometheus ~]# docker images
REPOSITORY        TAG       IMAGE ID       CREATED       SIZE
prom/prometheus   latest    a3d385fc29f9   11 days ago   201MB
grafana/grafana   latest    9b957e098315   2 weeks ago   275MB

使用官方grafana镜像运行容器

[root@prometheus ~]# docker run -d --name grafan -p 3000:3000 grafana/grafana
39f8ffa8c45b8b57a61e4526118384c15b56539bd878154a0a129edf60393849
[root@prometheus ~]# docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED             STATUS          PORTS                                       NAMES
39f8ffa8c45b   grafana/grafana          "/run.sh"                5 seconds ago       Up 4 seconds    0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   grafan
7c224924088b   prom/prometheus:latest   "/bin/prometheus --c…"   About an hour ago   Up 20 minutes   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus
[root@prometheus ~]# ss -antl
State         Recv-Q        Send-Q               Local Address:Port                Peer Address:Port        Process        
LISTEN        0             128                        0.0.0.0:3000                     0.0.0.0:*                          
LISTEN        0             128                        0.0.0.0:9090                     0.0.0.0:*                          
LISTEN        0             128                        0.0.0.0:22                       0.0.0.0:*                          
LISTEN        0             128                           [::]:3000                        [::]:*                          
LISTEN        0             128                           [::]:9090                        [::]:*                          
LISTEN        0             128                           [::]:22                          [::]:* 

使用prometheus主机IP地址192.168.85.131 + 端口号3000在浏览器中访问

登录用户和密码都是 admin，输入后提示修改密码

进入首页

配置数据源






导入仪表盘模板
仪表板模板下载地址