prometheus + grafana
prometheus 介绍
Prometheus(普罗米修斯)是一套开源的监控&报警&时间序列数据库的组合, 由go语言开发。适合监控容器平台, 因为kubernetes(俗称k8s)的流行带动了prometheus的发展。
数据库分类:
关系型 mysql,oracle,sql server,sybase,db2,access等
非关系型(nosql)
key-value memcache redis etcd
文档型 mongodb elasticsearch
列式 hbase
时序 prometheus
图形数据库 Neo4j
时序数据主要的特点:
数据带有时间属性,且数据量随着时间递增
大都为插入操作较多且无更新的需求,插入数据多,每秒钟插入数据可到达千万甚至是上亿条
分析过去时序数据可以做成多纬度报表,揭示其趋势性、规律性、异常性
分析时序数据趋势可以做大数据分析,机器学习,实现预测和预警
能够按照条件筛选数据, 也可以按照时间范围统计,聚合,展示数据
Prometheus的主要特性有:
多维度数据模型
灵活的查询语言
不依赖分布式存储,单个服务器节点是自主的
以HTTP方式,通过pull模型拉去时间序列数据
也可以通过中间网关支持push模型
通过服务发现或者静态配置, 来发现目标服务对象
支持多种多样的图表和界面展示
部署前期:
时钟同步
[root@node1 ~]# yum -y install chrony
[root@node1 ~]# systemctl restart chronyd
[root@node1 ~]# systemctl enable chronyd
[root@node1 ~]# timedatectl
Local time: Fri 2024-08-16 11:26:40 CST
Universal time: Fri 2024-08-16 03:26:40 UTC
RTC time: Fri 2024-08-16 03:26:39
Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
修改主机名
[root@node1 ~]# hostnamectl hostname server.example.com
[root@node1 ~]# bash
[root@server ~]#
[root@node1 ~]# hostnamectl hostname agent1.example.com
[root@node1 ~]# bash
[root@agent1 ~]#
[root@node1 ~]# hostnamectl hostname grafana.example.com
[root@node1 ~]# bash
[root@grafana ~]#
添加/etc/hosts
[root@server ~]# systemctl restart chronyd
[root@server ~]# systemctl enable chronyd
[root@server ~]# timedatectl
Local time: Fri 2024-08-16 15:52:08 CST
Universal time: Fri 2024-08-16 07:52:08 UTC
RTC time: Fri 2024-08-16 07:52:08
Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
[root@server ~]# hwclock -w
prometheus 的部署
[root@server ~]# ls
'!' anaconda-ks.cfg -e -i.bak prometheus-2.54.0.linux-amd64.tar.gz
[root@server ~]# tar -zxvf prometheus-2.54.0.linux-amd64.tar.gz -C /usr/local/
prometheus-2.54.0.linux-amd64/
prometheus-2.54.0.linux-amd64/NOTICE
prometheus-2.54.0.linux-amd64/LICENSE
prometheus-2.54.0.linux-amd64/prometheus.yml
prometheus-2.54.0.linux-amd64/prometheus
prometheus-2.54.0.linux-amd64/consoles/
prometheus-2.54.0.linux-amd64/consoles/prometheus-overview.html
prometheus-2.54.0.linux-amd64/consoles/node-overview.html
prometheus-2.54.0.linux-amd64/consoles/index.html.example
prometheus-2.54.0.linux-amd64/consoles/node.html
prometheus-2.54.0.linux-amd64/consoles/node-disk.html
prometheus-2.54.0.linux-amd64/consoles/prometheus.html
prometheus-2.54.0.linux-amd64/consoles/node-cpu.html
prometheus-2.54.0.linux-amd64/promtool
prometheus-2.54.0.linux-amd64/console_libraries/
prometheus-2.54.0.linux-amd64/console_libraries/menu.lib
prometheus-2.54.0.linux-amd64/console_libraries/prom.lib
[root@server ~]# cd /usr/local/
[root@server local]# ls
bin etc games include lib lib64 libexec prometheus-2.54.0.linux-amd64 sbin share src
[root@server local]# mv prometheus-2.54.0.linux-amd64/ prometheus
[root@server local]# ls
bin etc games include lib lib64 libexec prometheus sbin share src
[root@server local]#
[root@server ~]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
[root@server ~]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 4096 *:9090 *:*
监控其他主机
[root@agent1 ~]# tar -zxvf node_exporter-1.8.2.linux-amd64.tar.gz -C /usr/local
node_exporter-1.8.2.linux-amd64/
node_exporter-1.8.2.linux-amd64/NOTICE
node_exporter-1.8.2.linux-amd64/node_exporter
node_exporter-1.8.2.linux-amd64/LICENSE
[root@agent1 ~]# cd /usr/local/
[root@agent1 local]# ls
bin etc games include lib lib64 libexec node_exporter-1.8.2.linux-amd64 sbin share src
[root@agent1 local]# mv node_exporter-1.8.2.linux-amd64/ node_exporter
[root@agent1 local]# ls
bin etc games include lib lib64 libexec node_exporter sbin share src
[root@agent1 local]#
[root@agent1 local]# nohup /usr/local/node_exporter/node_exporter &
[root@agent1 local]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 4096 *:9100 *:*
LISTEN 0 128 [::]:22 [::]:*
[root@agent1 local]#
添加到服务器(监控)
[root@server ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: "agent1"
static_configs:
- targets: ["192.168.100.80:9100"]
[root@server ~]# ps -ef | grep prometheus
root 8837 673 0 20:47 pts/0 00:00:00 /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
root 51926 673 0 21:04 pts/0 00:00:00 grep --color=auto prometheus
[root@server ~]# kill -9 8837
[root@server ~]# ps -ef | grep prometheus
root 52528 673 0 21:04 pts/0 00:00:00 grep --color=auto prometheus
[1]+ Killed /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml"
[root@server ~]#
[root@server ~]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
[root@server ~]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 4096 *:9090 *:*
监控远程mysql
[root@agent1 ~]# tar -zxvf mysqld_exporter-0.15.1.linux-amd64.tar.gz -C /usr/local
mysqld_exporter-0.15.1.linux-amd64/
mysqld_exporter-0.15.1.linux-amd64/LICENSE
mysqld_exporter-0.15.1.linux-amd64/mysqld_exporter
mysqld_exporter-0.15.1.linux-amd64/NOTICE
[root@agent1 ~]# cd /usr/local/
[root@agent1 local]# ls
bin games lib libexec node_exporter sbin src
etc include lib64 mysqld_exporter-0.15.1.linux-amd64 nohup.out share
[root@agent1 local]# mv mysqld_exporter-0.15.1.linux-amd64/ mysqld_exporter
[root@agent1 local]# ls
bin etc games include lib lib64 libexec mysqld_exporter node_exporter nohup.out sbin share src
[root@agent1 local]# yum -y install mariadb-server
[root@agent1 local]# systemctl restart mariadb.service
[root@agent1 local]# systemctl enable mariadb.service
Created symlink /etc/systemd/system/mysql.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/mysqld.service → /usr/lib/systemd/system/mariadb.service.
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /usr/lib/systemd/system/mariadb.service.
[root@agent1 local]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 3
Server version: 10.5.22-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> grant all ON *.* to 'mysql_monitor'@'localhost' identified by
-> '123';
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]>exit
[root@agent1 local]# vim /usr/local/mysqld_exporter/.my.cnf
[root@agent1 local]# cat /usr/local/mysqld_exporter/.my.cnf
[client]
user=mysql_monitor
password=123
[root@agent1 local]#
[root@agent1 local]# nohup /usr/local/mysqld_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysqld_exporter/.my.cnf &
[root@agent1 local]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 80 *:3306 *:*
LISTEN 0 4096 *:9100 *:*
LISTEN 0 4096 *:9104 *:*
LISTEN 0 128 [::]:22 [::]:*
[root@agent1 local]#
[root@server ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: "agent1_mariadb"
static_configs:
- targets: ["192.168.100.80:9104"]
[root@server ~]# /usr/local/prometheus/prometheus --config.file="/usr/local/prometheus/prometheus.yml" &
安装grafana
模板位置:https://grafana.com/grafana/dashboards
[root@grafana ~]# ls
anaconda-ks.cfg -e grafana-6.4.2-1.x86_64.rpm -i.bak
[root@grafana ~]# rpm -ivh grafana-6.4.2-1.x86_64.rpm
warning: grafana-6.4.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
error: Failed dependencies:
fontconfig is needed by grafana-6.4.2-1.x86_64
freetype is needed by grafana-6.4.2-1.x86_64
urw-fonts is needed by grafana-6.4.2-1.x86_64
[root@grafana ~]# yum -y install fontconfig freetype urw-fonts
[root@grafana ~]# rpm -ivh grafana-6.4.2-1.x86_64.rpm
[root@grafana ~]# systemctl restart grafana-server
[root@grafana ~]# systemctl enable grafana-server
Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service.
[root@grafana ~]# ss -anlt
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
LISTEN 0 4096 *:3000 *:*
[root@grafana ~]#
grafana+onealert报警
智能告警平台 – 设置名称领取AppKey – 新建监控策略