Prometheus+Grafana监控服务器、mysql数据库并配置报警规则推送邮箱

一、安装prometheus

1.1下载

https://prometheus.io/download/

1.2 安装

tar -zxvf prometheus-2.52.0.linux-amd64.tar.gz	
cp -R prometheus-2.47.2.linux-amd64 /usr/local/prometheus
vim /usr/lib/systemd/system/prometheus.service

prometheus.service配置文件内容

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/usr/local/prometheus
ExecStart=/usr/local/prometheus/prometheus 
Restart=on-failure

[Install]
WantedBy=multi-user.target

1.3 开机启动

systemctl daemon-reload
systemctl enable prometheus.service
systemctl start prometheus.service

1.4 验证

http://192.168.1.106:9090/

二、安装 Grafana

2.1 下载

官网

https://grafana.com/grafana/download

下载地址：https://dl.grafana.com/enterprise/release/grafana-enterprise-11.0.0-1.x86_64.rpm

2.2 安装

yum install -y grafana-enterprise-11.0.0-1.x86_64.rpm

2.3 启动

# 开机启动
systemctl enable grafana-server
# 启动
systemctl start grafana-server

2.4 验证

1. 地址：http://192.168.1.106:3000/
2. 默认密码：admin/admin
3. 登录后要求修改原始密码

三、安装服务器监控 node_exporter

3.1 下载

地址：https://github.com/prometheus/node_exporter

下载地址：https://objects.githubusercontent.com/github-production-release-asset-2e65be/9524057/e07e4ee4-e4b0-48dc-9c04-eaad890c81b3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240524T020509Z&X-Amz-Expires=300&X-Amz-Signature=4cc9ade7610f887ebff68dc1ff00fa7b3d6c9ba945a2c86f88de8e8c2366afff&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=9524057&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-1.8.1.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream

3.2 安装

tar -zxvf node_exporter-1.8.1.linux-amd64.tar.gz
cp -R node_exporter-1.8.1.linux-amd64 /usr/local/node_exporter

3.3 设置 node_exporter 系统服务

vim /usr/lib/systemd/system/node_exporter.service

[Unit]
Description=node_exporter
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/node_exporter/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

3.4 设置开机自动启动

systemctl daemon-reload
systemctl enable node_exporter.service
systemctl start node_exporter.service

# 查看服务状态
systemctl status node_exporter.service

3.5 验证

访问地址：http://192.168.1.106:9100

3.6配置Prometheus

3.7 修改 Prometheus 的配置文件，添加 node_exporter 的绑定

vim /usr/local/prometheus/prometheus.yml

  - job_name: 'centos-106'
    static_configs:
      - targets: ['192.168.1.106:9100']

3.8 重启 Prometheus

systemctl restart prometheus

3.9 在 Grafana 中添加数据源

3.9.1添加数据源

3.9.2数据源选择prometheus

3.9.3 填写prometheus地址

3.9.4选择仪表盘

四、安装mysql监控 mysqld_exporter

4.1 下载

地址：https://github.com/prometheus/mysqld_exporter

mysqld_exporter-0.15.1.linux-amd64.tar.gz

4.2 安装

tar -zxvf mysqld_exporter-0.15.1.linux-amd64.tar.gz
mv mysqld_exporter-0.15.1.linux-amd64 /usr/local/mysql_exporter
cd /usr/local/mysql_exporter
vim /usr/local/mysql_exporter/my-exporter.cnf

[client]
user=root
password=xxxx
host=localhost
port=3306

4.3 注册为系统服务

vim /etc/systemd/system/mysqld_exporter.service

[Unit]
Description=mysqld_exporter
After=network.target
[Service]
Type=simple
User=root
ExecStart=/usr/local/mysql_exporter/mysqld_exporter --config.my-cnf=/usr/local/mysql_exporter/my-exporter.cnf
Restart=on-failure
[Install]
WantedBy=multi-user.target

4.4 启动

# 重新加载系统配置
systemctl daemon-reload
# 开机启动
systemctl enable mysqld_exporter.service
# 启动服务
systemctl start mysqld_exporter
# 查看状态
systemctl status mysqld_exporter

4.5 配置prometheus

vim /usr/local/prometheus/prometheus.yml

- job_name: 'mysql_exporter'
    static_configs:
      - labels:
          instance: master:3306 # 主库标签
      - targets:
        - localhost:9104 # mysqld_exporter暴露的端口
      - labels:
          instance: slave:3306 # 从库标签
      - targets:
        - 172.17.191.255:9104

4.6 配置grafana

4.6.1 创建Dashboard

4.6.2 展示效果

五、配置邮箱报警

5.1 配置发送邮箱

vim /etc/grafana/grafana.ini
# 重启
systemctl restart grafana-server

[smtp]
enabled = true
host = smtp.exmail.qq.com
user = system@qeoten.com
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = NHo4QJEbnNoaU3Jd
;cert_file =
;key_file =
;skip_verify = false
from_address = system@qeoten.com
from_name = zheJiang
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
# SMTP startTLS policy (defaults to 'OpportunisticStartTLS')
;startTLS_policy = NoStartTLS
# Enable trace propagation in e-mail headers, using the 'traceparent', 'tracestate' and (optionally) 'baggage' fields (defaults to false)
;enable_tracing = false

5.2 发送邮箱配置

开启客户端账号密码登录，腾讯邮箱为例

5.3 配置报警规则

需求cpu使用率超过90%，且持续10分钟，触发报警