Prometheus学习

监控架构介绍：

基本架构：

Prometheus 和 Zabbix 的对比：

安装和使用：

• Prometheus 采集、存储数据
• Grafana 用于图表展示
• alertmanager 用于接收 Prometheus 发送的警告信息
• node-exporter 用于收集操作系统和硬件信息的 metrics

# 从创建一个专门的 prometheus 用户
useradd -M -s /usr/sbin/nologin prometheus
# 更改 prometheus 用户的文件权限
chown prometheus:prometheus -R /opt/prometheus

#  Prometheus 以 prometheus 用户身份运行
sudo chown -R prometheus:prometheus /opt/prometheus/prometheus/data
# 755 权限设置允许 Prometheus 用户读写目录，但其他用户只能读和执行（进入目录） 
sudo chmod -R 755 /opt/prometheus/prometheus/data

Prometheus Server 安装：

# 下载 prometheus 二进制压缩包
wget https://github.com/prometheus/prometheus/releases/download/v2.53.2/prometheus-2.53.2.linux-amd64.tar.gz

# 解压
tar -zxvf prometheus-2.53.2.linux-amd64.tar.gz

mkdir /opt/prometheus -p
mv prometheus-2.53.2.linux-amd64 /opt/prometheus/prometheus

# 创建 systemd 服务
cat > /etc/systemd/system/prometheus.service << "EOF"
[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/prometheus/prometheus \
  -- config.file=/opt/prometheus/prometheus/prometheus.yml \
  -- storage.tsdb.path=/opt/prometheus/prometheus/data \  # 数据存储的路径
  
  -- storage.tsdb.retention.time=60d \  # 数据存储的时长
  -- wed.enable-lifecycle  # 热加载配置

[Install]
WantedBy=multi-user.target
EOF

# 重启 Prometheus
systemctl daemon-reload
systemctl start prometheus.service
# 设置 Prometheus 为开机自启
systemctl enable prometheus.service
# 检查 Prometheus 服务的开启状态
systemctl status prometheus.service

# 查看 Prometheus 服务的日志，进行故障排除
journalctl -u prometheus.service -f 

prometheus 监控指标：http://192.168.10.6:9090/metrics

Alert Manager 安装：

wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
# 或者在本地下载，通过 scp 命令拷贝到远程 linux 服务端


# 解压
tar -zxvf alertmanager-0.27.0.linux-amd64.tar.gz

mkdir /opt/prometheus -p
mv alertmanager-0.27.0.linux-amd64 /opt/prometheus/alertmanager

# 更改 alertmanager 文件夹权限
chown prometheus:prometheus -R /opt/prometheus/alertmanager

# 创建 systemd 服务
cat > /etc/systemd/system/alertmanager.service << "EOF"

[Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/alertmanager/alertmanager \
  --config.file=/opt/prometheus/alertmanager/alertmanager.yml \
  --storage.path=/opt/prometheus/alertmanager/data

[Install]
WantedBy=multi-user.target

EOF

# 重启 Prometheus
systemctl daemon-reload
systemctl start alertmanager.service
# 设置 Prometheus 为开机自启
systemctl enable alertmanager.service
# 检查 Prometheus 服务的开启状态
systemctl status alertmanager.service

# 查看 alertmanager 服务的日志，进行故障排除
journalctl -u alertmanager.service -f 

Alertmanager 的指标：http://192.168.10.6:9093/metrics

# 修改 prometheus 配置：/opt/prometheus/prometheus/prometheus.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093 # 需根据实际情况填写 alertmanager 的地址

rule_files:
  - "alert.yml"   # 需根据实际修改文件名

# 增加触发器配置文件 /opt/prometheus/prometheus/alert.yml
cat > /opt/prometheus/prometheus/alert.yml << "EOF"
groups:
- name : Prometheus alert
  rules:
  # 对任何实例超过30s无法联系的情况，发送警报
  - alert: 服务警告
    expr: up == 0
    for: 30s
    labels:
      severity: critical
    annotations:
      instance: "{{ $labels.instance }}"
	  description: "{{ $labels.job }} 服务已关闭"
EOF
# 检查配置
cd /opt/prometheus/prometheus
./prometheus check config prometheus.yml

重启 prometheus 或 重新加载配置文件：

# 重启
systemctl restart prometheus

# 重载，需要 --web.enable-lifecycle 配置
curl -x POST http://localhost:9090/-/reload

Grafana 安装：

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.4.3.linux-amd64.tar.gz
tar -zxvf grafana-enterprise-9.4.3.linux-amd64.tar.gz

mv grafana-9.4.3 /opt/prometheus/grafana

chown prometheus:prometheus -R /opt/prometheus/grafana

cat > /etc/systemd/system/grafana-server.service << "EOF"
[Unit]
Description=Grafana Server
Documentation=http://docs.grafana.org

[Service]
Type=simple
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/grafana/bin/grafana-server \
  --config=/opt/prometheus/grafana/conf/defaults.ini \
  --homepath=/opt/prometheus/grafana

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server

# 查看 alertmanager 服务的日志，进行故障排除
journalctl -u grafana-server.service -f

Grafana 的访问地址：http://192.168.10.6:3000

Node_Exporter 安装：

wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -zxvf node_exporter-1.8.2.linux-amd64.tar.gz

mv node_exporter-1.8.2.linux-amd64 /opt/prometheus/node_exporter

# 更改 node_exporter 文件夹权限
chown prometheus:prometheus -R /opt/prometheus/node_exporter

cat > /etc/systemd/system/node_exporter.service << "EOF"
[Unit]
Description=node_exporter
Documentation=https://prometheus.io/
After=network.target
[Service]
User=prometheus
Group=prometheus
ExecStart=/opt/prometheus/node_exporter/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

systemctl start node_exporter
systemctl status node_exporter

# 查看 alertmanager 服务的日志，进行故障排除
journalctl -u node_exporter.service -f

Node_Exporter 的访问地址：http://192.168.11.61:9100

# 修改 prometheus 的配置
cat >> /opt/prometheus/prometheus/prometheus.yml << "EOF"
# 在 scrape_configs 这行下面添加如下配置：
  # node_exporter 配置
  - job_name: 'node_exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9100']
      labels:
        instance: Prometheus服务器
EOF

# 重载 Prometheus
systemctl restart prometheus
systemctl status prometheus

Prometheus Web 检查：http://192.168.10.6:9090/targets

通过 gitee 下载安装：

使用 grafana 展示 prometheus 的图形

PromQL：

快速入门：