一、下载安装包
1、prometheus
- 下载地址:https://github.com/prometheus/prometheus/releases/download/v2.32.1/prometheus-2.32.1.linux-amd64.tar.gz
2、grafana
- 下载地址:https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.3.linux-amd64.tar.gz
- 或者:https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.3-1.x86_64.rpm
3、node_exporter
- 下载地址:https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
4、mysqld_exporter
- 下载地址:https://github.com/prometheus/mysqld_exporter/releases/download/v0.13.0/mysqld_exporter-0.13.0.linux-amd64.tar.gz
5、redis_exporter
- 下载地址:https://github.com/oliver006/redis_exporter/releases/download/v1.33.0/redis_exporter-v1.33.0.linux-amd64.tar.gz
6、blackbox_exporter
- 下载地址:https://github.com/prometheus/blackbox_exporter/releases/download/v0.19.0/blackbox_exporter-0.19.0.linux-amd64.tar.gz
7、docker_exporter
- docker_exporter为容器启动,需要将镜像导出tar包,并导入到目标服务器上
二、安装并启动
- 需要先将下载好的tar.gz包上传至内网服务器
1、安装docker_exporter
#导出镜像
docker pull prometheusnet/docker_exporter:latest
docker save docker_exporter.tar prometheusnet/docker_exporter:latest
#导入镜像
docker load -i docker_exporter.tar
#启动docker_exporter
docker run -it -d --name docker_exporter --restart always --volume "/var/run/docker.sock":"/var/run/docker.sock" --publish 9417:9417 prometheusnet/docker_exporter
#开放防火墙
iptables -I INPUT -p tcp -s 10.10.0.170 --dport=9417 -j ACCEPT
2、安装redis_exporter
tar xf redis_exporter-v1.33.0.linux-amd64.tar.gz
mv redis_exporter-v1.33.0.linux-amd64 redis_exporter
cd redis_exporter
nohup ./redis_exporter -redis.addr=0.0.0.0:6379 -redis.password=IJ3P8MLb > /var/log/redis_exporter.log 2>&1 &
iptables -I INPUT -p tcp -s 10.10.0.170 --dport=9121 -j ACCEPT
3、安装node_exporter
tar xf node_exporter-1.3.1.linux-amd64.tar.gz
mv node_exporter-1.3.1.linux-amd64 node_exporter
cd node_exporter
nohup ./node_exporter --web.listen-address=":9100" > /var/log/node_exporter.log 2>&1 &
iptables -I INPUT -p tcp -s 10.10.0.170 --dport=9100 -j ACCEPT
4、安装blackbox_exporter
tar xf blackbox_exporter-0.19.0.linux-amd64.tar.gz
mv blackbox_exporter-0.19.0.linux-amd64 blackbox_exporter
cd blackbox_exporter
nohup ./blackbox_exporter --config.file="blackbox.yml" > /var/log/blackbox_exporter.log 2>&1 &
iptables -I INPUT -p tcp -s 10.10.0.170 --dport=9115 -j ACCEPT
5、安装mysqld_exporter
tar xf mysqld_exporter-0.13.0.linux-amd64.tar.gz
mv mysqld_exporter-0.13.0.linux-amd64 mysqld_exporter
cd mysqld_exporter
#配置数据库监控用户
授权
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'Your@Passw0rd' WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
flush privileges;
配置连接信息
cat > ~/.my.cnf<< EOF
[client]
user=exporter
password=Your@Passw0rd
EOF
#启动
nohup ./mysqld_exporter --web.listen-address=":9104" --config.my-cnf="/root/.my.cnf" > /var/log/mysqld_exporter.log 2>&1 &
iptables -I INPUT -p tcp -s 10.10.0.170 --dport=9104 -j ACCEPT
6、安装prometheus
tar xf prometheus-2.32.1.linux-amd64.tar.gz
mv prometheus-2.32.1.linux-amd64 prometheus
cd prometheus
nohup ./prometheus --config.file="prometheus.yml" --web.listen-address="0.0.0.0:9090" --log.level=info --web.enable-lifecycle --storage.tsdb.path="/data/prometheus" --storage.tsdb.retention=30d > /var/log/prometheus.log 2>&1 &
iptables -I INPUT -p tcp -s 10.10.0.170 --dport=9090 -j ACCEPT
7、安装grafana
yum安装:
yum -y install grafana-enterprise-8.3.3-1.x86_64.rpm
systemctl start grafana && systemctl enable grafana
二进制文件安装:
tar xf grafana-enterprise-8.3.3.linux-amd64.tar.gz
mv grafana-enterprise-8.3.3.linux-amd64 grafana
cd grafana/bin
nohup ./grafana-server > /var/log/grafana.log 2>&1 &
iptables -I INPUT -p tcp -s 0.0.0.0 --dport=3000 -j ACCEPT
三、配置prometheus
1、配置文件 prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx] # Look for a HTTP 200 response.
static_configs:
- targets:
- http://prometheus.io # 需要监控的应用接口(get请求)
- https://prometheus.io # 需要监控的应用接口(get请求)
- http://example.com:8080 # 需要监控的应用接口(get请求)
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 10.10.0.170:9115 # The blackbox exporter's real hostname:port.
- job_name: redis
static_configs:
- targets: ['10.10.0.170:9121']
- job_name: mysqld
static_configs:
- targets: ['10.10.0.170:9104']
- job_name: 'node'
scrape_interval: 5s
static_configs:
- targets: ['10.10.0.170:9100']
- targets: ['10.10.0.171:9100']
- targets: ['10.10.0.172:9100']
- targets: ['10.10.0.173:9100']
- job_name: 'docker'
scrape_interval: 5s
static_configs:
- targets: ['10.10.0.170:9100']
- targets: ['10.10.0.171:9100']
2、热加载使配置生效
curl -X POST http://localhost:9090/-/reload
四、配置grafana监控模版
- 模版地址:https://grafana.com/grafana/dashboards/
- 内网监控模版需要导出json文件,然后手动导入进去。
- 配置监控大盘,展示常需关注的监控项
1、prometheus自监控
- 模版id:3662
2、mysql监控
- 模版id:7362
3、redis监控
- 模版ID:9338
4、容器监控
- 模版id:893
5、服务器资源监控
- 模版id:11074
6、应用接口监控(blockbox)
- 模版id:7587
五、配置告警
1、配置告警规则
- 磁盘
- cpu
- 负载
- mysql
- redis
- docker容器
- 线上应用接口探测时长等监控
2、配置告警通道
- 钉钉(需能访问外网)
- 短信(需先对grafana告警信息进行处理,之后对接到短信接口)
- 邮件(需能访问邮件服务器)
3、设置重复告警频率
- 5m、10m等
六、权限控制
- admin(管理员)
- view(只读)
- edit(可编辑)