目录
一.总览
prometheus提供的一张图
这张图包含几部分
1.监控grafana
2.pull metrics:(1)pushgateway (2)exporters
3.服务发现service discovery,主要是动态发现exporters
4.告警alertmanager
5.prometheus server
二.安装prometheus + grafana
1.在/ops/prometheus下建立文件prometheus.yml 和targets.json(这个可以不用,也是用于动态添加targets)
编辑文件vim prometheus.yml
global:
scrape_interval: 1s
evaluation_interval: 1s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
labels:
instance: prometheus
- job_name: pushgateway #安装的pushgateway使用该配置项
metrics_path: '/metrics'
static_configs:
- targets: ['192.168.253.215:9091'] # 地址修改为实际地址
labels:
instance: pushgateway
honor_labels: true
- job_name: nodes
file_sd_configs:#提供文件方式动态添加targets
- files:
- 'targets.json'
refresh_interval: 10s
编辑文件 targets.json ,node_exporter会用该配置项
[
{
"targets": [
"192.168.253.215:9400"
],
"labels": {
"job": "node"
}
}
]
2.启动prometheus
docker run -d -p 9090:9090 -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/targets.json:/etc/prometheus/targets.json prom/prometheus
启动后访问地址 http://192.168.253.215:9090/
3.启动grafana
docker run -d \
-p 3000:3000 \
--name=grafana \
-v /opt/grafana-storage:/var/lib/grafana \
grafana/grafana
访问地址
3.在grafana添加prometheus,点击save保存就可以
4.grafana导入已经存在的图表配置
在这里下载一个模板locust :https://grafana.com/grafana/dashboards/12081
导入
5.grafana刷新时间是最小5秒,配置每秒刷新一次,注:网上说低版本有bug,该配置项不起作用
进入docker
docker exec -it --user root b1dbe0b4357a /bin/bash
修改配置,修改后重启docker
vi /etc/grafana/grafana.ini
搜min_refresh_interval改为1s
由:
;min_refresh_interval = 5s
改为:
min_refresh_interval = 1s
6.打开grafana相关的工程,配置刷新时间为1s
非docker安装
grafana下载包:https://grafana.com/grafana/download?platform=linux
./grafana-server -config=/home/ops/grafana-7.3.5/conf/grafana.ini cfg:default.paths.plugins=/home/ops/grafana-7.3.5/plugins
三. 安装一个exporter
1.下载地址 https://github.com/prometheus/node_exporter/releases
2.启动 ./node_exporter --web.listen-address=":9400"
3. 前面prometheus已经配置了9400这个ip地址的targets,所有现在应该已经开始监控
4. 随便打开一个面板图,在面板最下面添加一个【Query】,选择【Metrics】=》【node】=》【node_load1】,输入框可以根据条件过滤 node_load1{instance="192.168.253.215:9400", job="node"}
5.修改面板的刷新时间是1S
四.pushgateway安装
拓扑图
1. docker 安装 docker run -d -p 9091:9091 prom/pushgateway
2.启动后可以访问 http://192.168.253.215:9091/
3.推一条指标给pushgateway
cat <<EOF | curl --data-binary @- http://192.168.253.215:9091/metrics/job/some_job/instance/some_instance
locust_stats_avg_content_length 2398.283
EOF
4.在上面prometheus已经配置好pushgateway,如果没配,先配置Prometheus
5.删除指标,也可以去界面删除 curl -X DELETE http://192.168.253.215:9091/metrics/job/some_job/instance/some_instance
6.python用 prometheus_client实现参考链接:
https://www.cnblogs.com/xiao987334176/p/9933963.html
五. service discovery之consul
1.安装consul
docker run --name consul -d -p 8500:8500 consul
2.配置prometheus
- job_name: 'consul'
consul_sd_configs:
- server: '192.168.253.215:8500'
services: []
relabel_configs:
- source_labels: [__meta_consul_tags]
regex: .*dev.*
action: keep
3.向consul注册服务
curl -X PUT -d '{"id": "test1","name": "test1","address": "192.168.253.215","port": 9500,"tags": ["dev"],"checks": [{"http": "http://192.168.253.215:9500/","interval": "5s"}]}' http://192.168.253.215:8500/v1/agent/service/register
注销服务
curl -X PUT http://192.168.253.215:8500/v1/agent/service/deregister/服务id
4.启动服务node_exporter :
./node_exporter --web.listen-address=":9500"
5.查看注册成功 :http://192.168.253.215:8500/ui/dc1/services/test1/instances
6.查看peometheus是否正常 http://192.168.253.215:9090/targets
7.查看grafana,多了一条曲线
六.alertmanager
七.Grafana嵌入到自己的项目
八.Grafana API报告的导出为PDF
由于Grafana插件不支持docker,需要自己安装grafana程序
wget https://dl.grafana.com/oss/release/grafana-7.3.5-1.x86_64.rpm
sudo yum install grafana-7.3.5-1.x86_64.rpm
service grafana-server start
yum install -y libXcomposite libXdamage libXtst cups libXScrnSaver pango atk adwaita-cursor-theme adwaita-icon-theme at at-spi2-atk at-spi2-core cairo-gobject colord-libs dconf desktop-file-utils ed emacs-filesystem gdk-pixbuf2 glib-networking gnutls gsettings-desktop-schemas gtk-update-icon-cache gtk3 hicolor-icon-theme jasper-libs json-glib libappindicator-gtk3 libdbusmenu libdbusmenu-gtk3 libepoxy liberation-fonts liberation-narrow-fonts liberation-sans-fonts liberation-serif-fonts libgusb libindicator-gtk3 libmodman libproxy libsoup libwayland-cursor libwayland-egl libxkbcommon m4 mailx nettle patch psmisc redhat-lsb-core redhat-lsb-submod-security rest spax time trousers xdg-utils xkeyboard-config
安装插件和grafana-reporter
grafana-cli plugins install grafana-image-renderer
安装一下库,不然grafana-reporter会报错
yum install texlive
在第七步的基础上配置Grafana
手动下载pdf,grafana的看板是中文时会有问题,但指标是中文,没问题
九.prometheus API的使用
获取到所有job
http://192.168.253.215:9090/api/v1/label/job/values
查询10秒内数据
http://192.168.253.215:9090/api/v1/query_range?query=network_traffic_input&start=1607999428.447&end=1607999468.447&step=10s
查询指标
http://192.168.253.215:9090/api/v1/series?match[]=up{job=\"network_traffic\"}&match[]=network_traffic_input{adapter_name=\"ens33\"}
查询当前
http://192.168.253.215:9090/api/v1/query?query=network_traffic_input
查询
http://192.168.253.215:9090/api/v1/query_range?query=locust_stats_avg_content_length{instance="some_instance1"} offset 15m&start=1608009390&end=1608012990&step=15
编码后:
http://192.168.253.215:9090/api/v1/query_range?query=locust_stats_avg_content_length%7Binstance%3D%22some_instance1%22%7D%20offset%2015m&start=1608009390&end=1608012990&step=15
consul注册服务
curl -X PUT -d '{"id": "test1","name": "test1","address": "192.24.17.156","port": 9500,"tags": ["dev"],"checks": [{"http": "http://192.24.17.156:9500/","interval": "5s"}]}' http://192.24.17.156:8500/v1/agent/service/register
grafana下载包:https://grafana.com/grafana/download?platform=linux
./grafana-server -config=/home/ops/grafana-7.3.5/conf/grafana.ini cfg:default.paths.plugins=/home/ops/grafana-7.3.5/plugins
参考文章:
1.locust + Prometheus + grafana https://bugvanisher.cn/2020/04/05/locust-with-prometheus-and-grafana/
2.pushgateway安装使用: https://www.cnblogs.com/xiaobaozi-95/p/10684524.html
3.prometheus file_sd_config配置项 https://www.cnblogs.com/faberbeta/p/13597816.html
4.prometheus consul:https://blog.51cto.com/1000682/2363038
5.alertmanager:https://www.cnblogs.com/xiaobaozi-95/p/10740511.html
6.动态更新prometheus 报警规则 https://www.lijiaocn.com/%E6%8A%80%E5%B7%A7/2018/08/30/confd-prometheus-dynamic-config.html
7.Grafana嵌入到自己的项目:https://blog.csdn.net/weixin_44534391/article/details/107019567?utm_medium=distribute.pc_relevant.none-task-blog-baidulandingword-3&spm=1001.2101.3001.4242
8.Grafana导出pdf :https://blog.csdn.net/shen12138/article/details/110199638
9.pushgateway python 使用https://www.cnblogs.com/xiao987334176/p/9933963.html