prometheus存储

promethues本地存储简介

–storage.tsdb.retention.time=STORAGE.TSDB.RETENTION.TIME=1y 默认保存数据天数 默认15天
–storage.tsdb.retention.size=512MB 改 block大小 默认512M #注意单位KB, MB, GB, TB, PB, EB. Ex
./prometheus --help |grep -C5 storage.tsdb.retention.size
sevice修改即可
ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml --web.enable-lifecycle --storage.tsdb.retention.time=1y

prometheus本地存储配置参数

–storage.tsdb.path #指定数据保存位置
–storage.tsdb.retention.time #指定数据保存时间，默认15d
–storage.tsdb.retention.size #指定block可以保存的数据大小
–query.timeout #最大查询超时时间，默认2m
–query.max-concurrency #最大查询并发数，默认20

prometheus远端存储之VictoriaMetrics

简介
VictoriaMetrics是一个快速、支持高可用且可扩展的开源时序数据库和监控解决方案。可用做Prometheus的远端存储。

VictoriaMetrics官网：https://victoriametrics.com/
官方文档：https://docs.victoriametrics.com/
项目地址：https://github.com/VictoriaMetrics

VictoriaMetrics的优点：

兼容Prometheus相关API，可以直接用作Grafana的数据源
内存占用率低
查询速度快
设置和操作简单
支持水平扩容和HA(集群版)
高压缩比等等
VictoriaMetrics分为集群版和单机版，根据实际需求选择即可。

单机版：直接通过二进制文件运行即可，官方建议采集数据点(data points)低于100w/s，推荐VM单节点版，简单好维护，但不支持告警。
集群版：支持数据水平拆分，根据功能拆分为不同的组件vmselect、vminsert、vmstorage，如果替换Prometheus还可以加上vmagent和vmalert

相关组件作用如下：

vmstorage：数据存储即查询结果返回，默认端口8482
vminsert：写入组件，vminsert负责接收数据写入请求并根据对数据的hash结果将数据写入到后端不同的vmstorage之上，实现数据分片。默认端口8480，属于无状态服务可水平扩展
vmselect：查询组件，负责接收查询请求，并连接到vmstorage查询数据。默认端口8481，属于无状态服务可水平扩展
另外还有一些可选组件：

vmagent：类似于Prometheus，负责从各种数据来源收集指标数据
vmalert：类似于Alertmanager，负责实现告警功能
vmctl：命令行工具

单机部署

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.87.6/victoria-metrics-linux-amd64-v1.87.6.tar.gz
查看帮助
victoria-metrics-prod -h
配置service文件
cat /lib/systemd/system/victoria-metrics.service
[Unit]
Description=For Victoria-metrics-prod single service
After=network.target

[Service]
ExecStart=/usr/local/bin/victoria-metrics-prod -httpListenAddr=0.0.0.0:8428 -storageDataPath=/data/victoria-metrics a-retentionPeriod=3

[Install]
WantedBy=multi-user.target

查看数据目录
root@k8s-etcd03:~# ll /data/victoria-metrics/
total 0
drwxr-xr-x 8 root root 108 May 21 15:24 ./
drwxr-xr-x 5 root root 67 May 21 15:40 …/
drwxr-xr-x 6 root root 211 May 21 15:24 cache/
drwxr-xr-x 4 root root 48 May 21 14:44 data/
-rw-r–r-- 1 root root 0 May 21 14:44 flock.lock
drwxr-xr-x 5 root root 71 May 21 14:44 indexdb/
drwxr-xr-x 2 root root 43 May 21 14:44 metadata/
drwxr-xr-x 2 root root 6 May 21 14:44 snapshots/
drwxr-xr-x 3 root root 27 May 21 14:44 tmp/
root@k8s-etcd03:~#

prometheus添加配置
global: …
remote_write:

• url: http://192.168.1.23:8428/api/v1/write
  systemctl restart prometheus.service

http://192.168.1.23:8428/vmui/
ranfana添加数据源

在这里插入图片描述](https://img-blog.csdnimg.cn/68d6db6307744b14a916765a799527f8.png)

集群部署

192.168.1.21 vm-node-01 vmselect/vminsert/vmstorage
192.168.1.22 vm-node-02 vmselect/vminsert/vmstorage
192.168.1.23 vm-node-03 vmselect/vminsert/vmstorage
192.168.1.25 prometheus
192.168.1.188 VIP

下载包jie
https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.87.6/victoria-metrics-linux-amd64-v1.87.6-cluster.tar.gz

root@k8s-etcd03:~# tar -xf victoria-metrics-linux-amd64-v1.87.6-cluster.tar.gz
root@k8s-etcd03:~# scp /usr/local/bin/vm* 192.168.1.22:/usr/local/bin/
root@k8s-etcd03:~# scp /usr/local/bin/vm* 192.168.1.21:/usr/local/bin/

部署vmstorage

root@vm-node-01:~# cat /lib/systemd/system/vmstorage.service
[Unit]
Description=Vmstorage Server
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/tmp
ExecStart=/usr/local/bin/vmstorage-prod -loggerTimezone Asia/Shanghai -storageDataPath /data/vmstorage-data -httpListenAddr :8482 -vminsertAddr :8400 -vmselectAddr :8401

[Install]
WantedBy=multi-user.target

vmstorage会监听3个端口，8482是自身API端口；8400提供给vminsert，负责处理数据写入；8401提供给vmselect，负责数据读取。

mkdir --p /data/vmstorage-data
拷贝到其他服务器

systemctl daemon-reload
systemctl start vmstorage.service
systemctl status vmstorage.service
systemctl enable vmstorage.service
oot@k8s-etcd03:~# scp /lib/systemd/system/vmstorage.service 192.168.1.21:/lib/systemd/system/
root@192.168.1.21's password: 
vmstorage.service                                                                                                         100%  317   223.6KB/s   00:00    
root@k8s-etcd03:~# scp /lib/systemd/system/vmstorage.service 192.168.1.22:/lib/systemd/system/

部署vmselect-pord

准备service文件
vmselect自身监听8481端口对外提供服务，-storageNode用于指定vmstorage地址

root@vm-node-01:~# cat /lib/systemd/system/vmselect.service
[Unit]
Description=Vmselect Server
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/tmp
ExecStart=/usr/bin/vmselect-prod -httpListenAddr :8481 -storageNode=192.168.1.21:8401,192.168.1.22:8401,192.168.1.23:8401

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start vmselect.service
systemctl status vmselect.service
systemctl enable vmselect.service
scp到其他服务器启动

部署vminsert-prod

准备service文件
vminsert自身监听8480端口对外提供服务，-storageNode用于指定vmstorage地址

root@vm-node-01:~# cat /lib/systemd/system/vminsert.service
[Unit]
Description=Vminsert Server
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/tmp
ExecStart=/usr/local/bin/vminsert-prod -httpListenAddr :8480 -storageNode=192.168.1.21:8401,192.168.1.22:8401,192.168.1.23:8401

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl start vminsert.service
systemctl status vminsert.service
systemctl enable vminsert.service
scp到其他服务器启动

haproxy1.21添加配置

listen vmselect-8481
bind 192.168.1.188:8888
option tcplog
mode tcp
balance source
server vmselect-node1 192.168.1.21:8481 check inter 2000 fall 3 rise 5
server vmselect-node2 192.168.1.22:8481 check inter 2000 fall 3 rise 5
server vmselect-node3 192.168.1.23:8481 check inter 2000 fall 3 rise 5

listen vmselect-8480
bind 192.168.1.188:9999
option tcplog
mode tcp
balance source
server vmselect-node1 192.168.1.21:8480 check inter 2000 fall 3 rise 5
server vmselect-node2 192.168.1.22:8480 check inter 2000 fall 3 rise 5
server vmselect-node3 192.168.1.23:8480 check inter 2000 fall 3 rise 5
systemctl restart haproxy

配置Grafana以vmselect为数据源

prometheus 1.25 添加配置

remote_write:

• url: http://192.168.1.188:9999/insert/0/prometheus

Grafana添加数据源
http://192.168.1.188:8888/select/0/prometheus

导入模板11074

Prometheus保存数据的时间
root@k8s-etcd01:/apps/prometheus# ./prometheus --help |grep -B 7 15
Use with server mode only.
–storage.tsdb.retention.time=STORAGE.TSDB.RETENTION.TIME
How long to retain samples in storage.
When this flag is set it overrides
“storage.tsdb.retention”. If neither this
flag nor “storage.tsdb.retention” nor
“storage.tsdb.retention.size” is set,
the retention time defaults to 15d. Units

在service文件添加 --storage.tsdb.retention.time=1y #数据保存一年

ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml --web.enable-lifecycle --storage.tsdb.retention.time=1y --storage.tsdb.retention.size=512MB