prometheus监控数据远程存储到victoriametrics。
prometheus部署在集群里,victoriametrics和promxy使用docker部署在集群外。
prometheus本身没有高可用部署方案,有fedration,但是fedration也存在一些不适用的场景,fedration利弊
promxy可以实现prometheus ha。
部署VictoriaMetrics
- 镜像拉取
docker pull victoriametrics/vminsert:v1.34.9-cluster
docker pull victoriametrics/vmselect:v1.34.9-cluster
docker pull victoriametrics/vmstorage:v1.34.9-cluster
- 部署
1. vmstorage
docker run -d --network host --name vmstorage --restart always -v /var/lib/cloudtogo/data/vm:/vmstorage-data:rw victoriametrics/vmstorage:v1.34.9-cluster --retentionPeriod=12 --httpListenAddr=0.0.0.0:8482 --vminsertAddr=0.0.0.0:8400 --vmselectAddr=0.0.0.0:8401
2. vminsert
docker run -d --network host --name vminsert --restart always victoriametrics/vminsert:v1.34.9-cluster --httpListenAddr=0.0.0.0:8480 --storageNode=0.0.0.0:8400
3. vmselect
docker run -d --network host --name vmselect --restart always victoriametrics/vmselect:v1.34.9-cluster --httpListenAddr=0.0.0.0:8481 --storageNode=0.0.0.0:8401
部署Promxy
https://github.com/jacksontj/promxy
promxy从VictoriaMetrics查询数据。promxy与VictoriaMetrics部署在同一台机器,如不不是同一台机器,修改targets的ip。
1. 准备promxy.yaml
````
global:
evaluation_interval: 5s
external_labels:
source: promxy
promxy:
server_groups:
- static_configs:
- targets:
- 0.0.0.0:8481
remote_read: false
anti_affinity: 10s
path_prefix: /select/1/prometheus
query_params:
nocache: 1
ignore_error: false
http_client:
tls_config:
insecure_skip_verify: true
````
2. 运行promxy
docker pull quay.io/jacksontj/promxy:latest
docker run -d --name promxy --network host -v /etc/promxy/promxy.yaml:/etc/promxy/config.yaml:ro quay.io/jacksontj/promxy:latest --log-level=trace --config=/etc/promxy/config.yaml
部署Prometheus-Operator
kube-state-metrics pod状态一直处于CrashLoopBackOff
解决
默认的kube-state-metrics-deployment.yaml里面使用的kube-state-metrics 镜像为:quay.io/coreos/kube-state-metrics:v1.8.0
把镜像tag修改为v1.8.0以上的就可以了
我修改为了1.9.5
quay.io/coreos/kube-state-metrics:v1.9.5
相关issue:
https://github.com/kubernetes/kube-state-metrics/issues/362