监控 - Thanos - Prometheus的低时延与低开销数据查询

监控 - Thanos - Prometheus的低时延与低开销数据查询

---

---

前言

假设我们必须为多个团队提供集中的指标平台。对于每个团队，我们将有一个专门的普罗米修斯。这些可能在不同环境（数据中心，区域，群集等）的同一环境中。
然后，我们将尝试提供低成本，快速的全球概览。让我们看看如何通过Thanos实现这一目标。

---

一、Thanos是什么？

请移步官网。

二、步骤

1.初始化Prometheus与Thanos

创建持久化存储

[root@host01 ~]# mkdir -p prometheus_data

部署Prometheus，配置文件

[root@host01 ~]# for i in `ls prometheus*`; do echo "#cat $i"; cat $i; done
#cat prometheus0.yml
global:
  scrape_interval: 5s
  external_labels:
    cluster: eu0
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']
#cat prometheus1.yml
global:
  scrape_interval: 5s
  external_labels:
    cluster: eu1
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9091']
#cat prometheus2.yml
global:
  scrape_interval: 5s
  external_labels:
    cluster: eu2
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9092']

运行

[root@host01 ~]# for i in $(seq 0 2); do
> docker run -d --net=host --rm \
>     -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \
>     -v $(pwd)/prometheus_data:/prometheus"${i}" \
>     -u root \
>     --name prometheus"${i}" \
>     quay.io/prometheus/prometheus:v2.22.2 \
>     --config.file=/etc/prometheus/prometheus.yml \
>     --storage.tsdb.path=/prometheus \
>     --web.listen-address=:909"${i}" \
>     --web.external-url=https://2886795273-909"${i}"-jago02.environments.katacoda.com \
>     --web.enable-lifecycle \
>     --web.enable-admin-api && echo "Prometheus ${i} started!"
> done
05a760659183552f7f068578eeb36579e1872d4aabf4f77b4c46a2f9f1525b00
Prometheus 0 started!
275561e2ee97806eba4dc194cc1849881d9b4edb4633b58f2a5fc866e337f19a
Prometheus 1 started!
d7af58f008532a758df77e65491cc6c1101a6da307886f0657acb5b84412df49
Prometheus 2 started!
[root@host01 ~]# docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
d7af58f00853        quay.io/prometheus/prometheus:v2.22.2   "/bin/prometheus --c…"   2 seconds ago       Up 1 second                             prometheus2
275561e2ee97        quay.io/prometheus/prometheus:v2.22.2   "/bin/prometheus --c…"   3 seconds ago       Up 3 seconds                            prometheus1
05a760659183        quay.io/prometheus/prometheus:v2.22.2   "/bin/prometheus --c…"   5 seconds ago       Up 3 seconds                            prometheus0

注入Thanos Sidecars

[root@host01 ~]# for i in $(seq 0 2); do
> docker run -d --net=host --rm \
>     -v $(pwd)/prometheus"${i}".yml:/etc/prometheus/prometheus.yml \
>     --name prometheus-sidecar"${i}" \
>     -u root \
>     quay.io/thanos/thanos:v0.20.0 \
>     sidecar \
>     --http-address=0.0.0.0:1909"${i}" \
>     --grpc-address=0.0.0.0:1919"${i}" \
>     --reloader.config-file=/etc/prometheus/prometheus.yml \
>     --prometheus.url=http://127.0.0.1:909"${i}" && echo "Started Thanos Sidecar for Prometheus ${i}!"
> done
a2a6c5220d4fc416ef892263c9115a28839fa8ac64c280d0478233c0fd61a857
Started Thanos Sidecar for Prometheus 0!
3dc8a335ed147054340faacdf3c44614341d992b8678c1332cae4067b74b26d6
Started Thanos Sidecar for Prometheus 1!
1c98f7b1a16864030d7e1c48d4c150e4fada5f50d7b3bb00dbd06d7a45be2d60
Started Thanos Sidecar for Prometheus 2!
[root@host01 ~]# docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
1c98f7b1a168        quay.io/thanos/thanos:v0.20.0           "/bin/thanos sidecar…"   3 seconds ago       Up 2 seconds                            prometheus-sidecar2
3dc8a335ed14        quay.io/thanos/thanos:v0.20.0           "/bin/thanos sidecar…"   4 seconds ago       Up 3 seconds                            prometheus-sidecar1
a2a6c5220d4f        quay.io/thanos/thanos:v0.20.0           "/bin/thanos sidecar…"   5 seconds ago       Up 4 seconds                            prometheus-sidecar0
d7af58f00853        quay.io/prometheus/prometheus:v2.22.2   "/bin/prometheus --c…"   11 seconds ago      Up 10 seconds                           prometheus2
275561e2ee97        quay.io/prometheus/prometheus:v2.22.2   "/bin/prometheus --c…"   12 seconds ago      Up 11 seconds                           prometheus1
05a760659183        quay.io/prometheus/prometheus:v2.22.2   "/bin/prometheus --c…"   14 seconds ago      Up 12 seconds                           prometheus0

部署Querier

[root@host01 ~]# docker run -d --net=host --rm \
>     --name querier \
>     quay.io/thanos/thanos:v0.20.0 \
>     query \
>     --http-address 0.0.0.0:10912 \
>     --grpc-address 0.0.0.0:10901 \
>     --query.replica-label replica \
>     --store 127.0.0.1:19190 \
>     --store 127.0.0.1:19191 \
>     --store 127.0.0.1:19192 && echo "Started Thanos Querier!"
0b936fa91739b473de179353b99466cd2ace4b4470180462c2e1f94721624d99
Started Thanos Querier!

可以访问Prometheus与Querier的web页面了。

2.使用Nginx模拟延时

配置文件

server {
 listen 10902;
 server_name proxy;
 location / {
  echo_exec @default;
 }
 location ^~ /api/v1/query_range {
     echo_sleep 1;
     echo_exec @default;
 }
 location @default {
     proxy_pass http://127.0.0.1:10912;
 }
}

运行Nginx

[root@host01 ~]# docker run -d --net=host --rm \
>     -v $(pwd)/nginx.conf:/etc/nginx/conf.d/default.conf \
>     --name nginx \
>     yannrobert/docker-nginx && echo "Started Querier Proxy!"
201e991dbd08dc539abac84fb849579e028252ffe668f15a65a1fb4d77c1ef9c
Started Querier Proxy!

部署Thanos Query Frontend，配置文件

type: IN-MEMORY
config:
  max_size: "0"
  max_size_items: 2048
  validity: "6h"

运行Thanos Query Frontend

[root@host01 ~]# docker run -d --net=host --rm \
>     -v $(pwd)/frontend.yml:/etc/thanos/frontend.yml \
>     --name query-frontend \
>     quay.io/thanos/thanos:v0.20.0 \
>     query-frontend \
>     --http-address 0.0.0.0:20902 \
>     --query-frontend.compress-responses \
>     --query-frontend.downstream-url=http://127.0.0.1:10902 \
>     --query-frontend.log-queries-longer-than=5s \
>     --query-range.split-interval=1m \
>     --query-range.response-cache-max-freshness=1m \
>     --query-range.max-retries-per-request=5 \
>     --query-range.response-cache-config-file=/etc/thanos/frontend.yml \
>     --cache-compression-type="snappy" && echo "Started Thanos Query Frontend!"
f6e30add82bcc2a6d8b0ec0dc2845afecdf3257c090940fbba14d604ad9abf1d
Started Thanos Query Frontend!

---

总结

使用Thanos Query Fronted可以有效地将结果缓存在所在服务器的内存中，当后续相同请求时，数据可快速地从缓存中取出返回。