监控 - Minio - Prometheus的降准采样与无限存储容量保存

监控 - Minio - Prometheus的降准采样与无限存储容量保存

---

---

前言

Prometheus越来越普及，那么它产生的数据，我们如何来管理呢？
Thanos是一组组件，可以组成具有无限存储容量的高可用性指标系统，并且可以无缝地添加到现有Prometheus部署之上。

您将了解：

• 借助Thanos sidecar，如何开始将Prometheus数据无缝地无缝上传到廉价的对象存储中。
• 借助Thanos Store Gateway，如何进一步查询对象存储中的数据。
• 借助Thanos Querier以简单的方式查询新旧数据。

所有这些使您可以将指标保持在便宜且可靠的对象存储中，从而几乎无限地保留Prometheus的指标。
注意：本帖子使用Docker容器，并公开提供预构建的Thanos，Prometheus和Minio Docker映像。但是，类似的情况也可以与任何其他部署方法（例如Kubernetes或systemd等）一起使用。

一、Prometheus是什么？

请移步官网。

二、步骤

1.准备假数据

Your Interactive Bash Terminal.

[root@host01 ~]# mkdir -p /root/prom-eu1 && docker run -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block plan -p continuous-365d-tiny --labels 'cluster="eu1"' --max-time=6h | docker run -v /root/prom-eu1:/prom-eu1 -i quay.io/thanos/thanosbench:v0.2.0-rc.1 block gen --output.dir prom-eu1
level=info ts=2021-05-11T04:56:54.408835119Z caller=block.go:94 msg="generating block" spec="[1620727200001 - 1620734400000](1h59m59.999s) "
level=info ts=2021-05-11T04:56:54.417790284Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-05-11T04:56:54.417864575Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=10.847µs
level=info ts=2021-05-11T04:56:54.417886581Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:54.423967454Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-05-11T10:00:15.001Z maxt=2021-05-11T12:00:00.001Z
level=info ts=2021-05-11T04:56:54.460080448Z caller=compact.go:494 msg="write block" mint=1620727215001 maxt=1620734400002 ulid=01F5CWHYWRKQ7WCGWBSKV6518H duration=36.004591ms
level=info ts=2021-05-11T04:56:54.462805443Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHYWRKQ7WCGWBSKV6518H count=1
level=info ts=2021-05-11T04:56:54.469156289Z caller=block.go:94 msg="generating block" spec="[1620720000002 - 1620727200001](1h59m59.999s) "
level=info ts=2021-05-11T04:56:54.471970297Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-05-11T04:56:54.472017051Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=25.314µs
level=info ts=2021-05-11T04:56:54.472030163Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:54.481801138Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-05-11T08:00:15.002Z maxt=2021-05-11T10:00:00.002Z
level=info ts=2021-05-11T04:56:54.533046984Z caller=compact.go:494 msg="write block" mint=1620720015002 maxt=1620727200003 ulid=01F5CWHYYHXH6H9CQX5HQG0YQN duration=51.155789ms
level=info ts=2021-05-11T04:56:54.537662664Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHYYHXH6H9CQX5HQG0YQN count=2
level=info ts=2021-05-11T04:56:54.541044848Z caller=block.go:94 msg="generating block" spec="[1620712800003 - 1620720000002](1h59m59.999s) "
level=info ts=2021-05-11T04:56:54.546108418Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:54.548139894Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.548175311Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:54.548191437Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.548202992Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=95.055µs
level=info ts=2021-05-11T04:56:54.548212945Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:54.554382259Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-05-11T06:00:15.003Z maxt=2021-05-11T08:00:00.003Z
level=info ts=2021-05-11T04:56:54.589166905Z caller=compact.go:494 msg="write block" mint=1620712815003 maxt=1620720000004 ulid=01F5CWHZ0TG0S3CHG1F5HR98JF duration=34.708923ms
level=info ts=2021-05-11T04:56:54.592713215Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHZ0TG0S3CHG1F5HR98JF count=3
level=info ts=2021-05-11T04:56:54.596268346Z caller=block.go:94 msg="generating block" spec="[1620684000004 - 1620712800003](7h59m59.999s) "
level=info ts=2021-05-11T04:56:54.601569527Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:54.603303338Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.603338104Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:54.603353552Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.603364879Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=96.413µs
level=info ts=2021-05-11T04:56:54.603374537Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:54.619949757Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-05-10T22:00:15.004Z maxt=2021-05-11T06:00:00.004Z
level=info ts=2021-05-11T04:56:54.663111959Z caller=compact.go:494 msg="write block" mint=1620684015004 maxt=1620712800005 ulid=01F5CWHZ2W5206X7E0HVPH7D98 duration=42.888986ms
level=info ts=2021-05-11T04:56:54.667719088Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHZ2W5206X7E0HVPH7D98 count=4
level=info ts=2021-05-11T04:56:54.674314727Z caller=block.go:94 msg="generating block" spec="[1620050400005 - 1620684000004](175h59m59.999s) "
level=info ts=2021-05-11T04:56:54.681057108Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:54.68529999Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.685370765Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:54.685397817Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.685420255Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=187.74µs
level=info ts=2021-05-11T04:56:54.685439652Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:54.821792548Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-05-03T14:00:15.005Z maxt=2021-05-10T22:00:00.005Z
level=info ts=2021-05-11T04:56:54.860732383Z caller=compact.go:494 msg="write block" mint=1620050415005 maxt=1620684000006 ulid=01F5CWHZ966KG56NSAHCHNR027 duration=38.389698ms
level=info ts=2021-05-11T04:56:54.867858933Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHZ966KG56NSAHCHNR027 count=5
level=info ts=2021-05-11T04:56:54.874721727Z caller=block.go:94 msg="generating block" spec="[1619416800006 - 1620050400005](175h59m59.999s) "
level=info ts=2021-05-11T04:56:54.878859054Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:54.879158664Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.879424137Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:54.879687573Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:54.880010975Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=878.623µs
level=info ts=2021-05-11T04:56:54.88025637Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:54.955685635Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-04-26T06:00:15.006Z maxt=2021-05-03T14:00:00.006Z
level=info ts=2021-05-11T04:56:54.993337965Z caller=compact.go:494 msg="write block" mint=1619416815006 maxt=1620050400007 ulid=01F5CWHZDCTM74JPPY0D9T1FRX duration=37.096402ms
level=info ts=2021-05-11T04:56:54.999547869Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHZDCTM74JPPY0D9T1FRX count=6
level=info ts=2021-05-11T04:56:55.003511955Z caller=block.go:94 msg="generating block" spec="[1618783200007 - 1619416800006](175h59m59.999s) "
level=info ts=2021-05-11T04:56:55.007392563Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:55.007755817Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:55.008070946Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:55.008322945Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:55.008584292Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=855.387µs
level=info ts=2021-05-11T04:56:55.008824763Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:55.089809186Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-04-18T22:00:15.007Z maxt=2021-04-26T06:00:00.007Z
level=info ts=2021-05-11T04:56:55.123327117Z caller=compact.go:494 msg="write block" mint=1618783215007 maxt=1619416800008 ulid=01F5CWHZHJCS5QQSE4XFM5RVXB duration=32.935776ms
level=info ts=2021-05-11T04:56:55.128561572Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHZHJCS5QQSE4XFM5RVXB count=7
level=info ts=2021-05-11T04:56:55.133006791Z caller=block.go:94 msg="generating block" spec="[1618149600008 - 1618783200007](175h59m59.999s) "
level=info ts=2021-05-11T04:56:55.137247085Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:55.137729633Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:55.138015279Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:55.138289975Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:55.138571862Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=868.124µs
level=info ts=2021-05-11T04:56:55.138873513Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:55.220877076Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-04-11T14:00:15.008Z maxt=2021-04-18T22:00:00.008Z
level=info ts=2021-05-11T04:56:55.267909572Z caller=compact.go:494 msg="write block" mint=1618149615008 maxt=1618783200009 ulid=01F5CWHZNNFA138FQNRM3QW75R duration=46.467375ms
level=info ts=2021-05-11T04:56:55.27575418Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWHZNNFA138FQNRM3QW75R count=8
level=info ts=2021-05-11T04:56:55.280589698Z caller=block.go:94 msg="generating block" spec="[1612360800009 - 1618149600008](1607h59m59.999s) "
level=info ts=2021-05-11T04:56:55.284804965Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:55.285199952Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:55.285474795Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:55.285766019Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:55.286060032Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=887.117µs
level=info ts=2021-05-11T04:56:55.286359584Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:56.062159058Z caller=writer.go:123 msg=flushing series_count=5 mint=2021-02-03T14:00:15.009Z maxt=2021-04-11T14:00:00.009Z
level=info ts=2021-05-11T04:56:56.124002767Z caller=compact.go:494 msg="write block" mint=1612360815009 maxt=1618149600010 ulid=01F5CWJ0FY0M0CP4DFNNNCTPE0 duration=61.257038ms
level=info ts=2021-05-11T04:56:56.135411983Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWJ0FY0M0CP4DFNNNCTPE0 count=9
level=info ts=2021-05-11T04:56:56.177308651Z caller=block.go:94 msg="generating block" spec="[1606572000010 - 1612360800009](1607h59m59.999s) "
level=info ts=2021-05-11T04:56:56.181501167Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:56.182078574Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:56.182490237Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:56.182902757Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:56.18329007Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=1.256298ms
level=info ts=2021-05-11T04:56:56.183676873Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:56.928191137Z caller=writer.go:123 msg=flushing series_count=5 mint=2020-11-28T14:00:15.01Z maxt=2021-02-03T14:00:00.01Z
level=info ts=2021-05-11T04:56:57.01720953Z caller=compact.go:494 msg="write block" mint=1606572015010 maxt=1612360800011 ulid=01F5CWJ1B01SQEMX6XKYA7EZ1F duration=88.455284ms
level=info ts=2021-05-11T04:56:57.02798463Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWJ1B01SQEMX6XKYA7EZ1F count=10
level=info ts=2021-05-11T04:56:57.078618763Z caller=block.go:94 msg="generating block" spec="[1600783200011 - 1606572000010](1607h59m59.999s) "
level=info ts=2021-05-11T04:56:57.083591418Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:57.084145527Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:57.084465088Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:57.084744709Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:57.084999157Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=882.912µs
level=info ts=2021-05-11T04:56:57.085318117Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:57.812582079Z caller=writer.go:123 msg=flushing series_count=5 mint=2020-09-22T14:00:15.011Z maxt=2020-11-28T14:00:00.011Z
level=info ts=2021-05-11T04:56:57.873177073Z caller=compact.go:494 msg="write block" mint=1600783215011 maxt=1606572000012 ulid=01F5CWJ26N5W77E8B0Q5RB6F8K duration=60.008865ms
level=info ts=2021-05-11T04:56:57.881391326Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWJ26N5W77E8B0Q5RB6F8K count=11
level=info ts=2021-05-11T04:56:57.909724971Z caller=block.go:94 msg="generating block" spec="[1594994400012 - 1600783200011](1607h59m59.999s) "
level=info ts=2021-05-11T04:56:57.913259512Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:57.913749949Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:57.91408878Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:57.91439951Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:57.914670437Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=959.595µs
level=info ts=2021-05-11T04:56:57.914954718Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:58.573531464Z caller=writer.go:123 msg=flushing series_count=5 mint=2020-07-17T14:00:15.012Z maxt=2020-09-22T14:00:00.012Z
level=info ts=2021-05-11T04:56:58.627063831Z caller=compact.go:494 msg="write block" mint=1594994415012 maxt=1600783200013 ulid=01F5CWJ2YEQDCKB0TQEKT7CDY4 duration=52.816375ms
level=info ts=2021-05-11T04:56:58.636544906Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWJ2YEQDCKB0TQEKT7CDY4 count=12
level=info ts=2021-05-11T04:56:58.676208564Z caller=block.go:94 msg="generating block" spec="[1589205600013 - 1594994400012](1607h59m59.999s) "
level=info ts=2021-05-11T04:56:58.679676222Z caller=head.go:644 msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2021-05-11T04:56:58.680020535Z caller=head.go:649 msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:58.68033683Z caller=head.go:760 msg="Deleting mmapped chunk files"
level=info ts=2021-05-11T04:56:58.680627844Z caller=head.go:763 msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 1"
level=info ts=2021-05-11T04:56:58.680912745Z caller=head.go:658 msg="On-disk memory mappable chunks replay completed" duration=919.934µs
level=info ts=2021-05-11T04:56:58.681183765Z caller=head.go:660 msg="WAL not found"
level=info ts=2021-05-11T04:56:59.370129732Z caller=writer.go:123 msg=flushing series_count=5 mint=2020-05-11T14:00:15.013Z maxt=2020-07-17T14:00:00.013Z
level=info ts=2021-05-11T04:56:59.437010504Z caller=compact.go:494 msg="write block" mint=1589205615013 maxt=1594994400014 ulid=01F5CWJ3QAQTR43BB1VC1E1FNG duration=66.257359ms
level=info ts=2021-05-11T04:56:59.449880715Z caller=block.go:100 msg="generated block" path=prom-eu1/01F5CWJ3QAQTR43BB1VC1E1FNG count=13
level=info ts=2021-05-11T04:56:59.494743344Z caller=block.go:87 msg="all blocks done" count=13
level=info ts=2021-05-11T04:56:59.495299719Z caller=main.go:118 msg=exiting cmd="block gen"
[root@host01 ~]# 

查看目录，假数据已生成

[root@host01 ~]# ls -lR /root/prom-eu1
/root/prom-eu1:
total 52
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHYWRKQ7WCGWBSKV6518H
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHYYHXH6H9CQX5HQG0YQN
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHZ0TG0S3CHG1F5HR98JF
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHZ2W5206X7E0HVPH7D98
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHZ966KG56NSAHCHNR027
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHZDCTM74JPPY0D9T1FRX
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHZHJCS5QQSE4XFM5RVXB
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWHZNNFA138FQNRM3QW75R
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWJ0FY0M0CP4DFNNNCTPE0
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWJ1B01SQEMX6XKYA7EZ1F
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWJ26N5W77E8B0Q5RB6F8K
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWJ2YEQDCKB0TQEKT7CDY4
drwxr-xr-x 3 root root 4096 May 11 04:56 01F5CWJ3QAQTR43BB1VC1E1FNG

/root/prom-eu1/01F5CWHYWRKQ7WCGWBSKV6518H:
total 16
drwxr-xr-x 2 root root 4096 May 11 04:56 chunks
-rw-r--r-- 1 root root  932 May 11 04:56 index
-rw-r--r-- 1 root root  392 May 11 04:56 meta.json
-rw-r--r-- 1 root root    9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHYWRKQ7WCGWBSKV6518H/chunks:
total 8
-rw-r--r-- 1 root root 1241 May 11 04:56 000001

/root/prom-eu1/01F5CWHYYHXH6H9CQX5HQG0YQN:
total 16
drwxr-xr-x 2 root root 4096 May 11 04:56 chunks
-rw-r--r-- 1 root root  932 May 11 04:56 index
-rw-r--r-- 1 root root  392 May 11 04:56 meta.json
-rw-r--r-- 1 root root    9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHYYHXH6H9CQX5HQG0YQN/chunks:
total 8
-rw-r--r-- 1 root root 1241 May 11 04:56 000001

/root/prom-eu1/01F5CWHZ0TG0S3CHG1F5HR98JF:
total 16
drwxr-xr-x 2 root root 4096 May 11 04:56 chunks
-rw-r--r-- 1 root root  932 May 11 04:56 index
-rw-r--r-- 1 root root  392 May 11 04:56 meta.json
-rw-r--r-- 1 root root    9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHZ0TG0S3CHG1F5HR98JF/chunks:
total 8
-rw-r--r-- 1 root root 1241 May 11 04:56 000001

/root/prom-eu1/01F5CWHZ2W5206X7E0HVPH7D98:
total 16
drwxr-xr-x 2 root root 4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 1324 May 11 04:56 index
-rw-r--r-- 1 root root  392 May 11 04:56 meta.json
-rw-r--r-- 1 root root    9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHZ2W5206X7E0HVPH7D98/chunks:
total 12
-rw-r--r-- 1 root root 4703 May 11 04:56 000001

/root/prom-eu1/01F5CWHZ966KG56NSAHCHNR027:
total 24
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 11652 May 11 04:56 index
-rw-r--r-- 1 root root   396 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHZ966KG56NSAHCHNR027/chunks:
total 108
-rw-r--r-- 1 root root 102556 May 11 04:56 000001

/root/prom-eu1/01F5CWHZDCTM74JPPY0D9T1FRX:
total 24
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 11652 May 11 04:56 index
-rw-r--r-- 1 root root   396 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHZDCTM74JPPY0D9T1FRX/chunks:
total 108
-rw-r--r-- 1 root root 102635 May 11 04:56 000001

/root/prom-eu1/01F5CWHZHJCS5QQSE4XFM5RVXB:
total 24
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 11652 May 11 04:56 index
-rw-r--r-- 1 root root   396 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHZHJCS5QQSE4XFM5RVXB/chunks:
total 108
-rw-r--r-- 1 root root 102556 May 11 04:56 000001

/root/prom-eu1/01F5CWHZNNFA138FQNRM3QW75R:
total 24
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 11652 May 11 04:56 index
-rw-r--r-- 1 root root   396 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWHZNNFA138FQNRM3QW75R/chunks:
total 108
-rw-r--r-- 1 root root 102556 May 11 04:56 000001

/root/prom-eu1/01F5CWJ0FY0M0CP4DFNNNCTPE0:
total 112
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 99813 May 11 04:56 index
-rw-r--r-- 1 root root   398 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWJ0FY0M0CP4DFNNNCTPE0/chunks:
total 920
-rw-r--r-- 1 root root 936041 May 11 04:56 000001

/root/prom-eu1/01F5CWJ1B01SQEMX6XKYA7EZ1F:
total 112
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 99813 May 11 04:56 index
-rw-r--r-- 1 root root   398 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWJ1B01SQEMX6XKYA7EZ1F/chunks:
total 920
-rw-r--r-- 1 root root 936041 May 11 04:56 000001

/root/prom-eu1/01F5CWJ26N5W77E8B0Q5RB6F8K:
total 112
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 99813 May 11 04:56 index
-rw-r--r-- 1 root root   398 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWJ26N5W77E8B0Q5RB6F8K/chunks:
total 920
-rw-r--r-- 1 root root 936041 May 11 04:56 000001

/root/prom-eu1/01F5CWJ2YEQDCKB0TQEKT7CDY4:
total 112
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 99813 May 11 04:56 index
-rw-r--r-- 1 root root   398 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWJ2YEQDCKB0TQEKT7CDY4/chunks:
total 920
-rw-r--r-- 1 root root 936041 May 11 04:56 000001

/root/prom-eu1/01F5CWJ3QAQTR43BB1VC1E1FNG:
total 112
drwxr-xr-x 2 root root  4096 May 11 04:56 chunks
-rw-r--r-- 1 root root 99813 May 11 04:56 index
-rw-r--r-- 1 root root   398 May 11 04:56 meta.json
-rw-r--r-- 1 root root     9 May 11 04:56 tombstones

/root/prom-eu1/01F5CWJ3QAQTR43BB1VC1E1FNG/chunks:
total 920
-rw-r--r-- 1 root root 936041 May 11 04:56 000001
[root@host01 ~]# 

2.启动Prometheus

配置文件/root/editor/prometheus0_eu1.yml

global:
  scrape_interval: 5s
  external_labels:
    cluster: eu1
    replica: 0
    tenant: team-eu # Not needed, but a good practice if you want to grow this to multi-tenant system some day.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']
  - job_name: 'sidecar'
    static_configs:
      - targets: ['127.0.0.1:19090']
  - job_name: 'minio'
    metrics_path: /minio/prometheus/metrics
    static_configs:
      - targets: ['127.0.0.1:9000']
  - job_name: 'querier'
    static_configs:
      - targets: ['127.0.0.1:9091']
  - job_name: 'store_gateway'
    static_configs:
      - targets: ['127.0.0.1:19092']

启动容器

[root@host01 ~]# docker run -d --net=host --rm \
>     -v /root/editor/prometheus0_eu1.yml:/etc/prometheus/prometheus.yml \
>     -v /root/prom-eu1:/prometheus \
>     -u root \
>     --name prometheus-0-eu1 \
>     quay.io/prometheus/prometheus:v2.20.0 \
>     --config.file=/etc/prometheus/prometheus.yml \
>     --storage.tsdb.retention.time=1000d \
>     --storage.tsdb.path=/prometheus \
>     --storage.tsdb.max-block-duration=2h \
>     --storage.tsdb.min-block-duration=2h \
>     --web.listen-address=:9090 \
>     --web.external-url=https://2886795302-9090-elsy02.environments.katacoda.com \
>     --web.enable-lifecycle \
>     --web.enable-admin-api
744e16e490aa7f08639b9a575cde93fe52b6d2b8574b6d63946a34924b5bb678
[root@host01 ~]# 

访问web页面

3.启动Thanos Sidecar与Querier

[root@host01 ~]# docker run -d --net=host --rm \
>     --name prometheus-0-eu1-sidecar \
>     -u root \
>     quay.io/thanos/thanos:v0.20.0 \
>     sidecar \
>     --http-address 0.0.0.0:19090 \
>     --grpc-address 0.0.0.0:19190 \
>     --prometheus.url http://127.0.0.1:9090
f2384f126ff7de51f5ae98735a9f58228cad194793247cff75edd42ea77d1f6c
[root@host01 ~]# docker run -d --net=host --rm \
>     --name querier \
>     quay.io/thanos/thanos:v0.20.0 \
>     query \
>     --http-address 0.0.0.0:9091 \
>     --query.replica-label replica \
>     --store 127.0.0.1:19190
2d0128cb68121127ad6308499b94a63c15caedb54cdd9750611714f89bd80e22
[root@host01 ~]# 

访问web页面

4.对象存储持续备份

管理一年的数据是可行的，但不容易。长期调整大小，备份或维护此数据非常棘手。最重要的是，Prometheus不会进行任何复制，因此Prometheus的任何不可用都会导致查询不可用。
启动对象存储

[root@host01 ~]# mkdir /root/minio && \
> docker run -d --rm --name minio \
>      -v /root/minio:/data \
>      -p 9000:9000 -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=melovethanos" \
>      minio/minio:RELEASE.2019-01-31T00-31-19Z \
>      server /data
42e2fcc88bfd502a2dd6182a259c8dfa216c1541464d825c38985ad55bdcd140
[root@host01 ~]# 

创建thanos backet

[root@host01 ~]# mkdir /root/minio/thanos
[root@host01 ~]# 

访问web页面
登录，可查看到thanos bucket
编写配置文件

[root@host01 ~]# cat editor/bucket_storage.yaml 
type: S3
config:
  bucket: "thanos"
  endpoint: "127.0.0.1:9000"
  insecure: true
  signature_version2: true
  access_key: "minio"
  secret_key: "melovethanos"
[root@host01 ~]# 

重启sidecar

[root@host01 ~]# docker stop prometheus-0-eu1-sidecar
prometheus-0-eu1-sidecar
[root@host01 ~]# docker run -d --net=host --rm \
>     -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \
>     -v /root/prom-eu1:/prometheus \
>     --name prometheus-0-eu1-sidecar \
>     -u root \
>     quay.io/thanos/thanos:v0.20.0 \
>     sidecar \
>     --tsdb.path /prometheus \
>     --objstore.config-file /etc/thanos/minio-bucket.yaml \
>     --shipper.upload-compacted \
>     --http-address 0.0.0.0:19090 \
>     --grpc-address 0.0.0.0:19190 \
>     --prometheus.url http://127.0.0.1:9090
b6916de4c70fd715007ac370237e8b62cbaf9ee9cb45d30e1f51fbd73ba44ab5
[root@host01 ~]# 

回到web页面，可以看到数据都被上传到了thanos bucket
## 5.Thanos Store Gateway
接下来我们了解并部署Thanos Store Gateway。查看thanos命令

[root@host01 ~]# docker run --rm quay.io/thanos/thanos:v0.20.0 --help
usage: thanos [<flags>] <command> [<args> ...]

A block storage based long-term storage for Prometheus.

Flags:
  -h, --help               Show context-sensitive help (also try --help-long and
                           --help-man).
      --log.format=logfmt  Log format to use. Possible options: logfmt or json.
      --log.level=info     Log filtering level.
      --tracing.config=<content>  
                           Alternative to 'tracing.config-file' flag (mutually
                           exclusive). Content of YAML file with tracing
                           configuration. See format details:
                           https://thanos.io/tip/thanos/tracing.md/#configuration
      --tracing.config-file=<file-path>  
                           Path to YAML file with tracing configuration. See
                           format details:
                           https://thanos.io/tip/thanos/tracing.md/#configuration
      --version            Show application version.

Commands:
  help [<command>...]
    Show help.

  sidecar [<flags>]
    Sidecar for Prometheus server.

  store [<flags>]
    Store node giving access to blocks in a bucket provider. Now supported GCS,
    S3, Azure, Swift, Tencent COS and Aliyun OSS.

  query [<flags>]
    Query node exposing PromQL enabled Query API with data retrieved from
    multiple store nodes.

  rule [<flags>]
    Ruler evaluating Prometheus rules against given Query nodes, exposing Store
    API and storing old blocks in bucket.

  compact [<flags>]
    Continuously compacts blocks in an object store bucket.

  tools bucket verify [<flags>]
    Verify all blocks in the bucket against specified issues. NOTE: Depending on
    issue this might take time and will need downloading all specified blocks to
    disk.

  tools bucket ls [<flags>]
    List all blocks in the bucket.

  tools bucket inspect [<flags>]
    Inspect all blocks in the bucket in detailed, table-like way.

  tools bucket web [<flags>]
    Web interface for remote storage bucket.

  tools bucket replicate [<flags>]
    Replicate data from one object storage to another. NOTE: Currently it works
    only with Thanos blocks (meta.json has to have Thanos metadata).

  tools bucket downsample [<flags>]
    Continuously downsamples blocks in an object store bucket.

  tools bucket cleanup [<flags>]
    Cleans up all blocks marked for deletion.

  tools bucket mark --id=ID --marker=MARKER --details=DETAILS
    Mark block for deletion or no-compact in a safe way. NOTE: If the compactor
    is currently running compacting same block, this operation would be
    potentially a noop.

  tools bucket rewrite --id=ID [<flags>]
    Rewrite chosen blocks in the bucket, while deleting or modifying series
    Resulted block has modified stats in meta.json. Additionally
    compaction.sources are altered to not confuse readers of meta.json. Instead
    thanos.rewrite section is added with useful info like old sources and
    deletion requests. NOTE: It's recommended to turn off compactor while doing
    this operation. If the compactor is running and touching exactly same block
    that is being rewritten, the resulted rewritten block might only cause
    overlap (mitigated by marking overlapping block manually for deletion) and
    the data you wanted to rewrite could already part of bigger block.

    Use FILESYSTEM type of bucket to rewrite block on disk (suitable for vanilla
    Prometheus) After rewrite, it's caller responsibility to delete or mark
    source block for deletion to avoid overlaps. WARNING: This procedure is
    *IRREVERSIBLE* after certain time (delete delay), so do backup your blocks
    first.

  tools rules-check --rules=RULES
    Check if the rule files are valid or not.

  receive [<flags>]
    Accept Prometheus remote write API requests and write to local tsdb.

  query-frontend [<flags>]
    Query frontend command implements a service deployed in front of queriers to
    improve query parallelization and caching.


[root@host01 ~]# 

这里我们关注

  store [<flags>]
    Store node giving access to blocks in a bucket provider

部署Store

[root@host01 ~]# docker run -d --net=host --rm \
>     -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \
>     --name store-gateway \
>     quay.io/thanos/thanos:v0.20.0 \
>     store \
>     --objstore.config-file /etc/thanos/minio-bucket.yaml \
>     --http-address 0.0.0.0:19091 \
>     --grpc-address 0.0.0.0:19191
f304f8b7c791391c9caeb16b08b9464cbc33cb8003c0c61b9160c4eff96d7f1c
[root@host01 ~]# 

重启Querier

[root@host01 ~]# docker stop querier && \
> docker run -d --net=host --rm \
>    --name querier \
>    quay.io/thanos/thanos:v0.20.0 \
>    query \
>    --http-address 0.0.0.0:9091 \
>    --query.replica-label replica \
>    --store 127.0.0.1:19190 \
>    --store 127.0.0.1:19191
querier
0986bdcbb3a3c1d94e98418ad5cb35ca9efa63e5a1d45609e169e92dc4e71f97
[root@host01 ~]# 

访问Querier web页面

6.Thanos Compactor

Thanos Compactor用来对TSDB block data object storage进行压缩，保留，删除与降准采样操作。
部署

[root@host01 ~]# docker run -d --net=host --rm \
>  -v /root/editor/bucket_storage.yaml:/etc/thanos/minio-bucket.yaml \
>     --name thanos-compact \
>     quay.io/thanos/thanos:v0.20.0 \
>     compact \
>     --wait --wait-interval 30s \
>     --consistency-delay 0s \
>     --objstore.config-file /etc/thanos/minio-bucket.yaml \
>     --http-address 0.0.0.0:19095
2070cfd3c5728030f9ed5c790bbe1ca8c103dde5441739ba7800133ca6c26aed
[root@host01 ~]# 

访问Bucket View
点击块，可显示该块的metadata。

回到Querier web页面，我们查看continuous_app_metric0，勾选Enable Store Filtering

选择Max 5m downsampling，它使用5m分辨率，并且比原始数据要快。而且，下采样是建立在数据之上的，从来没有在年轻数据上进行过。

总结

由于Thanos store使用了对象存储，在Prometheus原始数据的同时，也可对数据进行压缩、降准采样、删除等等。Thanos Store使得数据可以无限存储下来。这样可以使得Prometheus近乎无状态。