prometheus监控搭建配置

1 consul下载安装
wget https://releases.hashicorp.com/consul/1.10.4/consul_1.10.4_linux_amd64.zip
unzip consul_1.10.4_linux_amd64.zip
mv consul /usr/local/bin/

启动
consul agent -server -ui -bind=0.0.0.0 -client=0.0.0.0 -bootstrap-expect=1 -data-dir=/data/work/consul_data -node=node1 -datacenter cbtest &

2 prometheus下载安装
https://prometheus.io/download/  （下载地址）
tar -zxf prometheus-2.31.1.linux-amd64.tar.gz
cd prometheus-2.31.1.linux-amd64

配置Prometheus通过consul实现自动服务发现
vim prometheus.yml

  - job_name: 'consul'    # 设置监控任务的名称
    consul_sd_configs:
      - server: '172.16.197.95:8500'
        services: []
    relabel_configs:
      - source_labels: [__meta_consul_service]
        regex: .*.*
        action: drop
      - source_labels: [__meta_consul_tags]
        target_label: job

3 启动服务：./prometheus --storage.tsdb.retention.time=180d --web.enable-admin-api --web.enable-lifecycle & 

安装完成，可以访问地址
http://172.16.197.95:8500/
http://172.16.197.95:9090/

4 下载配置模板文件
wget https://releases.hashicorp.com/consul-template/0.25.2/consul-template_0.25.2_linux_amd64.tgz
mv consul-template /usr/local/bin/

4.1 vim consul-template.conf
------------------------------------------------------------------------
log_level = "warn"
syslog {   # 启用syslog，这样服务日志可以记录到syslog里

enabled = true
# This is the name of the syslog facility to log to.
facility = "LOCAL5"
}
consul { 
# auth {
    # enabled = true
    # username = "test"
    # password = "test"
# }
address = "172.16.197.95:8500"  #consul地址
# token = "abcd1234"
retry {
    enabled = true
    attempts = 12
    backoff = "250ms" #backoff设置时间间隔，当未从consul获取到数据时会进行重试，并以2的倍数的时间间隔进行。比如设置250ms，重试5次
    # If max_backoff is set to 10s and backoff is set to 1s, sleep times
    # would be: 1s, 2s, 4s, 8s, 10s, 10s, ...
    #max_backoff = "3m"
}
}
template {
    source = "/data/work/prometheus-2.31.1.linux-amd64/template/prodjava-discovery.ctmpl" #模板文件地址
    destination = "/data/work/prometheus-2.31.1.linux-amd64/targets/prodjava-discovery.json" #输出的文件路径
    command = "" #文件渲染成功之后需要执行的命令。prometheus这里会自动发现文件的更改，所以我这里无需任何命令，给注释掉了。像nginx、haproxy之类的服务，一般更改完配置文件之后都需要重启，这里可以设置“nginx -s reload”之类的命令
    command_timeout = "60s"
    backup = true #当backup=true的时候，会备份上一次的配置，并以bak后缀结尾
    left_delimiter = "{$" 
    right_delimiter = "$}" #模板文件中分隔符。默认是用“{{}}”设置模板，当产生冲突的时候可以更改这里的设置。比如我这里由于用ansible去推送的模板文件，“{{}}”符号与Jinja2的语法产生了冲突，所以改为了“{$$}”符号
wait {
    min = "2s"
    max = "20s"
}
}
-------------------------------------------------------------------------
4.2 mkdir template targets
vim template/prodjava-discovery.ctmpl
-----------------------------------------------------
[
{$ range tree "prometheus/prodjava" $}
{
"targets": ["{$ .Value $}"],
"labels": {
"instance": "{$ .Key $}"
}
},
{$ end $}
{
"targets": ["172.16.197.95:9090"],
"labels": {
"instance": "prometheus01"
}
}
]
---------------------------------------------------
consul-template -config /data/work/prometheus-2.31.1.linux-amd64/consul-template.conf &

4.3 vim promethues.yml
-------------------------------
- job_name: 'linux'
    file_sd_configs:
     - files:
        - "targets/prodjava-discovery.json"
       refresh_interval: 5m
--------------------------------
        
4下载客户端node_exporter-1.3.1.linux-amd64.tar.gz并解压
启动服务 ./node_exporter &

5下载安装grafana
wget https://dl.grafana.com/oss/release/grafana-8.3.2-1.x86_64.rpm
yum install grafana-8.3.2-1.x86_64.rpm
启动服务：systemctl start grafana-server.service
浏览器访问：http://172.16.197.95:3000       admin / admin

5.1 配置grafana

5.1.1下载页面展示模板

官网下载地址：https://grafana.com/

 

 

 

5.1.2  grafana页面导入模板文件

 

选择下载的模板文件

 

5.2 添加prometheus数据源