Prometheus集群模式（联邦模式）部署日记-子节点篇

和主节点类似 部署docker-compose环境

部署子节点

# ls
docker-compose.yml  prometheus

# cat docker-compose.yml 
version: "3"
services:

  prometheus:
    #image: prom/prometheus:v2.0.0
    image: prom/prometheus:v2.18.1
    container_name: prometheus
    command: --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention=30d
    user: 0:0
    ports:
      - 9090:9090
    volumes:
      - /data/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - /data/prometheus/alerts/:/etc/prometheus/rules.d/
      - /data/prometheus/data/:/prometheus

  node-exporter:
    image: "prom/node-exporter:v1.3.1"
    #hostname: node-exporter
    container_name: node-exporter
    ports:
      - '9100:9100'
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
      - /etc/hostname:/etc/hostname:ro
    restart: always
    network_mode: host
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs

Prometheus配置

# cat prometheus/prometheus.yml
global:
  scrape_interval:     60s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'aws-prom-stack'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - 'rules.d/*'
  # - "second_rules.yml"

scrape_configs:
  - job_name: 'prom-stack'
    static_configs:
      - targets:
        - prometheus:9090
  
  - job_name: 'aws-pro-ec2-node-exporter'
    scrape_interval: 40s
    metrics_path: /metrics  #获取指标的url
    static_configs:
      - targets: ['x.x.x.x:9100','x.x.x.y:9100']

告警规则配置
这里声明下node up/down 必须在当前节点配置告警，联邦模式采集不到子节点的up/down，但是资源使用情况可以统一在master做，这些指标能被联邦模式一起采集

# cat prometheus/alerts/node-rule.yml 
###按照格式追加  ,instance!="x.x.xx.xx:9100" 即可，添加白名单
groups:
- name: example
  rules:
 # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up{instance!="x.x.x.x:9100",instance!="x.x.x.x:9100"} * on(instance) group_left(nodename) (node_uname_info)  == 0
    for: 5m
    labels:
      serverity: deadly
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

运行容器

docker-compose up -d