Promitheus+Grafana实现指标监控ClickHouse

监控效果图

软件部署情况：

机器IP	部署软件
192.168.242.152	ClickHouse（分片1及分片3的副本）、node_exporter、zookeeper_exporter、Docker、Promitheus、Grafana、Go、gocorn、gocorn-node
192.168.242.150	ClickHouse（分片2及分片1的副本）、node_exporter、zookeeper_exporter、Go、gocorn、gocorn-node
192.168.242.153	ClickHouse（分片3及分片2的副本）、node_exporter、zookeeper_exporter、Go、gocorn、gocorn-node

Promitheus:通过Exporte实现指标的提取(如：node_exporter、clickhouse-exporter、zookeeper_exporter)

#1.进入152的prometheus-2.21.0-rc.1.linux-amd64启动prometheus，如：nohup ./prometheus &

#2.监控系统资源，进入每个节点152,150,153的node_exporter-0.16.0-rc.3.linux-amd64启动node_exporter，如：nohup ./node_exporter &

#3.监控clickhouse，下载clickhouse-exporter监控插件 docker pull f1yegor/clickhouse-exporter

#单机启动多个clickhouse-exporter容器，用name区分。

docker run -d -p 9116:9116 f1yegor/clickhouse-exporter -scrape_uri=http://192.168.242.152:8123/

docker run --name=ck_export -d -p 9117:9116 f1yegor/clickhouse-exporter -scrape_uri=http://192.168.242.153:8123/

docker run --name=ck_export150 -d -p 9118:9116 f1yegor/clickhouse-exporter -scrape_uri=http://192.168.242.150:8123/

#4.监控zookeeper 

1.在152启动zookeeper_exporter(go语言版本）

nohup zookeeper_exporter -bind-addr :9141 -zookeeper 192.168.242.152:2181 &

nohup zookeeper_exporter -bind-addr :9142 -zookeeper 192.168.242.150:2181 &

nohup zookeeper_exporter -bind-addr :9143 -zookeeper 192.168.242.153:2181 &

#5.promethus中配置

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - scheme: http
    timeout: 10s
    api_version: v1
    static_configs:
    - targets: []
scrape_configs:
- job_name: prometheus
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 192.168.242.152:9090
- job_name: system_monitor
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 192.168.242.150:9100
    - 192.168.242.152:9100
    - 192.168.242.153:9100
- job_name: ch_monitor_150
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 192.168.242.152:9118
- job_name: ch_monitor_152
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 192.168.242.152:9116
- job_name: ch_monitor_153
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 192.168.242.152:9117
- job_name: zookeeper_monitor
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - 192.168.242.152:9141
    - 192.168.242.152:9142
    - 192.168.242.152:9143

#5.grafana中加入zookeeper_exporter的面板 9236 (https://grafana.com/grafana/dashboards/9236)

附1：gocron(简洁的脚本任务调度器)的启动

gocron web&
gocron-node -allow-root&

附2：grafana的启动

systemctl start grafana-server