使用Prometheus+Grafana 监控MySQL/MONGODB

1. 安装GO编译环境

wget https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz
tar -zxvf go1.10.3.linux-amd64.tar.gz
mv go /data/app/go

vim /etc/profile #添加环境变量

export GOROOT="/data/app/go"
export GOPATH="/data/app/go-path"
export PATH=$PATH:/data/app/go/bin

2.编译mongodb_exporter

go get github.com/percona/mongodb_exporter
cd /data/app/go-path/src/github.com/percona/mongodb_exporter/
make
mv mongodb_exporter /data/mongodb_exporter/

3. 通过supervisor启动mongodb_exporter

[program:mongodb-exporter]
directory = /data/exporter/
command= /data/exporter/bin/mongodb_exporter
user=myuser
process_name=%(program_name)s
redirect_stderr=true
autorestart=true
stdout_logfile=/data/exporter/logs/mongodb-exporter.log

4.部署和配置prometheus

cat conf/mongodb.json

[
  {
    "targets": ["xxx:9216"],
    "labels": {
      "instance": "vir-mongo1"
    }
  },
  {
    "targets": ["xxx:9216"],
    "labels": {
      "instance": "vir-mongo3"
    }
  },
  {
    "targets": ["xxx:9216"],
    "labels": {
      "instance": "vir-mongo4"
    }
  },
  {
    "targets": ["xxx:9216"],
    "labels": {
      "instance": "vir-mongo5"
    }
  },
  {
    "targets": ["xxx:9216"],
    "labels": {
      "instance": "vir-mongoDB-ETL"
    }
  }
]

cat conf/prometheus.yml

  - job_name: mongodb
    scrape_interval: 15s
    file_sd_configs:
    - files:
      - mongodb.json

5.prometheus的联邦模式介绍。

文章引用地址：http://blog.51cto.com/lee90/2062252

分层联邦：

分层联邦允许普罗米修斯扩展到数十个数据中心和数百万个节点的环境。在这种用例中，联邦拓扑类似于一棵树，更高级别的普罗米修斯服务器从大量的从属服务器收集汇总的时间序列数据。

跨服务联邦：

In cross-service federation, a Prometheus server of one service is configured to scrape selected data from another service’s Prometheus server to enable alerting and queries against both datasets within a single server.

在跨服务联合中，一个服务的普罗米修斯服务器被配置为从另一个服务的普罗米修斯服务器中刮取选定的数据，以使得能够针对单个服务器内的两个数据集进行警报和查询。

For example, a cluster scheduler running multiple services might expose resource usage information (like memory and CPU usage) about service instances running on the cluster. On the other hand, a service running on that cluster will only expose application-specific service metrics. Often, these two sets of metrics are scraped by separate Prometheus servers. Using federation, the Prometheus server containing service-level metrics may pull in the cluster resource usage metrics about its specific service from the cluster Prometheus, so that both sets of metrics can be used within that server.

例如，运行多个服务的集群调度程序可能会暴露有关在集群上运行的服务实例的资源使用情况信息（如内存和CPU使用情况）。另一方面，在该群集上运行的服务将仅公开特定于应用程序的服务度量标准。通常，这两套指标是由单独的普罗米修斯服务器来抓取的。使用联邦，包含服务级别度量的普罗米修斯服务器可以从集群普罗米修斯拉入有关其特定服务的集群资源使用度量，以便这两组度量可以在该服务器内使用。

【举个例子：我们要监控mysqld的运行状态，可以使用1个主Prometheus+2个分片Prometheus(一个用来采集node_exporter的metrics、一个用来采集mysql_exporter的metrics)，然后在主Prometheus上做汇总】

6.配置prometheus的联邦模式，应用于2机房。

cat conf/prometheus.yml

scrape_configs:
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
      labels:
        instance: vir-prometheus1

  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~"prometheus.*"}'
        - '{job="mongodb"}'
    static_configs:
      - targets:
        - 'xxx:9090'

  - job_name: mongodb
    scrape_interval: 15s
    file_sd_configs:
    - files:
      - mongodb.json

7.安装Grafana.

下载percona的监控面板，

wget https://codeload.github.com/percona/grafana-dashboards/zip/master 
导入dashboards目录下面的Mongodb相关dashbords.
调整相关Links和Host对应值
我这边用的host值是：label_values({__name__=~"mongodb_mongod_connections{cluster=~\"$cluster\"}|mongodb_up"}, instance)