Grafana+Prometheus监控mongodb
一,什么是exporter
如果你对Prometheus(普罗米修斯)监控体系有了解。它监控的基础是对各类服务的信息、数据进行采样收集,这一重要的角色就是Prometheus里的Exporter。而我们这里要用到社区的mongodb-exporter
Prometheus提供监控样本数据的程序都可以被称为一个Exporter。而Exporter的一个实例称为target,
- 社区提供的 Prometheus社区提供了丰富的Exporter实现,涵盖了从基础设施,中间件以及网络等各个方面的监控功能。这些Exporter可以实现大部分通用的监控需求。下表列举一些社区中常用的Exporter:
- 户自定义的 除了直接使用社区提供的Exporter程序以外,用户还可以基于Prometheus提供的Client Library创建自己的Exporter程序,目前Promthues社区官方提供了对以下编程语言的支持:Go、Java/Scala、Python、Ruby。独立研发的产品,为了满足监控需求,一般采用自定义的Exporter形式。
范围 | 常用的exporter |
数据库 | MySQL Exporter, Redis Exporter, MongoDB Exporter, MSSQL Exporter等 |
硬件 | Apcupsd Exporter,IoT Edison Exporter, IPMI Exporter, Node Exporter等 |
消息队列 | Beanstalkd Exporter, Kafka Exporter, NSQ Exporter, RabbitMQ Exporter等 |
HTTP服务 | Apache Exporter, HAProxy Exporter, Nginx Exporter等 |
API服务 | AWS ECS Exporter, Docker Cloud Exporter, Docker Hub Exporter, GitHub Exporter等 |
日志 | Fluentd Exporter, Grok Exporter等 |
监控系统 | Collectd Exporter, Graphite Exporter, InfluxDB Exporter, Nagios Exporter, SNMP Exporter等 |
其它 | Blockbox Exporter, JIRA Exporter, Jenkins Exporter, Confluence Exporter等 |
二,granfan dashboard示例
Grafana是一款用Go语言开发的开源数据可视化工具,可以做数据监控和数据统计,带有告警功能;官方和社区有提供大量的dashboard 模板,而我在使用社区的模板 mongodb-exporter: 2583 过程中用如果用eses/mongodb_exporter 镜像启动监控mongodb 没有数据,如果用mongodb_exporter 二进制包启动监控mongodb ,其展示的图形很多也没数据,所以我更改优化了dashboard 模板
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 1,
"id": 5,
"iteration": 1620463432162,
"links": [],
"panels": [
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": true,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"datasource": "Prometheus",
"decimals": null,
"format": "s",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 4,
"x": 0,
"y": 0
},
"id": 10,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"expr": "sum(mongodb_instance_uptime_seconds{job=\"$job\"})",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "",
"refId": "A",
"step": 60
}
],
"thresholds": "0,360",
"title": "Uptime",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"datasource": "Prometheus",
"decimals": null,
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 4,
"x": 4,
"y": 0
},
"id": 2,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": true,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "mongodb_mongod_connections{state=\"available\",job=\"$job\"}",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "",
"metric": "mongodb_connections",
"refId": "B",
"step": 60
}
],
"thresholds": "",
"title": "Available Connections",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "avg"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"datasource": "Prometheus",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 16,
"x": 8,
"y": 0
},
"id": 1,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": true,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "mongodb_mongod_connections{state=\"current\",job=\"$job\"}",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "",
"metric": "mongodb_connections",
"refId": "A",
"step": 60
}
],
"thresholds": "",
"title": "Open Connections",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "avg"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 10,
"x": 0,
"y": 4
},
"hiddenSeries": false,
"id": 7,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(mongodb_mongod_op_counters_total{job=\"$job\"}[5m])\r\n",
"format": "time_series",
"interval": "",
"intervalFactor": 4,
"legendFormat": "{{type}}",
"refId": "A",
"step": 10
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Query Operations",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 8,
"x": 10,
"y": 4
},
"hiddenSeries": false,
"id": 9,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(mongodb_mongod_metrics_document_total{job=\"$job\"}[5m])",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{state}}",
"refId": "A",
"step": 20
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Document Operations",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 6,
"x": 18,
"y": 4
},
"hiddenSeries": false,
"id": 8,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(mongodb_mongod_metrics_query_executor_total{job=\"$job\"}[5m])",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{state}}",
"refId": "A",
"step": 20
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Document Query Executor",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 10
},
"hiddenSeries": false,
"id": 6,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "avg(mongodb_mongod_replset_oplog_size_bytes{job=\"$job\"})",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{type}}",
"metric": "mongodb_locks_time_acquiring_global_microseconds_total",
"refId": "A",
"step": 20
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Oplog Size",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 10
},
"hiddenSeries": false,
"id": 4,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": false,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "mongodb_mongod_memory{job=\"$job\"}",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{type}}",
"refId": "A",
"step": 20
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Memory",
"tooltip": {
"shared": false,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": [
"total"
]
},
"yaxes": [
{
"format": "decmbytes",
"label": "MB",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 10
},
"hiddenSeries": false,
"id": 5,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(mongodb_mongod_network_bytes_total{job=\"$job\"}[5m])",
"format": "time_series",
"interval": "",
"intervalFactor": 2,
"legendFormat": "{{state}}",
"metric": "mongodb_metrics_operation_total",
"refId": "A",
"step": 20
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Network I/O",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"refresh": false,
"schemaVersion": 18,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": null,
"current": {
"selected": false,
"tags": [],
"text": "sz-mongodb-37017",
"value": [
"sz-mongodb-37017"
]
},
"datasource": "Prometheus",
"definition": "label_values(job)",
"hide": 0,
"includeAll": true,
"label": "job",
"multi": true,
"name": "job",
"options": [],
"query": "label_values(job)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "browser",
"title": "MongoDB-监控",
"uid": "YFfyWAnMz",
"version": 53
}
三、利用mongodb-exporter 监控mongodb
环境
服务器 centos7.8
1,安装prometheus
2,使用mongodb-exporter 的docker 镜像(eses/mongodb_exporter) 启动一个个监控客户端监控对应的mongodb实例
1、使用mongodb-exporter 镜像监控mongodb
- 准备镜像eses/mongodb_exporter
#linux服务器上拉取官方镜像
docker pull eses/mongodb_exporter
#查看镜像是否拉取成功
docker images
- 启动镜像监控mongodb实例
#如果注意 监听端口 9104:9104 ,前面的9104端口是服务器监听的端口不能重复,监听下一个mongodb实例则必须更改;而后面的9104是镜像容器里面的服务端口这个不能更改,
docker run -it -d --name mongo-ceshi -p 9104:9104 eses/mongodb_exporter --mongodb.uri mongodb://root:xxxxx@127.0.0.1:27017
docker run -it -d --name mongo-ceshi-02 -p 9106:9104 eses/mongodb_exporter --mongodb.uri mongodb://root:xxxxx@127.0.0.1:37017
#查看容器是否启动了和监听端口是否成功
docker ps
netstat -ntlp | grep 9104
netstat -ntlp | grep 9106
2、安装prometheus 收集mongodb-exporter 接口的数据
- 安装go环境
yum install go
go version
go version go1.6.3 linux/amd64
- 安装Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.11.0-rc.0/prometheus-2.11.0-rc.0.linux-amd64.tar.gz
tar xvf prometheus-2.11.0-rc.0.linux-amd64.tar.gz -C /usr/local/
ln -sv /usr/local/prometheus-2.11.0.linux-amd64/ /usr/local/prometheus
cd /usr/local/prometheus
- 修改配置文件
将监管的ip地址添加到prometheus.yml中,其中'127.0.0.1:9104'是mongodb_exporter监听端口,就是前面安装mongodb_exporter中监听的端口
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
#如果是要监控很多个mongodb实例,则参考以下3行,进行添加
- job_name: 'mongodb-ceshi'
static_configs:
- targets: ['127.0.0.1:9104']
启动服务
nohup ./prometheus --web.enable-lifecycle &
备注:启动时添加–web.enable-lifecycle可以自动加载配置文件,可以通过 curl -X POST http://localhost:9090/-/reload 重新加载配置
prometheus内置列一个web界面,可以通过http://install_host:9090访问。在Status->Targets页面中,我们可以看到配置的mongo1,状态为Down,说明未检测到数据。如何解决呢?需要安装好node_exporter,mongodb_exporter状态才会变为Up
3、安装grafana展示数据
-
安装grafana
-
wget https://dl.grafana.com/oss/release/grafana-6.2.5-1.x86_64.rpm sudo yum localinstall grafana-6.2.5-1.x86_64.rpm 启动后默认占用3000端口 sudo service grafana-server start - grafana显示mongodb数据
步骤1: 打开grafana前端页面http://install_host:3000
步骤2:在Data Sources添加数据源
-
步骤3: Create dashboard