【Prometheus】

1、问题描述

升级 grafana-server 到 8.3.1，仪表盘 MySQL 相关监控信息展示异常，监控仪表盘如图所示：

部分监控信息展示异常，报错信息如图：

2 、问题定位

2.1 检查Prometheus监控状态

Prometheus界面Targets mysql_exporter 插件信息收集正常

2.2 检查grafana-server状态

grafana服务有报错

[root@dev-dba-office~]# systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
   Active: active (running) since 四 2022-01-20 11:34:04 CST; 5h 14min ago
     Docs: http://docs.grafana.org
 Main PID: 21023 (grafana-server)
   CGroup: /system.slice/grafana-server.service
           └─21023 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins cfg:default.paths.provisioning=/etc/grafana/provisioning

1月 20 16:45:07 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:07+0800 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=admin error="failed to query data: time: invalid duration \"$interval\"" remote_addr=192.168.*.*
1月 20 16:45:07 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:07+0800 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=500 remote_addr=192.168.*.* time_ms=2 size=34 referer="http://dev-dba-monitor.cn:3000/d/d_jIN8eMz/mysql-jian-kong?orgId=1&refresh=1m"
1月 20 16:45:07 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:07+0800 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=admin error="failed to query data: time: invalid duration \"$interval\"" remote_addr=192.168.*.*
1月 20 16:45:07 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:07+0800 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=500 remote_addr=192.168.*.* time_ms=0 size=34 referer="http://dev-dba-monitor.cn:3000/d/d_jIN8eMz/mysql-jian-kong?orgId=1&refresh=1m"
1月 20 16:45:07 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:07+0800 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=admin error="failed to query data: time: invalid duration \"$interval\"" remote_addr=192.168.*.*
1月 20 16:45:07 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:07+0800 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=500 remote_addr=192.168.*.* time_ms=0 size=34 referer="http://dev-dba-monitor.cn:3000/d/d_jIN8eMz/mysql-jian-kong?orgId=1&refresh=1m"
1月 20 16:45:08 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:08+0800 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=admin error="failed to query data: time: invalid duration \"$interval\"" remote_addr=192.168.*.*
1月 20 16:45:08 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:08+0800 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=500 remote_addr=192.168.*.* time_ms=1 size=34 referer="http://dev-dba-monitor.cn:3000/d/d_jIN8eMz/mysql-jian-kong?orgId=1&refresh=1m"
1月 20 16:45:08 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:08+0800 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=admin error="failed to query data: time: invalid duration \"$interval\"" remote_addr=192.168.*.*
1月 20 16:45:08 dev-dba-office grafana-server[21023]: t=2022-01-20T16:45:08+0800 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=500 remote_addr=192.168.*.* time_ms=0 size=34 referer="http://dev-dba-monitor.cn:3000/d/d_jIN8eMz/mysql-jian-kong?orgId=1&refresh=1m"

3、问题解决

修改Grafana 的数据源 Data Sources ，将Access改成：Browser

Help：
Access mode controls how requests to the data source will be handled. Server should be the preferred way if nothing else is stated.

Server access mode (Default):
All requests will be made from the browser to Grafana backend/server which in turn will forward the requests to the data source and by that circumvent possible Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the grafana backend/server if you select this access mode.

Browser access mode:
All requests will be made from the browser directly to the data source and may be subject to Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the browser if you select this access mode.

更改后，监控图像恢复正常