搭建系统性能监控 Prometheus+Grafana

1. 安装Prometheus

1.1 安装启动node-exporter

https://github.com/prometheus/node_exporter

docker run -d \
  --name node_exporter \
  --restart always \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter \
  --path.rootfs=/host

访问 http://ip:9100/metrics 即可看到搜集到的数据

1.2 编辑 prometheus.yml

global:
  scrape_interval:     60s
  evaluation_interval: 60s
 
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
 
  - job_name: linux
    static_configs:
      - targets: ['localhost:9100']
        labels:
          instance: localhost

1.3 安装启动prometheus

docker run -d \
  --name prometheus \
  --restart always \
  -p 9090:9090 \
  -v /root/software/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml  \
  prom/prometheus

访问 http://ip:9090/graph 即可看到 graph 页面，如果看到状态为up则证明启动成功

2. 安装Grafana

2.1 新建数据存储文件夹

新建一个文件夹用来存储grafana数据

mkdir data

# 授权
chmod 777 -R data

2.2 安装启动grafana

docker run -d \
  --name grafana\
  --restart always \
  -p 3000:3000 \
  -v /root/software/prometheus/data:/var/lib/grafana \
  grafana/grafana

访问http://ip:3000/ 即可看到 grafana 登陆页面
默认用户名和密码都为admin，登陆之后会让你重置密码，重置之后就会进入到首页了

2.3 配置grafana

将Prometheus收集的数据配置给grafana

1. 新增一个 data source
   

2. 选择Prometheus
   

3. 输入ip+端口

这个地址是我们前面配置的prometheus访问地址

4. 保存

如果出现 data source is working 证明配置没有问题

5. 新建一个dashboard

回到首页点击 New dashboard

选择图表形式

即可看到主页面

6. 新增一个监控项

输入process_cpu_seconds_total 表示监控 用户和系统CPU花费的时间，点击Add Query可以再新增一个监控项，这里我们再新增一个node_disk_reads_merged_total 磁盘的读取，我们会看到表盘上会有两条线，记录这个监控项的走势

点击fix by adding rate()

修改表盘名称

3. 导入监控项

可直接去官网下载常用的监控模板，导入即可

• 常用模板
• 导入文档

4. grafana常用语法

https://www.kancloud.cn/huyipow/prometheus/525003
https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/promql/prometheus-promql-operators-v2

4.1 请求QPS

sum(rate(gin_requests_total{namespace="[[namespace]]",service=~"[[service]]", instance=~"[[instance]]", url=~"$endpoint"}[$interval])) by (url)
and
(sum(rate(gin_request_duration_seconds_sum{namespace="[[namespace]]",service=~"[[service]]", instance=~"[[instance]]", url=~"$endpoint"}[$interval])) by (url)
/
sum(rate(gin_request_duration_seconds_count{namespace="[[namespace]]",service=~"[[service]]", instance=~"[[instance]]", url=~"$endpoint"}[$interval])) by (url)) * 1000 >= $rt

4.2 接口平均RT

sum(rate(gin_request_duration_seconds_sum{namespace="[[namespace]]",service=~"[[service]]", instance=~"[[instance]]", url=~"$endpoint"}[$interval])) by (url)
/
sum(rate(gin_request_duration_seconds_count{namespace="[[namespace]]",service=~"[[service]]", instance=~"[[instance]]", url=~"$endpoint"}[$interval])) by (url)

4.3 调用接口QPS

sum(rate(gorequest_requested_count_total{job="[[service]]",namespace="[[namespace]]", instance=~"[[instance]]",endpoint=~"$endpoint"}[$interval])) by (uri)

4.4 调用接口平均RT

sum(rate(gorequest_requested_duration_seconds_sum{job=~"[[service]]", instance=~"[[instance]]",endpoint=~"$endpoint"}[$interval])) by (uri)
/
sum(rate(gorequest_requested_duration_seconds_count{job=~"[[service]]", instance=~"[[instance]]",endpoint=~"$endpoint"}[$interval])) by (uri)

节点维度

sum(rate(gorequest_requested_duration_seconds_sum{job=~"[[service]]", instance=~"[[instance]]",endpoint=~"$endpoint"}[$interval])) by (instance)
/
sum(rate(gorequest_requested_duration_seconds_count{job=~"[[service]]", instance=~"[[instance]]",endpoint=~"$endpoint"}[$interval])) by (instance)

4.5 调用接口错误率

sum(rate(gorequest_requested_count_total{job="[[service]]",namespace="[[namespace]]", instance=~"[[instance]]",endpoint=~"$endpoint",status!~"200"}[$interval])) by (uri)
/
sum(rate(gorequest_requested_count_total{job="[[service]]",namespace="[[namespace]]", instance=~"[[instance]]",endpoint=~"$endpoint"}[$interval])) by (uri)

4.6 Mysql连接状态

sum(go_mysql_total_conn{namespace="$namespace",service="$service", instance=~"$instance"}) by (instance)