监控 ---- Spring Boot + JVM + Druid + Prometheus + Grafana

---

1、前置

Prometheus + Grafana 采用docker方式部署, 如果不会docker的话, 后面的内容会很吃力

篇幅有些长, 但按照步骤肯定是没有问题的, 左边或者右边会有对应的目录, 文章开头也有

效果图在第 7、8、9中, 可以直接点击效果图跳转查看

最后会列出各个文档, 以及对应的说明, 需要注意的地方

版本: 
docker pull prom/prometheus:latest # 大概是v2.29.*
docker pull grafana/grafana:latest # 大概是7.5.*

Spring Boot: 2.2.5.RELEASE
spring-boot-starter-actuator: 2.2.5.RELEASE
micrometer-registry-prometheus: 1.3.5
druid-spring-boot-starter: 1.1.24

2、运行项目

pom.xml

<parent>
   <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.2.5.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

<!-- 写博客测试用例, 就没有引入该包, 如果监控druid, 这个肯定是要滴 -->
<!--引入druid依赖
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>druid-spring-boot-starter</artifactId>
    <version>1.1.24</version>
</dependency>-->

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

application.yml:

server:
  port: 8080
  tomcat:
    # tomcat检测, 这个不开启可能会导致tomcat的指标获取不到
    mbeanregistry:
      enabled: true

management:
  endpoints:
    web:
      exposure:
        include: "*"
  server:
    port: 8088

运行项目, 这一步结束

3、部署Prometheus

1.prometheus.yml

# my global config
global:
  scrape_interval:     60s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: Test
     # 多久采集一次数据
    scrape_interval: 15s
    # 采集时的超时时间
    scrape_timeout: 10s
    # 采集的路径
    metrics_path: '/actuator/prometheus'
    # 采集服务的地址，设置成Springboot应用所在服务器的具体地址
    # 8088:指定了management.server.port就是该值, 没有指定就是server.port
    static_configs:
    - targets: ['192.168.1.30:8088']

关于这个文件如何在哪里找:
https://prometheus.io/docs/prometheus/latest/installation/
分类Using pre-compiled binaries然后点击download section解压后就会有一个prometheus.yml
或者

# 先运行prometheus
docker run -p 9090:9090 --name prometheus -d prom/prometheus
# 从容器中复制出来
# E:// windows路径 Linux 路径就是/***/***
docker cp prometheus:/etc/prometheus/prometheus.yml E://

2.运行

1、docker run 方式

docker run --name prometheus -p 9090:9090 -v E:\Docker\prometheus\prometheus\prometheus.yml:/etc/prometheus/prometheus.yml  -d prom/prometheus

# -v 将prometheus.yml文件挂载进去
# Linux 将E:\Docker\prometheus\prometheus\prometheus.yml 换成本地prometheus.yml对应的路径 /**/**/prometheus.yml

2、docker compose方式
docker-prometheus.yaml:

# Compose 版本 Version 2支持更多的指令。Version 1将来会被弃用。
version: "3"

# 定义服务
services:

  # 为project定义服务
  prometheus:
    # 服务的镜像名称或镜像ID。如果镜像在本地不存在，Compose将会尝试拉取镜像
    image: prom/prometheus
    # 配置端口 - "宿主机端口:容器暴露端口"
    ports:
      - 9090:9090
    # 挂载
    # Linux 将E:\Docker\prometheus\prometheus\prometheus.yml 换成本地prometheus.yml对应的路径 /**/**/prometheus.yml
    volumes:
      - "E://Docker/prometheus/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml"
    # 指定一个自定义容器名称，而不是生成的默认名称。
    container_name: prometheus

# docker-prometheus.yaml对应的路径下, 或者指定好路径, Linux一样
docker-compose -f docker-prometheus.yaml up -d

3.检查

1、查看日志

docker logs -f prometheus

2、检查是否获取到数据
浏览器:http://localhost:9090/

4、部署Grafana

1. 创建文件夹

用于保存数据, 比如我们删除容器然后在运行, 只要将该文件夹挂载进入, 就可以进行恢复操作
官网也有详细的说明: https://grafana.com/docs/grafana/v7.5/installation/docker/

E:\Docker\grafana\grafana
# Linux
/docker/grafana
chmod 775 -R ./grafana/

在Linux上这个一定是要注意的点
官网也有详细的介绍: https://grafana.com/docs/grafana/latest/installation/docker/#migrate-to-v51-or-later

2.运行

1、docker run 方式

docker run --name grafana -p 3000:3000 -v E:\Docker\grafana\grafana:/var/lib/grafana  -d grafana/grafana

# -v 将grafana文件夹挂载进去
# Linux 将E:\Docker\grafana\grafana 换成本地对应的路径 /**/**/grafana

2、docker compose方式
docker-grafana.yaml:

# Compose 版本 Version 2支持更多的指令。Version 1将来会被弃用。
version: "3"

# 定义服务
services:

  # 为project定义服务
  grafana:
    # 服务的镜像名称或镜像ID。如果镜像在本地不存在，Compose将会尝试拉取镜像
    image: grafana/grafana
    # 配置端口 - "宿主机端口:容器暴露端口"
    ports:
      - 3000:3000
    # 挂载
    # Linux 将E:\Docker\grafana\grafana 换成本地对应的路径 /**/**/grafana
    volumes:
      - "E://Docker/grafana/grafana:/var/lib/grafana"
    # 指定一个自定义容器名称，而不是生成的默认名称。
    container_name: grafana

# docker-grafana.yaml对应的路径下, 或者指定好路径, Linux一样
docker-compose -f docker-grafana.yaml up -d

3.检查

1、查看日志

docker logs -f grafana

2、检查是否获取到数据
浏览器:http://localhost:3000/
账号密码:admin
首次登录会强制修改密码

5、监控面板选择

面板挑选: https://grafana.com/grafana/dashboards

我最终选择的三块面板:

Spring Boot: 	https://grafana.com/grafana/dashboards/10280
JVM: 			https://grafana.com/grafana/dashboards/12856
Druid: 			https://grafana.com/grafana/dashboards/11157

6、配置Grafana数据源

7、配置Spring Boot面板

1、配置

2、效果图

8、配置JVM面板

同Spring Boot步骤一致

1、效果图

9、配置Druid面板

1、配置

同Spring Boot步骤一致, 不过需要注意点他需要集成源代码, 代码在druid面板介绍里面, 地址: http://filecdn.code2life.top/DruidMetrics.zip

package ---

import com.alibaba.druid.pool.DruidDataSource;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;

import java.util.List;
import java.util.function.ToDoubleFunction;

class DruidCollector {

    private static final String LABEL_NAME = "pool";

    private final List<DruidDataSource> dataSources;

    private final MeterRegistry registry;

    DruidCollector(List<DruidDataSource> dataSources, MeterRegistry registry) {
        this.registry = registry;
        this.dataSources = dataSources;
    }

    void register() {
        this.dataSources.forEach((druidDataSource) -> {
            // basic configurations
            createGauge(druidDataSource, "druid_initial_size", "Initial size", (datasource) -> (double) druidDataSource.getInitialSize());
            createGauge(druidDataSource, "druid_min_idle", "Min idle", datasource -> (double) druidDataSource.getMinIdle());
            createGauge(druidDataSource, "druid_max_active", "Max active", datasource -> (double) druidDataSource.getMaxActive());

            // connection pool core metrics
            createGauge(druidDataSource, "druid_active_count", "Active count", datasource -> (double) druidDataSource.getActiveCount());
            createGauge(druidDataSource, "druid_active_peak", "Active peak", datasource -> (double) druidDataSource.getActivePeak());
            createGauge(druidDataSource, "druid_pooling_peak", "Pooling peak", datasource -> (double) druidDataSource.getPoolingPeak());
            createGauge(druidDataSource, "druid_pooling_count", "Pooling count", datasource -> (double) druidDataSource.getPoolingCount());
            createGauge(druidDataSource, "druid_wait_thread_count", "Wait thread count", datasource -> (double) druidDataSource.getWaitThreadCount());

            // connection pool detail metrics
            createGauge(druidDataSource, "druid_not_empty_wait_count", "Not empty wait count", datasource -> (double) druidDataSource.getNotEmptyWaitCount());
            createGauge(druidDataSource, "druid_not_empty_wait_millis", "Not empty wait millis", datasource -> (double) druidDataSource.getNotEmptyWaitMillis());
            createGauge(druidDataSource, "druid_not_empty_thread_count", "Not empty thread count", datasource -> (double) druidDataSource.getNotEmptyWaitThreadCount());

            createGauge(druidDataSource, "druid_logic_connect_count", "Logic connect count", datasource -> (double) druidDataSource.getConnectCount());
            createGauge(druidDataSource, "druid_logic_close_count", "Logic close count", datasource -> (double) druidDataSource.getCloseCount());
            createGauge(druidDataSource, "druid_logic_connect_error_count", "Logic connect error count", datasource -> (double) druidDataSource.getConnectErrorCount());
            createGauge(druidDataSource, "druid_physical_connect_count", "Physical connect count", datasource -> (double) druidDataSource.getCreateCount());
            createGauge(druidDataSource, "druid_physical_close_count", "Physical close count", datasource -> (double) druidDataSource.getDestroyCount());
            createGauge(druidDataSource, "druid_physical_connect_error_count", "Physical connect error count", datasource -> (double) druidDataSource.getCreateErrorCount());

            // sql execution core metrics
            createGauge(druidDataSource, "druid_error_count", "Error count", datasource -> (double) druidDataSource.getErrorCount());
            createGauge(druidDataSource, "druid_execute_count", "Execute count", datasource -> (double) druidDataSource.getExecuteCount());
            // transaction metrics
            createGauge(druidDataSource, "druid_start_transaction_count", "Start transaction count", datasource -> (double) druidDataSource.getStartTransactionCount());
            createGauge(druidDataSource, "druid_commit_count", "Commit count", datasource -> (double) druidDataSource.getCommitCount());
            createGauge(druidDataSource, "druid_rollback_count", "Rollback count", datasource -> (double) druidDataSource.getRollbackCount());

            // sql execution detail
            createGauge(druidDataSource, "druid_prepared_statement_open_count", "Prepared statement open count", datasource -> (double) druidDataSource.getPreparedStatementCount());
            createGauge(druidDataSource, "druid_prepared_statement_closed_count", "Prepared statement closed count", datasource -> (double) druidDataSource.getClosedPreparedStatementCount());
            createGauge(druidDataSource, "druid_ps_cache_access_count", "PS cache access count", datasource -> (double) druidDataSource.getCachedPreparedStatementAccessCount());
            createGauge(druidDataSource, "druid_ps_cache_hit_count", "PS cache hit count", datasource -> (double) druidDataSource.getCachedPreparedStatementHitCount());
            createGauge(druidDataSource, "druid_ps_cache_miss_count", "PS cache miss count", datasource -> (double) druidDataSource.getCachedPreparedStatementMissCount());
            createGauge(druidDataSource, "druid_execute_query_count", "Execute query count", datasource -> (double) druidDataSource.getExecuteQueryCount());
            createGauge(druidDataSource, "druid_execute_update_count", "Execute update count", datasource -> (double) druidDataSource.getExecuteUpdateCount());
            createGauge(druidDataSource, "druid_execute_batch_count", "Execute batch count", datasource -> (double) druidDataSource.getExecuteBatchCount());

            // none core metrics, some are static configurations
            createGauge(druidDataSource, "druid_max_wait", "Max wait", datasource -> (double) druidDataSource.getMaxWait());
            createGauge(druidDataSource, "druid_max_wait_thread_count", "Max wait thread count", datasource -> (double) druidDataSource.getMaxWaitThreadCount());
            createGauge(druidDataSource, "druid_login_timeout", "Login timeout", datasource -> (double) druidDataSource.getLoginTimeout());
            createGauge(druidDataSource, "druid_query_timeout", "Query timeout", datasource -> (double) druidDataSource.getQueryTimeout());
            createGauge(druidDataSource, "druid_transaction_query_timeout", "Transaction query timeout", datasource -> (double) druidDataSource.getTransactionQueryTimeout());
        });
    }

    private void createGauge(DruidDataSource weakRef, String metric, String help, ToDoubleFunction<DruidDataSource> measure) {
        Gauge.builder(metric, weakRef, measure)
                .description(help)
                .tag(LABEL_NAME, weakRef.getName())
                .register(this.registry);
    }
}

package ---

import com.alibaba.druid.pool.DruidDataSource;
import io.micrometer.core.instrument.MeterRegistry;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

@Configuration
@ConditionalOnClass({DruidDataSource.class, MeterRegistry.class})
@Slf4j
public class DruidMetricsConfiguration {

    private final MeterRegistry registry;

    public DruidMetricsConfiguration(MeterRegistry registry) {
        this.registry = registry;
    }

    @Autowired
    public void bindMetricsRegistryToDruidDataSources(Collection<DataSource> dataSources) throws SQLException {
        List<DruidDataSource> druidDataSources = new ArrayList<>(dataSources.size());
        for (DataSource dataSource : dataSources) {
            DruidDataSource druidDataSource = dataSource.unwrap(DruidDataSource.class);
            if (druidDataSource != null) {
                druidDataSources.add(druidDataSource);
            }
        }
        DruidCollector druidCollector = new DruidCollector(druidDataSources, registry);
        druidCollector.register();
        log.info("finish register metrics to micrometer");
    }
}

2、效果图

10、备注

grafana官网: https://grafana.com/docs/grafana/latest/installation/docker/

prometheus官网: https://prometheus.io/docs/prometheus/latest/installation/

prometheus的GitHub: https://github.com/prometheus/prometheus

grafana的面板选择: https://grafana.com/grafana/dashboards

Druid的面板: https://grafana.com/grafana/dashboards/11157

Spring Boot 面板: https://grafana.com/grafana/dashboards/10280

JVM面板: https://grafana.com/grafana/dashboards/12856

因为 (Prometheus + Grafana + 项目) 采用的都是docker方式部署, 我配置的IP都是基于内网IP进行的通讯, 127.0.0.1 在容器内部只会指向容器内部的地址, 在我们网络没有进行特殊设置的时候

我是将Grafana 部署在一台服务器, (项目+Prometheus ) 另外一台服务器, 项目的健康监控8088端口不对外暴露, Prometheus 的9090端口, 只对Grafana 对应的服务器暴露. 在项目中对端口进行设置(或者说防火墙)可以避免后期很多问题,不管是技术还是售前售后

都是基于面板操作, 大家多玩玩就可以了, 对了, 监控的面板界面, 右上角+F11视觉效果很棒, 左上角可以切换

专栏后期会添加docker和Redis的监控, 基于Prometheus + Grafana