前言
之前使用springcloudalibaba搭建了一套新项目,项目使用Prometheus做监控数据存储,使用grafana显示数据
在这里,我就不介绍怎么搭建grafana和Prometheus,我们这里是运维搭建的,仅仅演示一下springcloud项目的常用监控埋点配置可视化
springcloud版本
<spring-boot.version>2.2.10.RELEASE</spring-boot.version>
<spring-cloud.version>Hoxton.SR4</spring-cloud.version>
<spring-cloud-alibaba.version>2.2.5.RELEASE</spring-cloud-alibaba.version>
查看监控埋点
在这里,作为举例,我就查看一个应用的监控埋点数据
调应用接口http://localhost:8095/export/actuator/prometheus,以下是没有手动添加埋点springcloud的默认监控埋点数据
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total{module="call-exportcenter",} 0.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total{module="call-exportcenter",} 4.1585576E7
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads{module="call-exportcenter",} 173.0
# HELP tomcat_sessions_alive_max_seconds
# TYPE tomcat_sessions_alive_max_seconds gauge
tomcat_sessions_alive_max_seconds{module="call-exportcenter",} 0.0
# HELP tomcat_sessions_expired_sessions_total
# TYPE tomcat_sessions_expired_sessions_total counter
tomcat_sessions_expired_sessions_total{module="call-exportcenter",} 0.0
# HELP tomcat_sessions_created_sessions_total
# TYPE tomcat_sessions_created_sessions_total counter
tomcat_sessions_created_sessions_total{module="call-exportcenter",} 0.0
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",module="call-exportcenter",} 1.2767232E7
jvm_memory_committed_bytes{area="heap",id="Par Eden Space",module="call-exportcenter",} 2.14827008E8
jvm_memory_committed_bytes{area="heap",id="CMS Old Gen",module="call-exportcenter",} 1.879048192E9
jvm_memory_committed_bytes{area="nonheap",id="Code Cache",module="call-exportcenter",} 4.2401792E7
jvm_memory_committed_bytes{area="heap",id="Par Survivor Space",module="call-exportcenter",} 2.6804224E7
jvm_memory_committed_bytes{area="nonheap",id="Metaspace",module="call-exportcenter",} 9.8537472E7
# HELP tomcat_sessions_active_current_sessions
# TYPE tomcat_sessions_active_current_sessions gauge
tomcat_sessions_active_current_sessions{module="call-exportcenter",} 0.0
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage{module="call-exportcenter",} 0.004332755632582322
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total{module="call-exportcenter",} 2.1063741444E11
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds{module="call-exportcenter",} 453481.834
# HELP jvm_gc_concurrent_phase_time_seconds Time spent in concurrent phase
# TYPE jvm_gc_concurrent_phase_time_seconds summary
jvm_gc_concurrent_phase_time_seconds_count{action="end of major GC",cause="No GC",module="call-exportcenter",} 1.0
jvm_gc_concurrent_phase_time_seconds_sum{action="end of major GC",cause="No GC",module="call-exportcenter",} 0.527
# HELP jvm_gc_concurrent_phase_time_seconds_max Time spent in concurrent phase
# TYPE jvm_gc_concurrent_phase_time_seconds_max gauge
jvm_gc_concurrent_phase_time_seconds_max{action="end of major GC",cause="No GC",module="call-exportcenter",} 0.0
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files{module="call-exportcenter",} 362.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{id="direct",module="call-exportcenter",} 841288.0
jvm_buffer_total_capacity_bytes{id="mapped",module="call-exportcenter",} 0.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",module="call-exportcenter",} 1.1903096E7
jvm_memory_used_bytes{area="heap",id="Par Eden Space",module="call-exportcenter",} 7.2530264E7
jvm_memory_used_bytes{area="heap",id="CMS Old Gen",module="call-exportcenter",} 4.8868704E7
jvm_memory_used_bytes{area="nonheap",id="Code Cache",module="call-exportcenter",} 4.086976E7
jvm_memory_used_bytes{area="heap",id="Par Survivor Space",module="call-exportcenter",} 8070536.0
jvm_memory_used_bytes{area="nonheap",id="Metaspace",module="call-exportcenter",} 9.3782344E7
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="direct",module="call-exportcenter",} 841290.0
jvm_buffer_memory_used_bytes{id="mapped",module="call-exportcenter",} 0.0
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{module="call-exportcenter",state="runnable",} 32.0
jvm_threads_states_threads{module="call-exportcenter",state="timed-waiting",} 30.0
jvm_threads_states_threads{module="call-exportcenter",state="blocked",} 0.0
jvm_threads_states_threads{module="call-exportcenter",state="waiting",} 111.0
jvm_threads_states_threads{module="call-exportcenter",state="new",} 0.0
jvm_threads_states_threads{module="call-exportcenter",state="terminated",} 0.0
# HELP tomcat_sessions_rejected_sessions_total
# TYPE tomcat_sessions_rejected_sessions_total counter
tomcat_sessions_rejected_sessions_total{module="call-exportcenter",} 0.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds{module="call-exportcenter",} 1.642036661381E9
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes{module="call-exportcenter",} 17309.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Allocation Failure",module="call-exportcenter",} 981.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Allocation Failure",module="call-exportcenter",} 8.495
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Allocation Failure",module="call-exportcenter",} 0.008
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{level="debug",module="call-exportcenter",} 0.0
logback_events_total{level="info",module="call-exportcenter",} 1103.0
logback_events_total{level="error",module="call-exportcenter",} 3.0
logback_events_total{level="trace",module="call-exportcenter",} 0.0
logback_events_total{level="warn",module="call-exportcenter",} 13.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes{module="call-exportcenter",} 1.879048192E9
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m{module="call-exportcenter",} 0.0
# HELP http_server_requests_seconds
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/minio/uploadFile",} 4.0
http_server_requests_seconds_sum{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/minio/uploadFile",} 0.481584833
http_server_requests_seconds_count{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/minio/downloadFile",} 27.0
http_server_requests_seconds_sum{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/minio/downloadFile",} 0.435441757
http_server_requests_seconds_count{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/login/getAppId",} 4.0
http_server_requests_seconds_sum{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/login/getAppId",} 0.016209059
http_server_requests_seconds_count{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/sendSmsCode",} 2.0
http_server_requests_seconds_sum{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/sendSmsCode",} 0.377353323
http_server_requests_seconds_count{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/send",} 8.0
http_server_requests_seconds_sum{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/send",} 1.013747286
http_server_requests_seconds_count{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/ffmpeg/getDuration/dto",} 7.0
http_server_requests_seconds_sum{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/ffmpeg/getDuration/dto",} 0.101806506
http_server_requests_seconds_count{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/boss/callforward",} 4.0
http_server_requests_seconds_sum{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/boss/callforward",} 3.448163758
http_server_requests_seconds_count{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/ffmpeg/wavtomp3",} 18.0
http_server_requests_seconds_sum{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/ffmpeg/wavtomp3",} 1.182240623
http_server_requests_seconds_count{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/iat/asr/base64",} 3.0
http_server_requests_seconds_sum{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/iat/asr/base64",} 5.338831925
http_server_requests_seconds_count{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/miniApp/getUserInfo",} 1.0
http_server_requests_seconds_sum{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/miniApp/getUserInfo",} 0.983358855
http_server_requests_seconds_count{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 151158.0
http_server_requests_seconds_sum{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 743.14145058
http_server_requests_seconds_count{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/verifySmsCode",} 21.0
http_server_requests_seconds_sum{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/verifySmsCode",} 0.281737548
# HELP http_server_requests_seconds_max
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/minio/uploadFile",} 0.0
http_server_requests_seconds_max{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/minio/downloadFile",} 0.0
http_server_requests_seconds_max{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/login/getAppId",} 0.0
http_server_requests_seconds_max{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/sendSmsCode",} 0.0
http_server_requests_seconds_max{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/send",} 0.0
http_server_requests_seconds_max{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/ffmpeg/getDuration/dto",} 0.0
http_server_requests_seconds_max{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/boss/callforward",} 0.0
http_server_requests_seconds_max{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/ffmpeg/wavtomp3",} 0.0
http_server_requests_seconds_max{exception="None",method="POST",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/iat/asr/base64",} 0.0
http_server_requests_seconds_max{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/miniApp/getUserInfo",} 0.0
http_server_requests_seconds_max{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.02309154
http_server_requests_seconds_max{exception="None",method="GET",module="call-exportcenter",outcome="SUCCESS",status="200",uri="/cmpp/verifySmsCode",} 0.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage{module="call-exportcenter",} 0.007267441860465116
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes{module="call-exportcenter",} 1.723964E7
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads{module="call-exportcenter",} 157.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files{module="call-exportcenter",} 11000.0
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",module="call-exportcenter",} 1.073741824E9
jvm_memory_max_bytes{area="heap",id="Par Eden Space",module="call-exportcenter",} 2.14827008E8
jvm_memory_max_bytes{area="heap",id="CMS Old Gen",module="call-exportcenter",} 1.879048192E9
jvm_memory_max_bytes{area="nonheap",id="Code Cache",module="call-exportcenter",} 2.5165824E8
jvm_memory_max_bytes{area="heap",id="Par Survivor Space",module="call-exportcenter",} 2.6804224E7
jvm_memory_max_bytes{area="nonheap",id="Metaspace",module="call-exportcenter",} -1.0
# HELP tomcat_sessions_active_max_sessions
# TYPE tomcat_sessions_active_max_sessions gauge
tomcat_sessions_active_max_sessions{module="call-exportcenter",} 0.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="direct",module="call-exportcenter",} 65.0
jvm_buffer_count_buffers{id="mapped",module="call-exportcenter",} 0.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count{module="call-exportcenter",} 8.0
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads{module="call-exportcenter",} 174.0
通过以上以上埋点我们可以知道,springcloud默认给我们添加了很多埋点,挺丰富的,也足够使用了,接下来我们就配置几个常用的埋点监控。
常用grafana配置
错误日志量/秒监控
配置参数:sum(irate(logback_events_total{module="$module",level=“error”}[1m]))
服务接口tps配置
sum(rate(http_server_requests_seconds_count{module="$module"}[5m]))by(uri)
tomcat当前线程配置
tomcat_threads_current_threads{module="$module"}
打开文件数配置
process_files_open_files{module="$module"}
接口耗时配置
sum(rate(http_server_requests_seconds_sum{module=“KaTeX parse error: Expected 'EOF', got '}' at position 8: module"}̲[5m]))by(uri)/s…module”}[5m]))by(uri)
效果如下: