指标监控 & 日志采集系统

最新推荐文章于 2023-05-01 18:27:08 发布

Coder_Joker

最新推荐文章于 2023-05-01 18:27:08 发布

阅读量433

点赞数

分类专栏： demo

本文链接：https://blog.csdn.net/coder_joker/article/details/108507825

版权

demo 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

指标监控 & 日志采集

TODO

AlterManager 告警添加
K8S 动态发现指标
日志上传

Prometheus

docker 安装: docker pull prom/prometheus,以及node-exporter: docker pull prom/node-exporter

启动node-exporter:

docker run -itd --name node_exporter -p 9100:9100 prom/node-exporter

编写yaml文件:

编写规则,就是要往哪个服务器监控,则编写一个相应的job_name 即可

global:
  scrape_interval:     60s
  evaluation_interval: 60s

scrape_configs:
  - job_name: prometheus-server
    static_configs:
      - targets: ['localhost:9090','localhost:9100']  # prometheus 的启动端口,本机的端口号,以及node_exporter
        labels:
          instance: prometheus

  - job_name: demo-server
    static_configs:
      - targets: ['192.168.1.101:8080']	# 目标服务器
        labels:
          instance: localhost

启动docker 文件
- docker run -itd --name prometheus -p 9090:9090 -v ${PWD}/prometheus.yaml:/etc/prometheus/prometheus.yml prom/prometheus
访问: http://localhost:9000/graph

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cC89Hd6b-1600522319989)(/Users/joker/Desktop/未闻花名/个人/指标和监控/graph.png)]
访问 http://localhost:9000/targets, down 先不用管,那是自定义的

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-c2pPqPBB-1600522319992)(/Users/joker/Desktop/未闻花名/个人/指标和监控/targets.png)]

Grafana

拉取grafana : docker pull grafana/grafana

启动grafana :

docker run -itd -p 3000:3000 --name=grafana  grafana/grafana

访问,添加DataSource

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-nuWbZ3hK-1600522319994)(/Users/joker/Desktop/未闻花名/个人/指标和监控/Grafana-addDataSource.png)]
- 主要为URL 即可,要与 prometheus 的一致
- 添加dashboard=> 选择graph => query 更改server 为prometheus server
  
  [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1lWemGAy-1600522319996)(/Users/joker/Desktop/未闻花名/个人/指标和监控/graph-添加query条件.png)]

添加自定义指标监控

编写程序

package main

import (
	"flag"
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"log"
	"net/http"
	"time"

)

var addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.")

var requestTotalCount = prometheus.NewCounterVec(prometheus.CounterOpts{
	Name: "reqeust_total_count", // 指定名字
	Help: "Number of hello requests in total",
},
	// 设置两个标签 请求方法和 路径 对请求总次数在两个
	[]string{"method", "endpoint"},
)

var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
	Namespace:   "request_duration",
	Subsystem:   "",
	Name:        "request_duration",
	Help:        "",
	ConstLabels: nil,
	Buckets:     nil,
}, []string{"method", "endpoint"})

func init() {
	// 注册监控指标
	prometheus.MustRegister(requestTotalCount)
	prometheus.MustRegister(requestDuration)
}

func main() {
	flag.Parse()
	http.Handle("/metrics", promhttp.Handler())
	http.Handle("/get", Filter(Get))
	log.Fatal(http.ListenAndServe(*addr, nil))
}

func Get(writer http.ResponseWriter, request *http.Request) {
	writer.Write([]byte("ok"))
}

func Filter(h http.HandlerFunc) http.HandlerFunc {
	return func(writer http.ResponseWriter, request *http.Request) {
		start := time.Now()
		h(writer, request)
		duration := time.Since(start)
		requestTotalCount.With(prometheus.Labels{"method": request.Method, "endpoint": request.URL.Path}).Inc()
		requestDuration.With(prometheus.Labels{"method": request.Method, "endpoint": request.URL.Path}).Observe(duration.Seconds())
	}
}

在grafana 中添加dashboard
- [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-d37uryWd-1600522319997)(/Users/joker/Desktop/未闻花名/个人/指标和监控/自定义指标.png)]

遇到的问题

为什么要使用node-exporter
- 因为内部提供了系统监控的相关信息,就不用自己写了
怎么导入 node-exporter的东西
- 需要导入的是json文件即可
- [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rTAXMJ6H-1600522319998)(/Users/joker/Desktop/未闻花名/个人/指标和监控/导入node-exporter 1.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-msEsYzqi-1600522319999)(/Users/joker/Desktop/未闻花名/个人/指标和监控/导入node-exporter 2.png)]