指标监控 & 日志采集
TODO
- AlterManager 告警添加
- K8S 动态发现指标
- 日志上传
Prometheus
-
docker 安装: docker pull prom/prometheus,以及node-exporter: docker pull prom/node-exporter
-
启动node-exporter:
-
docker run -itd --name node_exporter -p 9100:9100 prom/node-exporter
-
-
编写yaml文件:
-
编写规则,就是要往哪个服务器监控,则编写一个相应的job_name 即可
-
global: scrape_interval: 60s evaluation_interval: 60s scrape_configs: - job_name: prometheus-server static_configs: - targets: ['localhost:9090','localhost:9100'] # prometheus 的启动端口,本机的端口号,以及node_exporter labels: instance: prometheus - job_name: demo-server static_configs: - targets: ['192.168.1.101:8080'] # 目标服务器 labels: instance: localhost
-
-
启动docker 文件
- docker run -itd --name prometheus -p 9090:9090 -v ${PWD}/prometheus.yaml:/etc/prometheus/prometheus.yml prom/prometheus
-
访问: http://localhost:9000/graph
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cC89Hd6b-1600522319989)(/Users/joker/Desktop/未闻花名/个人/指标和监控/graph.png)]
-
访问 http://localhost:9000/targets, down 先不用管,那是自定义的
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-c2pPqPBB-1600522319992)(/Users/joker/Desktop/未闻花名/个人/指标和监控/targets.png)]
Grafana
-
拉取grafana : docker pull grafana/grafana
-
启动grafana :
docker run -itd -p 3000:3000 --name=grafana grafana/grafana
-
访问,添加DataSource
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-nuWbZ3hK-1600522319994)(/Users/joker/Desktop/未闻花名/个人/指标和监控/Grafana-addDataSource.png)]
-
主要为URL 即可,要与 prometheus 的一致
-
添加dashboard=> 选择graph => query 更改server 为prometheus server
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1lWemGAy-1600522319996)(/Users/joker/Desktop/未闻花名/个人/指标和监控/graph-添加query条件.png)]
-
添加自定义指标监控
-
编写程序
-
package main import ( "flag" "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp" "log" "net/http" "time" ) var addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.") var requestTotalCount = prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "reqeust_total_count", // 指定名字 Help: "Number of hello requests in total", }, // 设置两个标签 请求方法和 路径 对请求总次数在两个 []string{"method", "endpoint"}, ) var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{ Namespace: "request_duration", Subsystem: "", Name: "request_duration", Help: "", ConstLabels: nil, Buckets: nil, }, []string{"method", "endpoint"}) func init() { // 注册监控指标 prometheus.MustRegister(requestTotalCount) prometheus.MustRegister(requestDuration) } func main() { flag.Parse() http.Handle("/metrics", promhttp.Handler()) http.Handle("/get", Filter(Get)) log.Fatal(http.ListenAndServe(*addr, nil)) } func Get(writer http.ResponseWriter, request *http.Request) { writer.Write([]byte("ok")) } func Filter(h http.HandlerFunc) http.HandlerFunc { return func(writer http.ResponseWriter, request *http.Request) { start := time.Now() h(writer, request) duration := time.Since(start) requestTotalCount.With(prometheus.Labels{"method": request.Method, "endpoint": request.URL.Path}).Inc() requestDuration.With(prometheus.Labels{"method": request.Method, "endpoint": request.URL.Path}).Observe(duration.Seconds()) } }
-
-
在grafana 中添加dashboard
- [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-d37uryWd-1600522319997)(/Users/joker/Desktop/未闻花名/个人/指标和监控/自定义指标.png)]
遇到的问题
- 为什么要使用node-exporter
- 因为内部提供了系统监控的相关信息,就不用自己写了
- 怎么导入 node-exporter的东西
- 需要导入的是json文件即可
- [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rTAXMJ6H-1600522319998)(/Users/joker/Desktop/未闻花名/个人/指标和监控/导入node-exporter 1.png)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-msEsYzqi-1600522319999)(/Users/joker/Desktop/未闻花名/个人/指标和监控/导入node-exporter 2.png)]
- Q: 对于那些埋点的服务,需要创建DataSource 吗
- A: 不需要,因为埋点的服务,都汇集在了Prometheus服务中,不需要特殊操作,只需要通过 这个Prometheus DataSource 创建dashBoard 就可以了