Go语言开发一个完整的exporter
Prometheus 官方和社区提供了非常多的 exporter,涵盖数据库、中间件、OS、存储、硬件设备等,具体可查
看:
https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exporters.md
https://exporterhub.io/
通过这些 exporter 基本可以覆盖80%的监控需求,然后很多场景下需要自己定义监控指标,那么我们就需要自己
开发 exporter ,本文我们将通过获取CPU和内存的监控细信息来编写一个完整的expoter。
1、项目结构
# tree
my_go_exporter/
├── collector
│ └── collector.go
├── data
│ ├── cpu_data.go
│ └── memory_data.go
├── global
│ └── global.go
├── go.mod
├── go.sum
├── handle
│ ├── cpu_handle.go
│ └── memory_handle.go
├── main.go
└── scraper
├── cpu_scraper.go
├── memory_scraper.go
└── scrape.go
data:获取 CPU 和 Memory 信息。
global:主要是存放一些全局使用的公共属性。
handler:定义一个 httphandler 来处理http请求。
scrape:scrape 在 prometheus 里的功能是数据采集器,在这里主要是全局 scrape 的 interface 和其具体实现。
collector:collector 文件在本文这里类似数据注册器,所有的 scrape 需要被注册到 collector 里才能真正的被调
用,将数据暴露出去。
main.go:实现 web 接口,供外部访问。
1、global全局属性定义
global.go
package global
const (
Namespace = "my_exporter"
Subsystem = "sub_exporter"
)
2、cpu和memory数据获取
这里只是随机生成一些结果,并没有真正的去获取这些数据。
cpu_data.go
package data
import "math/rand"
// 这里模拟的是假数据
// CPU使用率
func GetCpuData(str string) float64 {
cpuData := map[string]float64{
"192.168.164.100:7547": rand.Float64() * 100,
"192.168.164.101:7547": rand.Float64() * 100,
}
return cpuData[str]
}
memory_data.go
package data
import "math/rand"
// 这里模拟的是假数据
// Memory使用率
func GetMemoryData(str string) float64 {
memoryData := map[string]float64{
"192.168.164.100:8547": rand.Float64() * 100,
"192.168.164.101:8547": rand.Float64() * 100,
}
return memoryData[str]
}
3、scrape定义指标信息
scrape.go
package scraper
import "github.com/prometheus/client_golang/prometheus"
// 所有指标收集需要满足Name和Scrape两个方法
// Name返回指标名字来用于用户开关指标
type Scraper interface {
// Scraper的名字,必须是唯一的
Name() string
// Scraper的描述信息,该方法可选
Help() string
// Scrape从客户端收集数据,并将其作为prometheus metric通过通道发送
Scrape(ch chan<- prometheus.Metric) error
}
Scrape 的参数没什么要说的,这里可以根据不同的监控目的添加不同的链接驱动
如果是 mysqld_exporter 的话需要换成
Scrape(db *sql.DB, ch chan<- prometheus.Metric) error
如果是 redis_exporter 的话换成
Scrape(c *redis.Conn, ch chan<- prometheus.Metric) error
cpu_scraper.go
package scraper
import (
"exporter/data"
"exporter/global"
"fmt"
"github.com/prometheus/client_golang/prometheus"
)
// 实现Scraper接口
type CpuScraper struct {
CSName string
CSHelp string
Ip string
Port string
}
func (cpuScraper *CpuScraper) Name() string {
return cpuScraper.CSName
}
func (cpuScraper *CpuScraper) Help() string {
return cpuScraper.CSHelp
}
func (cpuScraper *CpuScraper) Scrape(ch chan<- prometheus.Metric) error {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(
prometheus.BuildFQName(global.Namespace, global.Subsystem, cpuScraper.CSName),
// 动态标签的键,可以包含多个
cpuScraper.CSHelp, []string{"user1", "user2"},
prometheus.Labels{"ip": cpuScraper.Ip, "port": cpuScraper.Port},
),
prometheus.GaugeValue,
// 指标值
data.GetCpuData(fmt.Sprintf("%s:%s", cpuScraper.Ip, cpuScraper.Port)),
// 动态标签的值,与上面动态标签的键相互对应,可以包含多个,多个用逗号分割
"root1", "root2",
)
return nil
}
memory_scraper.go
package scraper
import (
"exporter/data"
"exporter/global"
"fmt"
"github.com/prometheus/client_golang/prometheus"
)
// 实现Scraper接口
type MemoryScraper struct {
MSName string
MSHelp string
Ip string
Port string
}
func (memoryScraper *MemoryScraper) Name() string {
return memoryScraper.MSName
}
func (memoryScraper *MemoryScraper) Help() string {
return memoryScraper.MSHelp
}
func (memoryScraper *MemoryScraper) Scrape(ch chan<- prometheus.Metric) error {
ch <- prometheus.MustNewConstMetric(
prometheus.NewDesc(
prometheus.BuildFQName(global.Namespace, global.Subsystem, memoryScraper.MSName),
// 动态标签的键,可以包含多个
memoryScraper.MSHelp, []string{"user3", "user4"},
prometheus.Labels{"ip": memoryScraper.Ip, "port": memoryScraper.Port},
),
prometheus.GaugeValue,
// 指标值
data.GetMemoryData(fmt.Sprintf("%s:%s", memoryScraper.Ip, memoryScraper.Port)),
// 动态标签的值,与上面动态标签的键相互对应,可以包含多个,多个用逗号分割
"root4", "root4",
)
return nil
}
NewDesc 参数第一个为指标的名称,第二个为帮助信息,显示在指标的上面作为注释,第三个是定义的 label 名
称,为数组,第四个是定义的 Labels。
4、collector抓取数据并返回数据
collector.go
package collector
import (
"exporter/global"
"exporter/scraper"
"github.com/prometheus/client_golang/prometheus"
log "github.com/sirupsen/logrus"
"sync"
)
// 这里额外可以添加一些指标,用来监控exporter
type Metrics struct {
// 收集指标的总次数
TotalScrapes prometheus.Counter
// 收集指标中发生错误的次数
ScrapeErrors *prometheus.CounterVec
// 最后一次是否发生了错误
Error prometheus.Gauge
}
type Exporter struct {
metrics Metrics
scrapers []scraper.Scraper
}
// 添加描述符
func (e *Exporter) Describe(ch chan<- *prometheus.Desc) {
ch <- e.metrics.TotalScrapes.Desc()
ch <- e.metrics.Error.Desc()
e.metrics.ScrapeErrors.Describe(ch)
}
// 收集指标
func (e *Exporter) Collect(ch chan<- prometheus.Metric) {
e.scrape(ch)
ch <- e.metrics.TotalScrapes
ch <- e.metrics.Error
e.metrics.ScrapeErrors.Collect(ch)
}
//判断*exporter是否实现了collector这个接口的所有方法
var _ prometheus.Collector = (*Exporter)(nil)
// 开启线程收集指标
func (e *Exporter) scrape(ch chan<- prometheus.Metric) {
var (
wg sync.WaitGroup
err error
)
defer wg.Wait()
for _, sc := range e.scrapers {
wg.Add(1)
// 使用匿名函数并且并发的收集指标
go func(sc scraper.Scraper) {
defer wg.Done()
label := sc.Name()
err = sc.Scrape(ch)
if err != nil {
log.WithField("scraper", sc.Name()).Error(err)
e.metrics.ScrapeErrors.WithLabelValues(label).Inc()
e.metrics.Error.Set(1)
}
// 请求次数加1次
e.metrics.TotalScrapes.Inc()
}(sc)
}
}
// 返回一个collector
func NewCollector(metrics Metrics, scrapers []scraper.Scraper) *Exporter {
return &Exporter{
metrics: metrics,
scrapers: scrapers,
}
}
// 返回一个Metrics
func NewMetrics() Metrics {
return Metrics{
TotalScrapes: prometheus.NewCounter(prometheus.CounterOpts{
Namespace: global.Namespace,
Subsystem: global.Subsystem,
Name: "scrapes_total",
Help: "Total number of times was scraped for metrics.",
}),
ScrapeErrors: prometheus.NewCounterVec(prometheus.CounterOpts{
Namespace: global.Namespace,
Subsystem: global.Subsystem,
Name: "scrape_errors_total",
Help: "Total number of times an error occurred scraping .",
}, []string{"collector"}),
Error: prometheus.NewGauge(prometheus.GaugeOpts{
Namespace: global.Namespace,
Subsystem: global.Subsystem,
Name: "last_scrape_error",
Help: "Whether the last scrape of metrics resulted in an error (1 for error, 0 for success).",
}),
}
}
5、handler注册collector
cpu_handle.go
package handle
import (
"exporter/collector"
"exporter/scraper"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
// 这里可以添加多台机器
var cpuScrapers = []scraper.Scraper{
&scraper.CpuScraper{CSName: "cpu_usage_100", CSHelp: "cpu_usage_100_help", Ip: "192.168.164.100", Port: "7547"},
&scraper.CpuScraper{CSName: "cpu_usage_101", CSHelp: "cpu_usage_101_help", Ip: "192.168.164.101", Port: "7547"},
}
func NewCpuHandle() http.Handler {
// 创建一个自定义的注册表
registry := prometheus.NewRegistry()
registry.MustRegister(collector.NewCollector(collector.NewMetrics(), cpuScrapers))
return promhttp.HandlerFor(registry, promhttp.HandlerOpts{Registry: registry})
}
memory_handle.go
package handle
import (
"exporter/collector"
"exporter/scraper"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
// 这里可以添加多台机器
var memoryScrapers = []scraper.Scraper{
&scraper.MemoryScraper{MSName: "memory_usage_100", MSHelp: "memory_usage_100_help", Ip: "192.168.164.100", Port: "8547"},
&scraper.MemoryScraper{MSName: "memory_usage_101", MSHelp: "memory_usage_101_help", Ip: "192.168.164.101", Port: "8547"},
}
func NewMemoryHandle() http.Handler {
// 创建一个自定义的注册表
registry := prometheus.NewRegistry()
registry.MustRegister(collector.NewCollector(collector.NewMetrics(), memoryScrapers))
return promhttp.HandlerFor(registry, promhttp.HandlerOpts{Registry: registry})
}
6、main
main.go
package main
import (
"exporter/handle"
log "github.com/sirupsen/logrus"
"net/http"
"os"
)
func main() {
http.Handle("/metrics/cpu", handle.NewCpuHandle())
http.Handle("/metrics/memory", handle.NewMemoryHandle())
log.Infoln("Start server at :9090")
if err := http.ListenAndServe(":9090", nil); err != nil {
log.Errorf("Error occur when start server %v", err)
os.Exit(1)
}
}
7、启动访问
http://localhost:9090/metrics/cpu
# HELP my_exporter_sub_exporter_cpu_usage_100 cpu_usage_100_help
# TYPE my_exporter_sub_exporter_cpu_usage_100 gauge
my_exporter_sub_exporter_cpu_usage_100{ip="192.168.164.100",port="7547",user1="root1",user2="root2"} 51.52126285020654
# HELP my_exporter_sub_exporter_cpu_usage_101 cpu_usage_101_help
# TYPE my_exporter_sub_exporter_cpu_usage_101 gauge
my_exporter_sub_exporter_cpu_usage_101{ip="192.168.164.101",port="7547",user1="root1",user2="root2"} 30.091186058528706
# HELP my_exporter_sub_exporter_last_scrape_error Whether the last scrape of metrics resulted in an error (1 for error, 0 for success).
# TYPE my_exporter_sub_exporter_last_scrape_error gauge
my_exporter_sub_exporter_last_scrape_error 0
# HELP my_exporter_sub_exporter_scrapes_total Total number of times was scraped for metrics.
# TYPE my_exporter_sub_exporter_scrapes_total counter
my_exporter_sub_exporter_scrapes_total 6
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
http://localhost:9090/metrics/memory
# HELP my_exporter_sub_exporter_last_scrape_error Whether the last scrape of metrics resulted in an error (1 for error, 0 for success).
# TYPE my_exporter_sub_exporter_last_scrape_error gauge
my_exporter_sub_exporter_last_scrape_error 0
# HELP my_exporter_sub_exporter_memory_usage_100 memory_usage_100_help
# TYPE my_exporter_sub_exporter_memory_usage_100 gauge
my_exporter_sub_exporter_memory_usage_100{ip="192.168.164.100",port="8547",user3="root4",user4="root4"} 66.45600532184905
# HELP my_exporter_sub_exporter_memory_usage_101 memory_usage_101_help
# TYPE my_exporter_sub_exporter_memory_usage_101 gauge
my_exporter_sub_exporter_memory_usage_101{ip="192.168.164.101",port="8547",user3="root4",user4="root4"} 43.771418718698015
# HELP my_exporter_sub_exporter_scrapes_total Total number of times was scraped for metrics.
# TYPE my_exporter_sub_exporter_scrapes_total counter
my_exporter_sub_exporter_scrapes_total 74
# HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
# TYPE promhttp_metric_handler_errors_total counter
promhttp_metric_handler_errors_total{cause="encoding"} 0
promhttp_metric_handler_errors_total{cause="gathering"} 0
至此,exporter 开发完成!