prometheus relabel_config 详解加示例

relabel_config配置

Relabeling(重定义标签),是在拉取(scraping)阶段前,修改target和它的labels;
在每个scrape_configs可以定义多个重定义标签的步骤;

默认的, target的job标签设置为配置文件里的job_name的值;
__address__设置为配置里的targets的值;
而instance标签的值,是重定义标签操作之后__address__的值
__scheme__和__metrics_path__标签的值,也是从配置里取值的;
__param_<name>标签的值,是传给URL的查询参数<name>的值;

以__meta_开头的标签,也可以用于重定义标签;

重定义标签完成后,__开头的标签会被删除;

重定义标签阶段,如果要临时存储值用于下一阶段的处理,使用__tmp开头的标签名,这种标签不会被Prometheus使用;

# 从已有的标签选择值的源标签组;多个标签,使用separator分隔;
# regex匹配源标签里的值
# 动作为replace, keep, 和 drop
[ source_labels: '[' <labelname> [, ...] ']' ]
# 多个源标签的分隔符;
[ separator: <string> | default = ; ]
# replace动作必需,regex匹配到的值,要替换的目标标签;
[ target_label: <labelname> ]
# 正则匹配源标签的值
[ regex: <regex> | default = (.*) ]
# 源标签值取hash的模块;
[ modulus: <uint64> ]
# 要替换的匹配分组号
[ replacement: <string> | default = $1 ]
# 基于正则匹配的动作
[ action: <relabel_action> | default = replace ]

正则匹配是到结尾的, 可以使用.*<regex>.*匹配更多内容

relabel_action

replace: 正则匹配源标签的值用来替换目标标签;如果有replacement,使用replacement替换目标标签;
keep: 如果正则没有匹配到源标签,删除targets 
drop: 正则匹配到源标签,删除targets
hashmod: 设置目标标签值为源标签值的hash值
labelmap: 正则匹配所有标签名; 将匹配的标签的值复制到由replacement提供的标签名
labeldrop: 正则匹配所有标签名;匹配则移除标签;
labelkeep: 正则匹配所有标签名;不匹配的标签会被移除;

示例环境

使用默认的prometheus target和监控指标

默认配置文件

cat prometheus.yml
------
global:
  scrape_interval:     15s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets:
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
======

查看target

curl 'http://localhost:9090/api/v1/targets?state=active'
------
{
	"status": "success",
	"data": {
		"activeTargets": [{
			"discoveredLabels": { // 这块只能作为源标签
				"__address__": "localhost:9090",
				"__metrics_path__": "/metrics",
				"__scheme__": "http",
				"job": "prometheus"
			},
			"labels": {
				"instance": "localhost:9090",
				"job": "prometheus"
			},
			"scrapePool": "prometheus",
			"scrapeUrl": "http://localhost:9090/metrics",
			"lastError": "",
			"lastScrape": "2020-04-25T08:42:19.254191755-04:00",
			"lastScrapeDuration": 0.012091634,
			"health": "up"
		}],
		"droppedTargets": []
	}
}
======

relabel_config主要是处理target的labels字段

重载配置命令

# 检查配置
./promtool check config prometheus.yml
# 重载配置;需要以--web.enable-lifecycle启动promethues
curl -X POST http://localhost:9090/-/reload
# 查看使用的配置 
curl http://localhost:9090/api/v1/status/config

添加自定义标签

添加2个自定义标签,用于后续的演示

vi prometheus.yml
------
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels: 
          userLabel1: value1
          userLabel2: value2
======

重载配置后查询

curl 'http://localhost:9090/api/v1/targets?state=active'
------
{
	"status": "success",
	"data": {
		"activeTargets": [{
			"discoveredLabels": {
				"__address__": "localhost:9090",
				"__metrics_path__": "/metrics",
				"__scheme__": "http",
				"job": "prometheus",
				"userLabel1": "value1", //新增
				"userLabel2": "value2"  //新增
			},
			"labels": {
				"instance": "localhost:9090",
				"job": "prometheus",
				"userLabel1": "value1",  //新增
				"userLabel2": "value2"  //新增
			},
            ...
		}],
		"droppedTargets": []
	}
}

替换标签的值

用源标签的值,替换目标标签的值

vi prometheus.yml
------
scrape_configs:
    ...
    relabel_configs:
    - source_labels: [userLabel1] 
      target_label:  userLabel2
      #默认action 是 'replace'
======

重载配置后查询

curl 'http://localhost:9090/api/v1/targets?state=active'
------
			"discoveredLabels": {
				"__address__": "localhost:9090",
				"__metrics_path__": "/metrics",
				"__scheme__": "http",
				"job": "prometheus",
				"userLabel1": "value1", //不变
				"userLabel2": "value2"  //不变
			},
			"labels": {
				"instance": "localhost:9090",
				"job": "prometheus",
				"userLabel1": "value1",
				"userLabel2": "value1"  //用userLabel1的值替换了userLabel2
			},
======

relabel_configs是在拉取(scraping)前,修改target和它的labels

用userLabel1的部分值替换userLabel2

vi prometheus.yml
------
scrape_configs:
    ...
    relabel_configs:
    - source_labels: [userLabel1]
      regex: 'value([0-9]+)'
      target_label:  userLabel2
      replacement: '$1'
      action: replace
======

重载配置后查询

curl 'http://localhost:9090/api/v1/targets?state=active'
------
			"labels": {
				"instance": "localhost:9090",
				"job": "prometheus",
				"userLabel1": "value1",
				"userLabel2": "1"
			},
======

删除匹配的标签

vi prometheus.yml
------
scrape_configs:
    ...
    relabel_configs:
    - regex: userLabel1
      action: labeldrop
======      

重载配置后查询

curl 'http://localhost:9090/api/v1/targets?state=active'
------
{
	"status": "success",
	"data": {
		"activeTargets": [{
			"discoveredLabels": { //这部分只能做为源标签
				"__address__": "localhost:9090",
				"__metrics_path__": "/metrics",
				"__scheme__": "http",
				"job": "prometheus",
				"userLabel1": "value1",
				"userLabel2": "value2"
			},
			"labels": {
				"instance": "localhost:9090",
				"job": "prometheus",
				"userLabel2": "value2" //删除了userLabel1
			},
            ...
	}
}
======

删除匹配的target

vi prometheus.yml
------
scrape_configs:
    ...
    relabel_configs:
    - source_labels: [userLabel1] 
      action: drop
======      

重载配置后查询

curl 'http://localhost:9090/api/v1/targets?state=active'
------
{
	"status": "success",
	"data": {
		"activeTargets": [],
		"droppedTargets": []
	}
}
======

取匹配的标签名的一部分生成新标签

vi prometheus.yml
------
scrape_configs:
    ...
    relabel_configs:
    - regex: user(.*)1
      action: labelmap
======      

重载配置后查询

curl 'http://localhost:9090/api/v1/targets?state=active'
------
			"labels": {
				"Label": "value1", //新生成的标签
				"instance": "localhost:9090",
				"job": "prometheus",
				"userLabel1": "value1",
				"userLabel2": "value2"
			},
======

参考链接
官方配置示例
官方文档