总结：Promethus配置文件

小魏的博客

已于 2023-05-23 12:04:45 修改

阅读量2w

点赞数 7

分类专栏： K8S 文章标签： kubernetes

于 2022-04-07 09:29:31 首次发布

本文链接：https://blog.csdn.net/w2009211777/article/details/124005822

版权

K8S 专栏收录该内容

38 篇文章

订阅专栏

一、介绍

Prometheus 启动的时候，可以加载运行参数 -config.file 指定配置文件，默认为 prometheus.yml。

Prometheus的配置文件是YAML格式。Prometheus的解压包里自带了一个默认的配置文件prometheus.yml。让我们来看一下：

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

在这个缺省的配置文件里定义了4个单元：global、alerting、rule_files和scrape_configs。

在配置文件中我们可以指定 global, alerting, rule_files, scrape_configs, remote_write, remote_read 等属性。

global：配置全局的信息，如抓取监控数据的间隔，抓取业务数据接口的超时时间，告警规则执行周期等

alerting：配置告警发送到的alermanager的地址

rule_files：告警规则文件，数据聚合配置

scrape_configs：配置抓取业务监控数据的相关信息，如url，拉取时间间隔，拉取的超时时间等

remote_write：将数据投递到远程地址，如聚合数据投递到hubble-adapter

remote_read：

下面介绍下每个单元。

二、global

global 属于全局的默认配置，它主要包含 4 个属性，

scrape_interval: 拉取 targets 的默认时间间隔，即拉取业务监控数据的间隔时间。
scrape_timeout: 拉取一个 target 的超时时间，即拉取业务监控数据接口的超时时间。
evaluation_interval: 执行 rules 的时间间隔。即多久遍历一次告警规则列表，判断每个规则是否触发告警。和rule_files的加载没关系
external_labels: 额外的属性，会添加到拉取的数据并存到数据库中。

配置文件结构大概为：

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  scrape_timeout: 10s # is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

三、alerting

通常我们可以使用运行参数 -alertmanager.xxx 来配置 Alertmanager，但是这样不够灵活，没有办法做到动态更新加载，以及动态定义告警属性。

所以 alerting 配置主要用来解决这个问题，它能够更好的管理 Alertmanager, 主要包含 2 个参数：alert_relabel_configs 和 alertmanagers

1、alertmanagers

用于动态发现 Alertmanager 的地址。

如下配置：alertmanager.prom-alert.svc:9093，通过K8S自动发现机制找到本集群内的alertmanager的地址并将告警发送过去。

目前，我们所有prometheus集群都是部署在徐水（另外有个武汉不用管）的K8S集群，Alertmanager也部署在这个集群，所以用自动发现的方式配置alertmanager地址即可。

如下，系统中指定了Alertmanager路径，因为最终需要投递告警到这个服务，如下图：是通过静态配置

另外，qpaas上是通过服务发现配置的alertmanager：

2、alert_relabel_configs

作用：在告警发生时，动态修改标签内容，一般作用是在告警产生时修改标签，如保留哪些标签（labelkeep），删除哪些标签（labeldrop）。具体的有哪些属性，请参考：Configuration | Prometheus

下面着重说明两个属性：action和regex

action

基于正则表达式匹配执行的操作。包括移除标签，保留标签等，具体可参考：Configuration | Prometheus

action枚举：

replace: 将正则表达式与串联的source_labels匹配。然后，将target_label设置为replace，用替换中的匹配组引用(${1}, ${2}, ...)替换为其值。 如果正则表达式不匹配，则不会进行替换。
keep: 删除其正则表达式与串联的source_labels不匹配的目标。
drop: 删除其正则表达式与串联的source_labels匹配的目标。
hashmod: 将target_label设置为串联的source_labels的哈希的模数。
labelmap: 将正则表达式与所有标签名称匹配。 然后，将匹配标签的值复制到通过替换为它们的值替换的匹配组引用(${1}, ${2}, ...)给出的标签名称。
labeldrop: 将正则表达式与所有标签名称匹配。 任何匹配的标签将从标签集中删除。
labelkeep: 将正则表达式与所有标签名称匹配。 任何不匹配的标签将从标签集中删除。

regex

作用是匹配标签的正则表达式。

案例

下面的案例中，action为labeldrop，就是字面意思，需要移除key为prometheus_replica的标签。

prometheus_replica是自定义的标签，告警的时候就会带上这个标签，由于我们prometheus有两个节点pod0和pod1，但是告警产生的时候我们只需要报出来一条就行了，因此把pod的标签去掉后，两个节点产生的告警就完全一样了，就能控制只产生一条。

source_labels

__开头的是保留label

source_labels也可以定义自定义的label

以下为QKE上的案例：下面写错了一句话，应该是将两个标签合并替换为node，而不是Node

四、rule_files

1、介绍

作用：获取所有规则文件中的规则，包括记录规则(recording rules)与告警规则(alerting rule)。

2、记录规则(recording rules)

record规则其实就是数据处理的规则，如数据聚合，如下：

我们可以单独定义数据聚合规则文件，也可以和告警规则文件放一起，但是一般分开放好理解

注意：一定是规则文件，不包含配置文件，如果指定的文件中包含配置文件内容，则会报错。

按照配置的目录，找了下rancher上对应武汉集群的prometheus项目下的prometheus服务，进入控制台：

我们的配置文件是这样的：

rule_files:
- /etc/prometheus/rules/*rules.yaml

所以进入此目录下发现只有一个alert-rules.yaml，恰好就是我们的告警的配置。

3、rule_files的修改

关于rule_files的修改途径：

1、rule_files文件可以在rancher上修改；

2、通过Prometheus Operator提供的CRD修改。Prometheus Operator会去创建Prometheus、PodMonitor、ServiceMonitor、AlertManager以及PrometheusRule这5个CRD资源对象，所以，可以直接调用K8S的API去修改PrometheusRule，从而达到修改rule以及其他配置的效果，如增加record配置等。

以下是PrometheusRule案例（Prometheus Operator）：

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    prometheus: k8s
    role: alert-rules
  name:  mimir-record-rule
  namespace: cnhb4-public-prod04-hubble
spec:
  groups:
  - name: mimir_api_1
    rules:
    - expr: histogram_quantile(0.99, sum(rate(cortex_request_duration_seconds_bucket[1m]))
        by (le, cluster, job))
      record: cluster_job:cortex_request_duration_seconds:99quantile
    - expr: histogram_quantile(0.50, sum(rate(cortex_request_duration_seconds_bucket[1m]))
        by (le, cluster, job))
      record: cluster_job:cortex_request_duration_seconds:50quantile
  - name: mimir_api_2
    rules:
    - expr: histogram_quantile(0.99, sum(rate(cortex_request_duration_seconds_bucket[1m]))
        by (le, cluster, job, route))
      record: cluster_job_route:cortex_request_duration_seconds:99quantile

以下是ServiceMonitor案例（Prometheus Operator）：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    labels:
        cloud.xxx.domain/project-enName: hubble
        cloud.xxx.domain/project-id: "109772"
    name: mimir-alertmanager-sm
    namespace: iks-ns-hubble-mimir
spec:
    endpoints:
        - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
          path: /metrics
          port: http-metrics
          relabelings:
            - targetLabel: cluster
              replacement: cnhb4-public-prod04
            - targetLabel: job
              replacement: iks-ns-hubble-mimir/alertmanager
    namespaceSelector:
        matchNames:
            - iks-ns-hubble-mimir
    selector:
        matchLabels:
            storage: hubble-mimir
            app: alertmanager

4、rule_files的加载

rule_files文件经常会配置多个规则文件，prometheus是同时去加载这些文件，当某个文件中有错误的配置时，则本次加载所有的规则都会失败。

所以，多个规则文件会相互影响，一旦有配置错误，则会导致其它的规则文件加载失败。

但是，prometheus加载失败并不会影响之前的已经加载好的规则。

五、scrape_configs

参考：Configuration | Prometheus

scrape_configs 主要用于配置拉取数据节点，每一个拉取配置主要包含以下参数：

job_name：任务名称
honor_labels：用于解决拉取数据标签有冲突，当设置为 true, 以拉取数据为准，否则以服务配置为准
params：数据拉取访问时带的请求参数
scrape_interval：拉取时间间隔
scrape_timeout: 拉取超时时间
metrics_path：拉取节点的 metric 路径
static_configs：配置访问路径前缀，如ip+port，或者域名地址，或者通过服务发现，类似alertmanager.prom-alert.svc:9093
scheme：拉取数据访问协议，如http
sample_limit：存储的数据标签个数限制，如果超过限制，该数据将被忽略，不入存储；默认值为0，表示没有限制
relabel_configs：拉取数据重置标签配置
metric_relabel_configs：metric 重置标签配置

六、remote_write

remote_write 主要用于可写远程存储配置，主要包含以下参数：

url: 访问地址
remote_timeout: 请求超时时间
write_relabel_configs: 标签重置配置, 拉取到的数据，经过重置处理后，发送给远程存储

案例：

remote_write:
- url: http://xxx:9988/prom2hubble/push?group=xxx
  remote_timeout: 30s
  write_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: obser:(.*)
    replacement: $1
    action: keep
  - separator: ;
    regex: (.*)
    target_label: hubble_endpoint
    replacement: hubble_qpaas_obser
    action: replace
  - separator: ;
    regex: (.*)
    target_label: group
    replacement: hubble
    action: replace
  - separator: ;
    regex: (.*)
    target_label: hubble_step
    replacement: "60"
    action: replace
  - separator: ;
    regex: label_qke_cloud_qiyi_domain_(.*)
    replacement: $1
    action: labelmap
  - separator: ;
    regex: (job|endpoint|service|pod|instance|namespace|prometheus.*|label_qke_cloud_qiyi_domain_.*)
    replacement: $1
    action: labeldrop
  queue_config:
    capacity: 2500
    max_shards: 200
    min_shards: 1
    max_samples_per_send: 500
    batch_send_deadline: 5s
    min_backoff: 30ms
    max_backoff: 100ms
  metadata_config:
    send: true
    send_interval: 1m

七、remote_read

remote_read 主要用于可读远程存储配置，主要包含以下参数：

url: 访问地址
remote_timeout: 请求超时时间

八、服务发现

1、介绍

ServiceDiscoveryConfig 主要用于 target 发现，大体分为两类，静态配置和动态发现。

在 Prometheus 的配置中，一个最重要的概念就是数据源 target，而数据源的配置主要分为静态配置和动态发现, 大致为以下几类：

static_configs: 静态服务发现
eureka_sd_config：eureka服务发现，发现真实的实例节点的ip+port，参考：Configuration | Prometheus
- 可参考案例：prometheus/prometheus-eureka.yml at release-2.36 · prometheus/prometheus · GitHub
dns_sd_configs: DNS 服务发现
file_sd_configs: 文件服务发现
consul_sd_configs: Consul 服务发现
serverset_sd_configs: Serverset 服务发现
nerve_sd_configs: Nerve 服务发现
marathon_sd_configs: Marathon 服务发现
kubernetes_sd_configs: Kubernetes 服务发现
gce_sd_configs: GCE 服务发现
ec2_sd_configs: EC2 服务发现
openstack_sd_configs: OpenStack 服务发现
azure_sd_configs: Azure 服务发现
triton_sd_configs: Triton 服务发现

它们具体使用以及配置模板，请参考服务发现配置模板。

它们中最重要的，也是使用最广泛的应该是 static_configs, 其实那些动态类型都可以看成是某些通用业务使用静态服务封装的结果。

2、eureka_sd_configs案例介绍

代码地址：

prometheus/prometheus-eureka.yml at release-2.36 · prometheus/prometheus · GitHub

3、kubernetes_sd_configs

我们的微服务本质上是采用的kubernetes_sd_configs。

但是我们是通过Prometheus Operator提供的ServiceMonitor间接创建了kubernetes_sd_configs

如下关注点：

1、kubernetes_sd_configs中的api_server：抓取指标的地址前缀

2、metrics_path：抓取指标的具体路径

- job_name: qke-generic-hubble-manager/hubble-alarm-agg-condition-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: m-agg-condition
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager

十、关于relabel_configs

relabel_configs分为<metric_relabel_configs>和<alert_relabel_configs>两类。

指标：

告警：

七、AlertManager配置

案例：

为什么是webhook_configs？不支持remote_write？

global:
resolve_timeout: 10m

route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 24h   #重复报警的时间间隔为24h
receiver: hubble


receivers:
- name: 'hubble'
webhook_configs:
- url: 'http://hubble.adapter.qiyi.domain:9988/prom2hubble/alert'

八、告警规则配置

案例：

主要包含几部分：

groups:
- name: PrometheusRule                 #报警规则组的名字，可以类比为hubble的策略模板
  rules:                               #策略列表
  - expr: up{job="alertmanager"} == 0  #表达式
    alert: alertmanagerInstanceDown    #告警的triggername
    for: 2m                            #2分钟比较一次，和连续几个点类似
    annotations:                       #告警信息必要的信息，labels是告警消息的tag信息
      alertlevel: "P2"
      hubblegroup: "hubble-prometheus-k8s"
      alertvalue: "{{ $value }}"
      summary: "[prometheus-cluster-wh] alertmanager is down"

  - expr: increase(alertmanager_notifications_failed_total{job="alertmanager"}[5m])/increase(alertmanager_notifications_total{job="alertmanager"}[5m]) > 0.3
    alert: alertmanagerSendOutFail
    for: 5m
    annotations:
      alertlevel: "P2"
      hubblegroup: "hubble-prometheus-k8s"
      alertvalue: "{{ $value }}"            #value 就是数据的当前值
      summary: "[prometheus-cluster-wh] failed to sendout alerts >30%"
      description: "应用名： {{ $labels.job }}  实例名： {{ $labels.instance }}  ， 环境： {{ $labels.env }} , 当前值为 ： {{ $value }}"   # labels其实就是数据中的tag，如job，instance等

八、配置文件案例

QKE配置文件案例：

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: 58-hubble/k8s
    prometheus_replica: prometheus-k8s-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - scheme: http
    path_prefix: /
    timeout: 10s
    api_version: v1
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http
      replacement: $1
      action: keep
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - default
rule_files:
- /etc/prometheus/rules/prometheus-k8s-rulefiles-0/*.yaml
scrape_configs:
- job_name: qke-generic-hubble-grafana-dashboard/hubble-grafana-dashboard-servicemonitor/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: hubble-grafana-dashboard
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_app]
    separator: ;
    regex: hubble-grafana-dashboard
    replacement: $1
    action: keep
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: (grafana_stat_.*|grafana_.*_response_status_total|process_.*)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-grafana-dashboard
- job_name: qke-generic-hubble-manager/hubble-api-open-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-api-open
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-aiops-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-biz-aiops
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-cm-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-biz-cm
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-stat-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-biz-stat
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-biz-third-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: m-biz-third
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-task-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-hubble-task
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/hubble-transfer-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-hubble-transfer
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: m-hubble-transfer
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: m-hubble-transfer
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-manager/network-screen-sm/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /prometheus
  scheme: http
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: s-network-screen
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-manager
- job_name: qke-generic-hubble-p-hbs/hubble-p-hbs-svcm/0
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /json2metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: hubble-p-hbs
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: exporter-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: exporter-metrics
    action: replace
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: hubble_p_hbs_(.*)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-p-hbs
- job_name: qke-generic-hubble-p-query/hubble-p-query-svcm/0
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /json2metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: hubble-p-query
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: exporter-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: exporter-metrics
    action: replace
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: hubble_p_query_(.*)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-p-query
- job_name: qke-generic-hubble-p-transfer/hubble-p-transfer-svcm/0
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /json2metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: hubble-p-transfer
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: exporter-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: exporter-metrics
    action: replace
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: hubble_p_transfer_(.*)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-p-transfer
- job_name: qke-generic-hubble-platform/kube-state-metrics/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  bearer_token_file: /var/k8s-auth/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kube-state-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-main
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-main
    action: replace
  - separator: ;
    regex: (pod|service|endpoint|namespace)
    replacement: $1
    action: labeldrop
  metric_relabel_configs:
  - source_labels: [namespace]
    separator: ;
    regex: (qke-generic-hubble-platform|qke-generic-hubble-p-updater-server|qke-generic-hubble-grafana-dashboard|qke-generic-hubble-p-transfer|qke-generic-hubble-grafana-api|qke-generic-hubble-aiops|qke-generic-hubble-p-hbs|qke-generic-hubble-p-query|qke-generic-hubble-manager|qke-generic-hubble-self-monitor)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - monitoring
- job_name: qke-generic-hubble-platform/kubelet/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics/cadvisor
  scheme: https
  bearer_token_file: /var/k8s-auth/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: kubelet
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__metrics_path__]
    separator: ;
    regex: (.*)
    target_label: metrics_path
    replacement: $1
    action: replace
  metric_relabel_configs:
  - source_labels: [namespace]
    separator: ;
    regex: (qke-generic-hubble-platform|qke-generic-hubble-p-updater-server|qke-generic-hubble-grafana-dashboard|qke-generic-hubble-p-transfer|qke-generic-hubble-grafana-api|qke-generic-hubble-aiops|qke-generic-hubble-p-hbs|qke-generic-hubble-p-query|qke-generic-hubble-manager|qke-generic-hubble-self-monitor)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - kube-system
- job_name: qke-generic-hubble-self-monitor/hubble-p-transfer-svcm/0
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /json2metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: hubble-p-transfer
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: exporter-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: exporter-metrics
    action: replace
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: hubble_p_transfer_(.*)
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: endpoints
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-activator/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    separator: ;
    regex: activator
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: qke-generic-hubble-platform/knative-activator
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_knative_activator]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: knative-serving;activator;metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: pod
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - knative-serving
- job_name: qke-generic-hubble-platform/knative-autoscaler/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    separator: ;
    regex: autoscaler
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: qke-generic-hubble-platform/knative-autoscaler
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_knative_autoscaler]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: knative-serving;autoscaler;metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: pod
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - knative-serving
- job_name: qke-generic-hubble-platform/knative-controller/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    separator: ;
    regex: controller
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: qke-generic-hubble-platform/knative-controller
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_knative_controller_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: knative-serving;controller;metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: pod
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - knative-serving
- job_name: qke-generic-hubble-platform/knative-queue-proxy/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_qke_cloud_qiyi_domain_lite]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: qke-generic-hubble-platform/knative-queue-proxy
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_knative_queue_proxy]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: .+;http-usermetric
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: pod
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-platform
      - qke-generic-hubble-p-updater-server
      - qke-generic-hubble-grafana-dashboard
      - qke-generic-hubble-p-transfer
      - qke-generic-hubble-grafana-api
      - qke-generic-hubble-aiops
      - qke-generic-hubble-p-hbs
      - qke-generic-hubble-p-query
      - qke-generic-hubble-manager
      - qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-queue-proxy/1
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_qke_cloud_qiyi_domain_lite]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: qke-generic-hubble-platform/knative-queue-proxy
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_knative_queue_proxy]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_serving_knative_dev_revision, __meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: .+;http-autometric
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: pod
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - qke-generic-hubble-platform
      - qke-generic-hubble-p-updater-server
      - qke-generic-hubble-grafana-dashboard
      - qke-generic-hubble-p-transfer
      - qke-generic-hubble-grafana-api
      - qke-generic-hubble-aiops
      - qke-generic-hubble-p-hbs
      - qke-generic-hubble-p-query
      - qke-generic-hubble-manager
      - qke-generic-hubble-self-monitor
- job_name: qke-generic-hubble-platform/knative-webhook/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    separator: ;
    regex: webhook
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: job
    replacement: qke-generic-hubble-platform/knative-webhook
    action: replace
  - source_labels: [__meta_kubernetes_pod_label_knative_webhook]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
    separator: ;
    regex: knative-serving;webhook;metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  kubernetes_sd_configs:
  - api_server: https://kube-master-bjzyx-public-staging02.cloud.qiyi.domain:6443
    role: pod
    bearer_token_file: /var/k8s-auth/token
    tls_config:
      ca_file: /var/k8s-auth/ca.crt
      insecure_skip_verify: true
    namespaces:
      names:
      - knative-serving

全链路配置案例：

global:
evaluation_interval: 60s
external_labels:
prometheus_replica: $(POD_NAME)
scrape_interval: 60s
alerting:
alert_relabel_configs:
- action: labeldrop
regex: prometheus_replica
alertmanagers:
- scheme: http
static_configs:
- targets:
- alertmanager.prom-alert.svc:9093
rule_files:
- /etc/prometheus/rules/*rules.yaml
scrape_configs:
- job_name: 48e5a419-4721-5921-899c-86aa7122dfb6-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5eef17ab-f6b6-5d79-942b-ac41d35ba870-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-notice
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5f476f71-29c9-527e-b765-58b30c425751-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-bq-alarm-event
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 79f80b60-0ced-55b5-a439-504b15a620ce-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-alarm-storm
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 441b1a98-6c0c-5d8c-970d-425d2c1d412e-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-transfer
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 5b5d4de9-2c2e-579b-8854-8b4942b09e5e-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-bq-alarm-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: dfbfbb26-9632-5200-9b91-7e989c43969d-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-alarm-query
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: f6dc47d3-7e74-5ab4-8502-4b2fe9cb8123-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-biz-cm
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
- job_name: 24e69042-7557-5860-913e-9c7eeab76660-nacos-adapter
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 30s
metrics_path: /prometheus
consul_sd_configs:
- server: http://laputa.prometheus-nacos-adapter.online.qiyi.qae
services:
- 21c7187d-c748-4fc9-916e-0c270f0509ee@@hubble@@hubble-network-screen
relabel_configs:
- source_labels: [__meta_consul_service_id]
target_label: instance
- regex: __meta_consul_service_metadata_(.+)
action: labelmap
remote_write:
- url: http://hubble.adapter.qiyi.domain:9988/prom2hubble/push?group=hubble-test
write_relabel_configs:
- source_labels: [__name__]
regex: trace_(.*)
action: keep
- source_labels: [project, app]
separator: ':'
target_label: hubble_endpoint
replacement: prometheus_$1
- source_labels: [project]
target_label: hubble_group
- regex: (app|project|prometheus_replica)
action: labeldrop
- target_label: hubble_step
replacement: "60"
name: trace

九、告警规则案例

QKE告警规则案例：

全链路告警规则案例：

	groups:
- name: qytrace-agg.rules
rules:
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod"}[1m])) by (project, app, span, zone, status_code)
record: trace_span_requests_zone_code
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod"}[1m])) by (project, app, zone, status_code)
record: trace_service_requests_zone_code
- expr: |
avg by (project, app, zone, status_code) (sum by(instance, project, app, zone, status_code) (irate(http_server_requests_duration_seconds_count{env="prod"}[1m])))
record: trace_service_avg_requests_zone_code
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, span, zone)
record: trace_span_success_requests_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, zone)
record: trace_service_success_requests_zone


- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, span, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, span, zone)
record: trace_span_success_rate_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (project, app, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, zone)
record: trace_service_success_rate_zone

- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (project, app, span, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, span, zone)
record: trace_span_avg_latency_zone
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (project, app, zone)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (project, app, zone)
record: trace_service_avg_latency_zone

- expr: |
sum(trace_span_requests_zone_code) by (project, app, span, zone)
record: trace_span_requests_zone
- expr: |
sum(trace_service_requests_zone_code) by (project, app, zone)
record: trace_service_requests_zone

- expr: sum(jvm_gc_pause_seconds_count{env="prod"} - jvm_gc_pause_seconds_count{env="prod"} offset 1m) by (project, app, instance, zone)
record: trace_service_jvm_gc_cnt_zone
- expr: sum(jvm_gc_pause_seconds_sum{env="prod"} - jvm_gc_pause_seconds_sum{env="prod"} offset 1m) by (project, app, instance, zone)
record: trace_service_jvm_gc_elapsed_zone

- expr: |
sum(trace_service_requests_zone_code) by (app)
record: trace_service_requests_app
- expr: |
sum(irate(http_server_requests_duration_seconds_count{env="prod",success="true"}[1m])) by (app)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (app)
record: trace_service_success_rate_app
- expr: |
sum(irate(http_server_requests_duration_seconds_sum{env="prod",success="true"}[1m])) by (app)
/ sum(irate(http_server_requests_duration_seconds_count{env="prod",}[1m])) by (app)
record: trace_service_avg_latency_app	 	
samples-scraped-rules.yaml		groups:
- name: samples-monitoring
rules:
- alert: SamplesScrapedTotal
expr: sum(scrape_samples_scraped{}) > 3000000
for: 2m
labels:
prometheus: hubble-prod
annotations:
alertlevel: "P3"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{$value}}"
summary: "total samples in hubble-prod > 300w"
- alert: SamplesScrapedByJob
expr: sum by (job) (scrape_samples_scraped{}) > 1000000
for: 2m
labels:
prometheus: hubble-prod
annotations:
alertlevel: "P3"
hubblegroup: "hubble-prometheus-k8s"
alertvalue: "{{$value}}"
summary: "samples from {{$labels.job}} in hubble-prod > 100w"

参考：

Prometheus配置文件

Configuration | Prometheus