Prometheus-Operater修改Prometheus.yaml配置文件

#背景#

在多个原生的k8s集群里面安装部署Prometheus-operater后，需要将Prometheus的数据发送至主Prometheus统一管理。发现使用Prometheus-Operater安装的组件受k8s的CRD影响，无法直接修改Prometheus的配置文件。

一、理解CRD(以下结果来自AI)

在 Kubernetes（K8s）中，CRD（Custom Resource Definition）是一种强大的扩展机制，允许用户自定义新的资源类型。这就是说，除了 Kubernetes 自带的资源类型（如 Pod、Deployment、ReplicaSet 等），用户还可以通过 CRD 创建自己定义的资源类型。

CRD 的全称为 CustomResourceDefinitions，即自定义资源。你可以把 CRD 理解成两部分，Customer Resource 和 Customer Controller。Customer Resource 可以让用户简单地存储和获取结构化数据，将这种对象动态地注册到 Kubernetes 集群中。

如果你的资源是个 CRD，因为没有对应的控制器，你就得为它自己写 Controller。Controller 是「控制器」，控制 Kubernetes 的资源实体。它通过监听资源变化事件来控制。这个事件可能是用户发起的（他希望把资源从 A 状态更新到 B 状态），Controller 就会获取这个事件并处理事件，即更新目标资源。

  分析：也就是说，在部署整个Prometheus-Operater过程中，Prometheus的相关配置全部都通过自定义资源来控制，包括自定kind类型：

[root@kubemaster01 ~]# kubectl  -n monitoring get   crd
NAME                                          CREATED AT
alertmanagerconfigs.monitoring.coreos.com     2023-09-18T09:39:32Z
alertmanagers.monitoring.coreos.com           2023-09-18T09:39:32Z
bgpconfigurations.crd.projectcalico.org       2020-12-04T06:31:26Z
bgppeers.crd.projectcalico.org                2020-12-04T06:31:26Z
blockaffinities.crd.projectcalico.org         2020-12-04T06:31:26Z
clusterinformations.crd.projectcalico.org     2020-12-04T06:31:26Z
felixconfigurations.crd.projectcalico.org     2020-12-04T06:31:26Z
globalnetworkpolicies.crd.projectcalico.org   2020-12-04T06:31:26Z
globalnetworksets.crd.projectcalico.org       2020-12-04T06:31:26Z
hostendpoints.crd.projectcalico.org           2020-12-04T06:31:26Z
ipamblocks.crd.projectcalico.org              2020-12-04T06:31:26Z
ipamconfigs.crd.projectcalico.org             2020-12-04T06:31:26Z
ipamhandles.crd.projectcalico.org             2020-12-04T06:31:26Z
ippools.crd.projectcalico.org                 2020-12-04T06:31:26Z
networkpolicies.crd.projectcalico.org         2020-12-04T06:31:26Z
networksets.crd.projectcalico.org             2020-12-04T06:31:26Z
podmonitors.monitoring.coreos.com             2023-09-18T09:39:33Z
probes.monitoring.coreos.com                  2023-09-18T09:39:33Z
prometheusagents.monitoring.coreos.com        2023-09-18T09:39:33Z
prometheuses.monitoring.coreos.com            2023-09-18T09:39:33Z
prometheusrules.monitoring.coreos.com         2023-09-18T09:39:33Z
scrapeconfigs.monitoring.coreos.com           2023-09-18T09:39:33Z
servicemonitors.monitoring.coreos.com         2023-09-18T09:39:33Z
thanosrulers.monitoring.coreos.com            2023-09-18T09:39:33Z

[root@kubemaster01 ~]# kubectl  -n monitoring get   Prometheus   
NAME      AGE
inst-cc   15d
k8s       114d
[root@kubemaster01 ~]# kubectl  -n monitoring get   Prometheus  k8s  -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"baseImage":"quay.io/prometheus/prometheus","nodeSelector":{"kubernetes.io/os":"linux"},"podMonitorNamespaceSelector":{},"podMonitorSelector":{},"replicas":2,"resources":{"requests":{"memory":"400Mi"}},"ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"alert-rules"}},"securityContext":{"fsGroup":2000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"prometheus-k8s","serviceMonitorNamespaceSelector":{},"serviceMonitorSelector":{},"version":"v2.11.0"}}
  creationTimestamp: "2023-09-19T03:10:41Z"
  generation: 5
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
  resourceVersion: "348782637"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/k8s
  uid: 37ac4ee8-fed2-4837-bb2a-1d6e12835051
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 0
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  secrets:
  - prometheus-k8s
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0

 由此可以看出，Prometheus-Operater的statefulset类型实际是由自定义的Prometheus的kind控制，而Prometheus的配置文件是由名为prometheus-k8s的Opaque类型的secret控制。数据全部写入CRD内。

故我们无法直接修改statefulset和Prometheus的yaml配置。

官方找到说明 ：https://yunlzheng.gitbook.io/prometheus-book/part-iii-prometheus-shi-zhan/operator/use-custom-configuration-in-operatorhttps://yunlzheng.gitbook.io/prometheus-book/part-iii-prometheus-shi-zhan/operator/use-custom-configuration-in-operator

 二、解决思路

1.创建新的Prometheus的kind类型：

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  creationTimestamp: "2023-12-26T08:51:08Z"
  generation: 18
  labels:
    prometheus: inst-cc
  name: inst-cc
  namespace: monitoring
  resourceVersion: "349073537"
  selfLink: /apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheuses/inst-cc
  uid: f81a2b8e-b9c8-49f9-827e-116997861b61
spec:
  baseImage: quay.io/prometheus/prometheus
  replicas: 3
  resources:
    requests:
      memory: 400Mi
  serviceAccountName: prometheus-k8s
  version: v2.33.3

会自动创建secret如下：

 将自定义的配置文件压缩成.gz并且生成base64码：

[root@kubemaster01 ~]#  base64 -w 0  prometheus.yaml.gz >file.b64 
[root@kubemaster01 ~]# cat file.b64 
H4sICGw4jWUAA3Byb21ldGhldXMueWFtbADtXUtz2zgSvvtX8LAHJVt62eNxopxSM3OYw8xkd38ACiJbElYkwQCgHe1y//s2QFIkJJIiLctRykiVUhIejUY/vgbRLXkd8iUNFzeeB480TKliPCYsViDw48K7m0nskr6gCRw1wzf8HNOQhHQJodREPC8RPAK1gVQuvIjHTHHB4vV0+0EedBMBSch8uvD+Nvry16/kz89//PbuRqQhkBULAcmNvSkof1pNmepeWWsYI9mxbjQzxrPp+8mORuFNwbDP4xVbG0r/5ksS0wgspmgIQkU0pmsQ0xnyt+ExF+V2vBUNJWDrNl3iPkGBJDKoiHre2BM8RJIQBwlH6eRb1MvIhPpQSKRoKT+Maxy0y1aA4cJejfpaPQtvC5AYapKnwgdL/mOPEJQOJXWukTQrx5H6rs0UAWv4hoKhLD5rmVIMJOFCGWnXyT/B0lAfSowGgQApiaJiDYpsWRwMnrTnRUJCBUXhL7xPdeb+5AF8Gk3evysakxD1F0Gs0Dr/O/+faS1oGcYXXowzrnc/X3gwbDsJD4bsZm/jTZKp9fUmWJrofmc2zaJ7CEXcURu1gZs9wRtCS6uYxwdjS4UeT9Du0QJTPhcQxPIVEUrTG8udVBA1QdT8XtNZAhUgiOJbiA1kL7zpIxUI0vFUgi9AyWnF2ITxaSFI6vs8jdXUzLws1mF8IDRJ6q5h9obivCDU4WjBfHm98ODgzsHdZeDubC9tW8zoeTT5e7uO+2Nt6Z8teLsWdIXHo6s5EeZwe8kToY2Qxf4vCJAbpZLrRROHjg4dv+9h0PhHCzqZ8wtNmGbkVR9bA1jRNFRNCFVeE2xAM6u5159VKItlcyIsxmNhKoDILUsIMs9Wu4pdz/Pps46RPp34QhUeoGVSyKya9IMcVn0eJTyGvUEY591r+gUXSgR/ZIH99G9J65K4747FDvjfHPC/AB68xsG49M/8gEwasa5lK1oshBwiSoirjrQOnrjYgo5RCmJ/RyLmC464y+NAZrpfIouqudtfC54mpLixbCdRLNFBKQlhrTfFZI8BZXSzR2C8UAx3yhMQ5rK8qamZeMM4EIILmQEq3Vy8I+8KH4XWYM8MwFhFEqZrFiN/a+QvJ0FMcDoYQMOQ+808oJlpMZUjT7Ccm1QZCgLBOw/txxagDwRBGqJSRnCLYT//iIeYZuZq/TRc42lHbSKMVRAwH4eTKkHRNpLpd7seAwGixIjv1NBGPpcI1n02Ufa/kDT3JwEyEvA1BbTRXPvlp3w9BvK4hcg0iqjYZXrZBAJSjEB+MWxogwuohiLYkTXEhVHUpttbLKagDcZyxUWUj15RFuLpDsMTVzRsG9VCE48k33ZEpXGMeCN3sV+X3tmiK6Eo4L6GiFHNe0957bG31lq0P/NUncuegFUIPsqLjJiCCKOWxjl08qz6+ESVv8kMMgVpoZ+SR926F/tGH7zM6L0qzKfjadagc/cAyg/IaAOhZtanSJhsWGmhVnPEpGxqxwgkdgcmnffksevQlO0xeDbqGHPUc+52D6x61M/KbRc5l4cRDbQs9XpfUySo47TCRzsNuCgOjJwiIDRFu7Ijxn6QCQWlWWSfv/z+r/z88VcCMX76vF7jTNP5y37SP1JIYZ7T//yI+6HLELA7YPY4RBg8Q0DOBLpMjPBVXzvFI0XMJIJlzkYpooqLf9a4bqDbsfmQx2utdYy4sQZiBBd8rJXoX/bBQgS/YcRdovtsDF4/b1rbTlo23kszdeEhdRqy/2DrVy37Eqa6pW+rNjZnC5H6CltCE4gjI7F8Xn3lABK1Oclfq/a6eDI7/5VJn2MQ2/0yWF8VQ0eyOMWvWXqAFNo8o8Um9zqtdGVT0H3owHpzjesNtKGeAmuSdS/ze45CurbQtfVjRXaNLh1jgC6HLZDbf6cdPxMnKs+qrKQ/qU6mczQegt9nWJAtz6YRA625EPkQ9g+QcMjUNuaaNmI7sWXuvfzo2PQq1ZeW3GlqbcxarPTU5QCXOR2cewi6hyflQ5ok3yPK7/3J8rLhrlCLtRYlW3kDpPcc9DwOFw2OsjeZU/HoiJq1seO1KqtsiGQVbjWwdIYjDoyxwwJ+D+OzCXYY2rAA0kuT9qAmB6ik3rVYT1M7KdxDCG4ZNiBcVfbTEQ0ss+zh8m2kBjjnAT60bLQVd9t564gVuXStpnKz+mYiv/swlx75W1Pm2nKJaJ54Q4YPy8R+uN09RxoFGy/z8B/AMl2vUZVZwOS2fPbO8kurd5P3L3b/VWnBUmzZWHvYzyVCLrI26jG50KpjL4Tm5UudH17mkGXqb0F9Gs0m8/tsNrnV/93pl37zk37pNz/j6wFfH/D1MZvrYfOJfj3cZ7f45i67w/9/wtfP2UP2IcMx2IHts+x+1mAlXTnqSjfj16+xPlHBePEi67bCw2OhXDjdOnbFiC7r6rKu11uMeOCkLvXqUq8u9epSry716lKvLvXqUq8u9epSry716lKvLvXqUq8u9epSry716lKvLvXqUq8u9epSr+ekXrsyd/vLQ5ewq2Th8nQuT+fydC5P1/XLITliKAwK5cgGBMXw8L1+Va5oLu7Oz/vefrGNH+GL9a3YXtfUhfFdjve/r3eVaOjQ3aH7G0f3mouWxFFOWcFYVk7O9uq1z+5mjdNnays6zF/tfN0vPLhA8AqBQEK4ul7gdIHABYI3HwiMi3bAeAjq9U72va5G3ip2oyYuf3J3VzMOsx1mXzlm9/NSPYQkVG2Ka3R7jfqAG1eM7YqxfVeM7YqxXTG2K8Z2xdiuGNsVY7tibFeM7YqxXTG2K8Z2xdiuGNsVY7tibFeM7YqxXTF2q0p/+GLsE0nA4wKO75kErF9fL7xpWWXi0+CRoXW6NKFLE7o0oUsTujThFaYJdTSiLNbX8mV2SvkJSaVJHpj767I9Dex2ReVWmqQZZH6S4vGE4qkQwVsPmc8GhDXt72P4pkHuiv4kev4HMN9i6LL0cekAdr1A7wKXC1xvPXANFi56QJtMECoVjX2weW20x7kVO4q+tvCBj23IygbS1/xj9X0q2y+I2tWerWeOD5esI3+C5fWCmwNrB9aXAeu+eKnd4yRCjfP6Fd50zH2Nr2he9u+51zsZJ41/Wri6/7ro2ofg1aCCi66PTwL6prLOwqP+XfTJ/MIH6uuFNAfRDqK/L0Qb/6AIPQrRUPNi3jdfoRx/wbIRTYhegPm0JFYUShdEzF1OImDFcNI0t6bahYJpaMX7NsRvwvwD1D/C/UbkbwSiRijqqbOqWrYmivL7reeu1gJ91vH05kZAxBWQJ8EULDAepyLMZb2YTuezyfzuYfJxcjt7WHycfZxNacKmj/OpGY3UllQyX2c2Nwuv9s+sk+Ju88Cu01Bx1ZFQKZ+4CIqO+e1dNa3g5uCnDwbdIeX5z/pwn6LqmcIh97OZaYnoNyI3VAR4ipjPykYWV43VMBolOr+lKz8litRMML1LU6eq20gANAhZjHu9l3taS+pv+WqlNxHJPb19K5LB5v8D7NQqwdqRAAA=

2.修改secret的data值使其生效：

kubectl -n monitoring  edit  secret prometheus-inst-cc

 3.注意删除原来的Prometheus的默认配置

4.查看效果：