K8S环境部署Prometheus

K8S环境部署Prometheus

记录在K8S 1.18版本环境下部署Prometheus 0.5版本。

1. 下载kube-prometheus仓库

git clone https://github.com/coreos/kube-prometheus.git
cd kube-prometheus

笔者安装的K8S版本是1.18 ,prometheus选择配套的分支release-0.5:

# 切换到release-0.5
git checkout remotes/origin/release-0.5 -b 0.5

K8S和Pormetheus的配套关系:

kube-prometheus stackKubernetes 1.14Kubernetes 1.15Kubernetes 1.16Kubernetes 1.17Kubernetes 1.18
release-0.3
release-0.4
release-0.5
HEAD

最新的版本配套关系查看kube-prometheus官方仓库:https://github.com/prometheus-operator/kube-prometheus ,可以切换版本查看配套关系。

2. 查看manifest

[root@k8s-master kube-prometheus]# cd manifests/
[root@k8s-master manifests]# ll
total 1684
-rw-r--r-- 1 root root     405 Jun 12 16:20 alertmanager-alertmanager.yaml
-rw-r--r-- 1 root root     973 Jun 12 16:20 alertmanager-secret.yaml
-rw-r--r-- 1 root root      96 Jun 12 16:20 alertmanager-serviceAccount.yaml
-rw-r--r-- 1 root root     254 Jun 12 16:20 alertmanager-serviceMonitor.yaml
-rw-r--r-- 1 root root     308 Jun 12 16:22 alertmanager-service.yaml
-rw-r--r-- 1 root root     550 Jun 12 16:20 grafana-dashboardDatasources.yaml
-rw-r--r-- 1 root root 1405645 Jun 12 16:20 grafana-dashboardDefinitions.yaml
-rw-r--r-- 1 root root     454 Jun 12 16:20 grafana-dashboardSources.yaml
-rw-r--r-- 1 root root    7539 Jun 12 16:20 grafana-deployment.yaml
-rw-r--r-- 1 root root      86 Jun 12 16:20 grafana-serviceAccount.yaml
-rw-r--r-- 1 root root     208 Jun 12 16:20 grafana-serviceMonitor.yaml
-rw-r--r-- 1 root root     238 Jun 12 16:22 grafana-service.yaml
-rw-r--r-- 1 root root     376 Jun 12 16:20 kube-state-metrics-clusterRoleBinding.yaml
-rw-r--r-- 1 root root    1651 Jun 12 16:20 kube-state-metrics-clusterRole.yaml
-rw-r--r-- 1 root root    1925 Jun 12 16:20 kube-state-metrics-deployment.yaml
-rw-r--r-- 1 root root     192 Jun 12 16:20 kube-state-metrics-serviceAccount.yaml
-rw-r--r-- 1 root root     829 Jun 12 16:20 kube-state-metrics-serviceMonitor.yaml
-rw-r--r-- 1 root root     403 Jun 12 16:20 kube-state-metrics-service.yaml
-rw-r--r-- 1 root root     266 Jun 12 16:20 node-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     283 Jun 12 16:20 node-exporter-clusterRole.yaml
-rw-r--r-- 1 root root    2775 Jun 12 16:20 node-exporter-daemonset.yaml
-rw-r--r-- 1 root root      92 Jun 12 16:20 node-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root     711 Jun 12 16:20 node-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root     355 Jun 12 16:20 node-exporter-service.yaml
-rw-r--r-- 1 root root     292 Jun 12 16:20 prometheus-adapter-apiService.yaml
-rw-r--r-- 1 root root     396 Jun 12 16:20 prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
-rw-r--r-- 1 root root     304 Jun 12 16:20 prometheus-adapter-clusterRoleBindingDelegator.yaml
-rw-r--r-- 1 root root     281 Jun 12 16:20 prometheus-adapter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     188 Jun 12 16:20 prometheus-adapter-clusterRoleServerResources.yaml
-rw-r--r-- 1 root root     219 Jun 12 16:20 prometheus-adapter-clusterRole.yaml
-rw-r--r-- 1 root root    1378 Jun 12 16:20 prometheus-adapter-configMap.yaml
-rw-r--r-- 1 root root    1344 Jun 12 16:20 prometheus-adapter-deployment.yaml
-rw-r--r-- 1 root root     325 Jun 12 16:20 prometheus-adapter-roleBindingAuthReader.yaml
-rw-r--r-- 1 root root      97 Jun 12 16:20 prometheus-adapter-serviceAccount.yaml
-rw-r--r-- 1 root root     236 Jun 12 16:20 prometheus-adapter-service.yaml
-rw-r--r-- 1 root root     269 Jun 12 16:20 prometheus-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     216 Jun 12 16:20 prometheus-clusterRole.yaml
-rw-r--r-- 1 root root     621 Jun 12 16:20 prometheus-operator-serviceMonitor.yaml
-rw-r--r-- 1 root root     751 Jun 12 16:20 prometheus-prometheus.yaml
-rw-r--r-- 1 root root     293 Jun 12 16:20 prometheus-roleBindingConfig.yaml
-rw-r--r-- 1 root root     983 Jun 12 16:20 prometheus-roleBindingSpecificNamespaces.yaml
-rw-r--r-- 1 root root     188 Jun 12 16:20 prometheus-roleConfig.yaml
-rw-r--r-- 1 root root     820 Jun 12 16:20 prometheus-roleSpecificNamespaces.yaml
-rw-r--r-- 1 root root   86744 Jun 12 16:20 prometheus-rules.yaml
-rw-r--r-- 1 root root      93 Jun 12 16:20 prometheus-serviceAccount.yaml
-rw-r--r-- 1 root root    6829 Jun 12 16:20 prometheus-serviceMonitorApiserver.yaml
-rw-r--r-- 1 root root     395 Jun 12 16:20 prometheus-serviceMonitorCoreDNS.yaml
-rw-r--r-- 1 root root    6172 Jun 12 16:20 prometheus-serviceMonitorKubeControllerManager.yaml
-rw-r--r-- 1 root root    6778 Jun 12 16:20 prometheus-serviceMonitorKubelet.yaml
-rw-r--r-- 1 root root     347 Jun 12 16:20 prometheus-serviceMonitorKubeScheduler.yaml
-rw-r--r-- 1 root root     247 Jun 12 16:20 prometheus-serviceMonitor.yaml
-rw-r--r-- 1 root root     297 Jun 12 16:21 prometheus-service.yaml
drwxr-xr-x 2 root root    4096 Jun 12 16:20 setup

3. 修改镜像源

修改prometheus-operator,prometheus,alertmanager,kube-state-metrics,node-exporter,prometheus-adapter的镜像源为中科大的镜像源。

sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' setup/prometheus-operator-deployment.yaml
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' prometheus-prometheus.yaml 
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' alertmanager-alertmanager.yaml
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' kube-state-metrics-deployment.yaml
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' node-exporter-daemonset.yaml
sed -i 's/quay.io/quay.mirrors.ustc.edu.cn/g' prometheus-adapter-deployment.yaml

4. 修改promethes,alertmanager,grafana的service类型为NodePort类型

为了可以从外部访问 prometheus,alertmanager,grafana,我们这里修改 promethes,alertmanager,grafana的 service 类型为 NodePort 类型。

  1. 修改 prometheus 的 service
[root@k8s-master kube-prometheus]# cat manifests/prometheus-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  type: NodePort        # 增加NodePort配置
  ports:
  - name: web
    port: 9090
    targetPort: web
    nodePort: 30090       # 增加NodePort配置
  selector:
    app: prometheus
    prometheus: k8s
  sessionAffinity: ClientIP
  1. 修改 grafana 的 service
[root@k8s-master kube-prometheus]# cat manifests/grafana-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  type: NodePort      # 增加NodePort配置
  ports:
  - name: http
    port: 3000
    targetPort: http
    nodePort: 32000   # 增加NodePort配置
  selector:
    app: grafana
  1. 修改 alertmanager 的 service
[root@k8s-master kube-prometheus]# cat manifests/alertmanager-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    alertmanager: main
  name: alertmanager-main
  namespace: monitoring
spec:
  type: NodePort    # 增加NodePort配置
  ports:
  - name: web
    port: 9093
    targetPort: web
    nodePort: 30093   # 增加NodePort配置
  selector:
    alertmanager: main
    app: alertmanager
  sessionAffinity: ClientIP

5. 安装kube-prometheus

安装CRD和prometheus-operator

[root@k8s-master manifests]# kubectl apply -f setup/
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
namespace/monitoring configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl                                    apply
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com configured
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
[root@k8s-master manifests]# kubectl get pod -n monitoring
NAME                                   READY   STATUS              RESTARTS   AGE
prometheus-operator-5cd4d464cc-b9vqq   0/2     ContainerCreating   0          16s

下载prometheus-operator镜像需要花费几分钟,等待prometheus-operator变成running状态。

安装prometheus, alertmanager, grafana, kube-state-metrics, node-exporter等资源

[root@k8s-master manifests]# kubectl apply -f .
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-datasources created
configmap/grafana-dashboard-apiserver created
configmap/grafana-dashboard-cluster-total created
configmap/grafana-dashboard-controller-manager created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-node created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-kubelet created
configmap/grafana-dashboard-namespace-by-pod created
configmap/grafana-dashboard-namespace-by-workload created
configmap/grafana-dashboard-node-cluster-rsrc-use created
configmap/grafana-dashboard-node-rsrc-use created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pod-total created
configmap/grafana-dashboard-prometheus-remote-write created
configmap/grafana-dashboard-prometheus created
configmap/grafana-dashboard-proxy created
configmap/grafana-dashboard-scheduler created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboard-workload-total created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus-operator created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created

等待monitoring命名空间下的pod都变为运行:

[root@k8s-master ~]# kubectl get  pod -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   0          157m
alertmanager-main-1                    2/2     Running   0          157m
alertmanager-main-2                    2/2     Running   0          157m
grafana-5c55845445-gh8g7               1/1     Running   0          20h
kube-state-metrics-75f946484-lqrbf     3/3     Running   0          20h
node-exporter-5h5cs                    2/2     Running   0          20h
node-exporter-f28gj                    2/2     Running   0          20h
node-exporter-w9rhr                    2/2     Running   0          20h
prometheus-adapter-7d68d6f886-qwrfg    1/1     Running   0          20h
prometheus-k8s-0                       3/3     Running   0          20h
prometheus-k8s-1                       3/3     Running   0          20h
prometheus-operator-5cd4d464cc-b9vqq   2/2     Running   0          20h

博主部署测试遇到如下问题,解决方式记录如下:

  1. alertmanager-main的三个容器启动失败,状态为crashLoopBackOff,pod中的其中一个容器无法启动。
  Warning  Unhealthy  11m (x5 over 11m)    kubelet, k8s-node2  Liveness probe failed: Get http://10.244.2.8:9093/-/healthy: dial tcp 10.244.2.8:9093: connect: connection refused
  Warning  Unhealthy  10m (x10 over 11m)   kubelet, k8s-node2  Readiness probe failed: Get http://10.244.2.8:9093/-/ready: dial tcp 10.244.2.8:9093: connect: connection refused

下面的解决方法参考自:https://github.com/prometheus-operator/kube-prometheus/issues/653

# 暂停更新,修改如下资源文件,增加paused:true
kubectl -n monitoring edit alertmanagers.monitoring.coreos.com
...
spec:
  image: quay.io/prometheus/alertmanager:v0.23.0
  nodeSelector:
    kubernetes.io/os: linux
  paused: true
  podMetadata:
    labels:
...

[root@k8s-master ~]# kubectl -n monitoring get statefulset.apps/alertmanager-main -o yaml > dump.yaml
# 修改alertmanager-main.yaml,在spec.template.spec添加hostNetwork: true,在文件的234行左右的位置
[root@k8s-master manifests]# vi dump.yaml
...
    spec:
      hostNetwork: true   # 增加的内容
      containers:
      - args:
        - --config.file=/etc/alertmanager/config/alertmanager.yaml
...
# 删除livenessProbe和readinessProbe探针
[root@k8s-master manifests]# vi dump.yaml
...
        livenessProbe:
          failureThreshold: 10
          httpGet:
            path: /-/healthy
            port: web
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
...
        readinessProbe:
          failureThreshold: 10
          httpGet:
            path: /-/ready
            port: web
            scheme: HTTP
          initialDelaySeconds: 3
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 3
...

# 删除原有的statefulset,重新创建
[root@k8s-master ~]# kubectl delete statefulset.apps/alertmanager-main -n monitoring
[root@k8s-master ~]# kubectl create -f dump.yaml
  1. 其中一个alertmanager状态为pendding,查看原因为不满足节点调度要求。

解决方法如下,

# 去除污点
kubectl describe node k8s-master | grep Taints
kubectl taint nodes k8s-master node-role.kubernetes.io/master-

6. 访问prometheus,alert-manager,grafana

  1. 访问prometheus

浏览器打开http://192.168.0.51:30090,192.168.0.51为master的IP

  1. 访问alert-manager

浏览器打开http://192.168.0.51:30093

  1. 访问grafana

浏览器打开http://192.168.0.51:32000

用户名/密码:admin/admin

  • 29
    点赞
  • 7
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论
### 回答1: 要在Kubernetes 1.26上部署Prometheus和钉钉告警,可以遵循以下步骤: 1. 首先,您需要创建一个Prometheus的配置文件,其中包含Prometheus服务器的相关配置信息。您可以通过使用Prometheus Operator或手动创建一个配置文件。 2. 在该配置文件中,您需要定义钉钉告警的接收者。您可以添加一个名为"dingtalk"的接收者,并提供钉钉机器人的Webhook地址。这将允许Prometheus发送告警通知到钉钉机器人。 3. 接下来,您需要将该配置文件应用到Kubernetes集群中。您可以使用kubectl命令应用一个YAML文件,其中定义了Prometheus服务器的Deployment和Service。 4. 在Prometheus服务器运行后,您可以使用kubectl port-forward命令将Prometheus的端口映射到本地,以便可以访问Prometheus的Web界面。 5. 在Prometheus的Web界面中,您可以配置和监控各种指标,并设置相应的报警规则。您可以通过PromQL查询语言定义报警规则,并指定钉钉接收者。 6. 当报警规则触发时,Prometheus将发送告警通知到钉钉机器人。您可以在钉钉中收到告警通知,并进行相应的处理。 总结:以上是在Kubernetes 1.26上部署Prometheus和钉钉告警的大致步骤。具体的实施方式可能会根据您的环境和需求而有所不同。需要注意的是,配置和使用Prometheus和钉钉告警需要一定的技术知识和经验。 ### 回答2: 部署k8s 1.26版本中的Prometheus并配置钉钉告警需要以下步骤: 1. 安装Prometheus:根据k8s版本选择适当的Prometheus版本,并将其部署kubernetes集群中。可以使用Helm进行安装:`helm install prometheus stable/prometheus` 2. 配置告警规则:在Prometheus中创建告警规则,以定义要监控的指标和阈值。可以通过编辑Prometheus配置文件或使用Prometheus的API来创建规则。 3. 设置钉钉告警:首先,需要在钉钉开发者平台上创建一个机器人,并获取其Webhook地址。然后,在Prometheus的配置文件中添加相应的钉钉告警配置,包括Webhook地址和其他相关参数。 4. 测试告警:修改某些指标或触发条件,以便使Prometheus触发告警规则。可以使用下面的命令手动触发告警:`kubectl -n <prometheus-namespace> delete po -l <label-selector>` 5. 验证告警:检查钉钉群组中是否收到Prometheus的告警通知。如果一切配置正确,钉钉机器人将发送告警消息到指定的群组中。 需要注意的是,部署Prometheus和配置钉钉告警可能会因k8s版本和Prometheus版本的不同而有所差异。以上步骤仅提供了一个大致的实现方案,在实际操作中可能需要根据具体版本和需求进行一些调整和修改。 ### 回答3: 在k8s 1.26版本中部署Prometheus并配置钉钉告警的过程如下: 1. 在k8s集群中创建一个命名空间,用于部署和管理Prometheus及相关组件。 2. 在k8s集群中创建一个ConfigMap,用于存储Prometheus的配置文件。配置文件包含了需要监控的目标(例如,Pod、Service等)和具体的告警规则。 3. 创建一个ServiceAccount,并为其授权访问钉钉的权限。这是为了Prometheus能够发送告警通知到钉钉。 4. 部署Prometheus Operator。Prometheus Operator是一个用于管理Prometheus实例的部署程序。 5. 创建Prometheus资源。使用Prometheus Operator创建一个Prometheus资源,并关联之前创建的ConfigMap。 6. 配置Prometheus的钉钉告警规则。在ConfigMap内部,添加相应的告警规则配置,用于定义在何种情况下触发告警,并发送通知到钉钉。 7. 部署Alertmanager。Alertmanager是Prometheus的一个组件,用于接收来自Prometheus的告警通知,并将其发送到相应的接收者。 8. 配置Alertmanager的钉钉通知方式。在Alertmanager的配置文件中,添加钉钉的通知方式,并填写相应的钉钉机器人Webhook地址。 9. 部署钉钉机器人。在钉钉中创建一个机器人,并获取其Webhook地址。 10. 部署完成后,Prometheus会开始监控集群中的目标,并根据配置的告警规则发送告警通知到Alertmanager。Alertmanager会将告警通知通过钉钉机器人的Webhook地址发送到钉钉。 通过以上步骤,我们成功在k8s 1.26版本中部署Prometheus,并配置了钉钉告警。这样就能及时获取到集群中的异常情况,并通过钉钉进行告警通知,便于及时处理和监控运维。
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

lldhsds

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值