Prometheus服务发现
Prometheus添加被监控端支持两种方式:
• 静态配置:手动配置
• 服务发现:动态发现需要监控的Target实例
支持服务发现的来源
• azure_sd_configs
• consul_sd_configs
• dns_sd_configs
• ec2_sd_configs
• openstack_sd_configs
• file_sd_configs
• gce_sd_configs
• kubernetes_sd_configs
• marathon_sd_configs
• nerve_sd_configs
• serverset_sd_configs
• triton_sd_configs
集成cAdvisor
Kubernetes主要提供了如下5种服务发现模式和Prometheus进行集成:
- Node
- Pod
- Endpoints
- Service
- Ingress
使用cAdvisor主要需要使用Node服务发现模式,配置方式如下所示
Node服务发现模式
kubernetes_sd_configs:
- role: node
监控方法
监控对象 | 监控指标内容 | 服务发现模式 | 监控方式 | 数据来源 |
---|---|---|---|---|
集群各节点Kubelet组件 | 各节点Kubelet的基本运行状态相关的监控指标 | node | 白盒监控 | Kubelet |
监控Kubernetes
基于k8s的监控使用基于k8s的服务发现来实现
监控指标
Cadvisor内置在kubelet当中,通过kubelet暴露的api去访问的
监控Kubernetes
普罗米修斯通过Cadvisor监控k8s
Kubernetes默认提供cAdvisor和特定节点的时间序列。我们可以创建一个作业来从每个节点的Kubernetes API中抓取这些时间序列。我们可以使用这些时间序列来监控节点,以及每个节点上的Docker守护进程和容器。
监控K8s集群Pod步骤:
现在普罗米修斯要通过服务发现连接到k8s集群,k8s授权普罗米修斯可以访问如下地址
[root@k8s-master ~]# kubectl get ep
NAME ENDPOINTS AGE
kubernetes 192.168.179.102:6443 74d
Prometheus -> apiserver(192.168.179.102:6443)->kubelet(cadvisor)
这个过程是需要授权的,所以第一步就是授权
[root@k8s-master ~]# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: kube-system
[root@k8s-master ~]# kubectl apply -f rbac.yaml
serviceaccount/prometheus created
现在要拿到创建rbac产生的token,,这是非常关键的,让普罗米修斯拿着这个token去访问api那么就具有rbac里面授予的权限了
怎么拿到这个token呢,产生的sa在kube-system上,怎么拿到这个token呢,产生的sa在kube-system上
[root@k8s-master ~]# kubectl get sa -n kube-system | grep prome
prometheus 1 4m59s
[root@k8s-master ~]# kubectl describe sa prometheus -n kube-system
Name: prometheus
Namespace: kube-system
Labels: <none>
Annotations: <none>
Image pull secrets: <none>
Mountable secrets: prometheus-token-jq2kg
Tokens: prometheus-token-jq2kg
Events: <none>
Token保存在这个secret当中 prometheus-token-jq2kg
[root@k8s-master ~]# kubectl describe secret prometheus-token-jq2kg -n kube-system
token: eyJhbGciOiJSUzI1NiIsImtpZCI6InR0cTRHNDNQUGFMeUZ5Rnp1azZnSUEyRVU0WEY1dWdEMEYwd056ZnNkWWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLXRva2VuLWpxMmtnIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxNDYxYTU1Mi0xZWE0LTRjYWQtOTdhOC05YmE1Zjg2YjhkMmYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06cHJvbWV0aGV1cyJ9.FfRTfjE5ih9ZvCy0XFL1Trc00H7k1s6kkGmFvnkKJghswTLeATRPfziAJqrBYBYY0dA8IK52WEa0JR2TevtotnWOyIXZnv6KWcPb0RObvlL4dxp1ZJyZRAc01rliyukTU2HphgX2NlLnf_TZHMo1bapPf8crDdMlZHoEe42ukMtr1nZrPgChXJCtGoR383bAWDoDrq1nZ7e8xCQnoxEkq_khLO9ypHqAlFfMCG-w0x35uC1Wa06FdoeygW0gABDK_Ltgvz6_IuLM9wLl54SnPZJEPSMfiNpuvN8vDWNUcjqPj1Lqi3eSMKLf7b3zBvlTEcLQKoUQdXBdg-97pfeDVw
2、获取Token并保存到文件
拿到这个token,拷贝到普罗米修斯这个节点
[root@k8s-master ~]# kubectl describe secret prometheus-token-jq2kg -n kube-system > token.k8s
[root@k8s-master ~]# scp token.k8s root@192.168.179.99:/usr/local/prometheus
在普罗米修斯上只保存这token值,其余的全部去掉
[root@localhost prometheus]# cat token.k8s
eyJhbGciOiJSUzI1NiIsImtpZCI6InR0cTRHNDNQUGFMeUZ5Rnp1azZnSUEyRVU0WEY1dWdEMEYwd056ZnNkWWcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLXRva2VuLWpxMmtnIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InByb21ldGhldXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIxNDYxYTU1Mi0xZWE0LTRjYWQtOTdhOC05YmE1Zjg2YjhkMmYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06cHJvbWV0aGV1cyJ9.FfRTfjE5ih9ZvCy0XFL1Trc00H7k1s6kkGmFvnkKJghswTLeATRPfziAJqrBYBYY0dA8IK52WEa0JR2TevtotnWOyIXZnv6KWcPb0RObvlL4dxp1ZJyZRAc01rliyukTU2HphgX2NlLnf_TZHMo1bapPf8crDdMlZHoEe42ukMtr1nZrPgChXJCtGoR383bAWDoDrq1nZ7e8xCQnoxEkq_khLO9ypHqAlFfMCG-w0x35uC1Wa06FdoeygW0gABDK_Ltgvz6_IuLM9wLl54SnPZJEPSMfiNpuvN8vDWNUcjqPj1Lqi3eSMKLf7b3zBvlTEcLQKoUQdXBdg-97pfeDVw
现在可以让普罗米修斯拿着这个token访问api了
3、创建Job和kubeconfig_sd_configs
现在可以让普罗米修斯拿着这个token访问api了,这里启用的是k8s服务发现的配置
[root@localhost ~]# vim /usr/local/prometheus/prometheus.yml
- job_name: kubernetes-nodes-cadvisor
metrics_path: /metrics
scheme: https #访问api使用https访问
kubernetes_sd_configs:
- role: node #指定服务发现类型的角色为node
api_server: https://192.168.179.102:6443
bearer_token_file: /usr/local/prometheus/token.k8s
tls_config:
insecure_skip_verify: true #跳过https验证,因为自签发,不受信任,跳过证书校验
bearer_token_file: /usr/local/prometheus/token.k8s
tls_config:
insecure_skip_verify: true #跳过证书
relabel_configs:
# 将标签(.*)作为新标签名,原有值不变
- action: labelmap
regex: __meta_kubernetes_node_label_(.*)
# 修改NodeIP:10250为APIServerIP:6443
- action: replace
regex: (.*)
source_labels: ["__address__"]
target_label: __address__
replacement: 192.168.31.61:6443
# 实际访问指标接口 https://NodeIP:10250/metrics/cadvisor 这个接口只能APISERVER访问,故此重新标记>标签使用APISERVER代理访问
- action: replace
source_labels: [__meta_kubernetes_node_name]
target_label: __metrics_path__
regex: (.*)
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
[root@localhost prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
SUCCESS: 0 rule files found
https://192.168.179.102:6443/api/v1/nodes/k8s-node1/proxy/metrics/cadvisor
这些数据就是从这个地址下面拿到的 ,如果你将该段去掉,可以看到没有重新标记标签会采集不到数据
# - action: replace
# source_labels: [__meta_kubernetes_node_name]
# target_label: __metrics_path__
# regex: (.*)
# replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor