k8s 部署 prometheus及集群层面的监控

最新推荐文章于 2024-09-18 18:27:02 发布

努力中的懒癌晚期

最新推荐文章于 2024-09-18 18:27:02 发布

阅读量222

点赞数

文章标签： kubernetes prometheus 容器云原生

Prometheus部署

部署对外可访问Prometheus:

首先需要创建Prometheus所在命名空间；
然后创建Prometheus使用的RBAC规则；
创建Prometheus的configmap来保存配置文件；
创建service暴露Prometheus服务；
创建deployment部署Prometheus容器；
最后创建Ingress实现外部域名访问Prometheus。

创建名称空间

创建 RABC规则

使用ConfigMap方式创建prometheus rules配置文件:

包含的内容是两块，分别是general.rules和node.rules。使用以下命令创建Prometheus的另外两个配置文件：

#创建一个yaml文件，包含sa账号、clusterrole及clusterrolebinding
vim prometheus.yaml    //加入下列内容


apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitor

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources: ["nodes","nodes/proxy","services","endpoints","pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources: ["ingress"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitor
  
  
  
#创建
kubectl apply -f prometheus.yaml

验证sa账号、clusterrole及clusterrolebinding  如下图所示

k8s 部署 prometheus及集群层面的监控_主机名

创建ConfigMap类型的Prometheus配置文件

cat configmap-prometheus-01.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitor
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s
      external_labels:
        cluster: "kubernetes"

    ############ 数据采集job ###################
    scrape_configs:
    - job_name: prometheus
      static_configs:
      - targets: ['127.0.0.1:9090']
        labels:
          instance: prometheus

    ############ 指定告警规则文件路径位置 ###################
    rule_files:
    - /etc/prometheus/rules/*.rules
    
#应用
kubectl apply -f configmap-prometheus-01.yaml

创建ConfigMap类型的prometheus rules配置文件

使用ConfigMap方式创建prometheus rules配置文件:

包含的内容是两块，分别是general.rules和node.rules。使用以下命令创建Prometheus的另外两个配置文件：

cat configmap-prometheus-02.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rules
  namespace: monitor
data:
  general.rules: |
    groups:
    - name: general.rules
      rules:
      - alert: InstanceDown
        expr: |
          up{job=~"k8s-nodes|prometheus"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} 停止工作"
          description: "{{ $labels.instance }} 主机名：{{ $labels.hostname }} 已经停止1分钟以上."

  node.rules: |
    groups:
    - name: node.rules
      rules:
      - alert: NodeFilesystemUsage
        expr: |
          100 - (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"
          description: "{{ $labels.instance }} 主机名：{{ $labels.hostname }} : {{ $labels.mountpoint }} 分区使用大于85% (当前值: {{ $value }})"


#应用
kubectl apply -f configmap-prometheus-02.yaml

创建prometheus svc

# vim prometheus-svc.yaml  加入如下内容


apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitor
  labels:
    k8s-app: prometheus
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 9090
    targetPort: 9090
  selector:
    k8s-app: prometheus
    
    
# 创建   如下图所示
# kubectl apply -f prometheus-svc.yaml

k8s 部署 prometheus及集群层面的监控_Group_02

创建prometheus 的控制器 deployment

先创建一个pvc

创建pvc 之前还需要存储类，所以需要先创建一个存储类

存储类(storageclass),使用类型可以动态的自动创建pv，k8s管理员通过创建storageclass可以动态生成一个存储卷pv供pvc使用。

kubectl explain storageclass 使用该命令可以查看该字段信息。

其中比较重要的字段解释如下

provisioner

可理解为提供商，使用storageclass时需要有一个供应者用来动态的生成符合条件的pv，然后由该字段指定供应者来创建pv

reclaimPolicy

定义回收策略，默认的是delete

创建nfs提供商

以nfs共享存储为例进行创建，将nfs作为提供商，从nfs共享目录中划分存储。

nfs属于是k8s外部供应商，要想使用nfs需要安装一个自动装载程序--nfs-client，称之为provisioner，这个程序会使用已经配置好的nfs服务在配置nfs共享目录的服务器上自动创建持久卷，也就是自动创建pv。

创建nfs提供商需要使用到这个镜像，可在docker上拉取下载

创建运行nfs-provisioner需要的sa账号

因为nfs 提供商是以pod中的容器形式在k8s中运行的，所以需要创建一个sa账号并赋权，让他能有权限操作k8s并与k8s api通信

才能从k8s中创建存储出来。

sa的全称是serviceaccount。

serviceaccount是为了方便Pod里面的进程调用Kubernetes API或其他外部服务而设计的。

指定了serviceaccount之后，我们把pod创建出来了，我们在使用这个pod时，这个pod就有了我们指定的账户的权限了。

# 然后创建一个由deployment控制器管理的pod（prometheus）
# vim prometheus-deploy.yaml    //加入下列内容


apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitor
  labels:
    k8s-app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: prometheus
  template:
    metadata:
      labels:
        k8s-app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.36.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9090
        securityContext:
          runAsUser: 65534
          privileged: true
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--web.enable-lifecycle"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention.time=10d"
        - "--web.console.libraries=/etc/prometheus/console_libraries"
        - "--web.console.templates=/etc/prometheus/consoles"
        resources:
          limits:
            cpu: 2000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 9090
          initialDelaySeconds: 5
          timeoutSeconds: 10
        livenessProbe:
          httpGet:
            path: /-/healthy
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
        volumeMounts:
        - name: data
          mountPath: /prometheus
          subPath: prometheus
        - name: config
          mountPath: /etc/prometheus
        - name: prometheus-rules
          mountPath: /etc/prometheus/rules
      - name: configmap-reload
        image: jimmidyson/configmap-reload:v0.5.0
        imagePullPolicy: IfNotPresent
        args:
        - "--volume-dir=/etc/config"
        - "--webhook-url=http://localhost:9090/-/reload"
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
        volumeMounts:
        - name: config
          mountPath: /etc/config
          readOnly: true
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: prometheus-data-pvc
      - name: prometheus-rules
        configMap:
          name: prometheus-rules
      - name: config
        configMap:
          name: prometheus-config
          
          
# 创建
# kubectl apply -f prometheus-deploy.yaml

# 验证
kubectl get pods -n monitor

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.

创建nfs提供商

创建nfs提供商之前，先找一个集群内的机器做服务端，我这里使用控制节点

#配置nfs共享
yum -y install nfs-utils
systemctl enable nfs --now
mkdir -p /nfs/test001
vim /etc/exports   //加入如下内容
/nfs/test001 *(rw,no_root_squash)

#重新加载生效
exportfs -arv


#创建nfs提供商，使用deploymen控制器管理pod以容器方式运行
[root@k8s-master ~]# vim nfs-deployment.yaml 

kind: Deployment
apiVersion: apps/v1
metadata:
  name: mynfs-provisioner
  namespace: monitor
spec:
  selector:
    matchLabels:
       name: nfs-01
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: nfs-01
    spec:
      serviceAccount: nfs-provisioner  #这里指定的就是上方创建的sa的名称，通过kubectl get sa查看
      containers:
        - name: nfs-02
          image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: nfs-client
              mountPath: /persistentvolumes   #这里定义容器内的挂载路径
          env:
            - name: PROVISIONER_NAME #这里定义nfs提供商的名称
              value: nfs-test
            - name: NFS_SERVER       #这里写nfs共享的宿主机IP
              value: 192.168.57.131
            - name: NFS_PATH         #这里定义nfs共享宿主机上的路径
              value: /nfs/test001
      volumes:
        - name: nfs-client
          nfs:
            server: 192.168.57.131   #这里要与配置nfs共享的宿主机的ip一致
            path: /nfs/test001       #这里是nfs共享的路径
            

#创建
kubectl apply -f nfs-deployment.yaml

#验证  如下图：
kubectl get pods -n monitor
NAME                                READY   STATUS    RESTARTS   AGE
mynfs-provisioner-cf4888b9d-znbzx   1/1     Running   0          17m

k8s 部署 prometheus及集群层面的监控_vim_03

创建存储类（torageclass）

vi nfs-storageclass.yaml   //加入下列内容 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage             #定义存储类的名称
  namespace: monitor
provisioner: nfs-test   #这里要与创建nfs提供商时env里面时定义提供商的名称一致

#创建
kubectl apply -f nfs-storageclass.yaml

验证如图

k8s 部署 prometheus及集群层面的监控_Group_04

创建pvc

# vim prometheus-pvc.yaml   //加入下列内容


apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data-pvc
  namespace: monitor
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "nfs-storage"
  resources:
    requests:
      storage: 10Gi
      
  
# 创建
# kubectl apply -f prometheus-pvc.yaml 

# 验证 如下图已经绑定
# kubectl get pvc prometheus-data-pvc -n monitor

k8s 部署 prometheus及集群层面的监控_Group_05

# 然后创建一个由deployment控制器管理的pod（prometheus）
# vim prometheus-deploy.yaml    //加入下列内容


apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitor
  labels:
    k8s-app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: prometheus
  template:
    metadata:
      labels:
        k8s-app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.36.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9090
        securityContext:
          runAsUser: 65534
          privileged: true
        command:
        - "/bin/prometheus"
        args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--web.enable-lifecycle"
        - "--storage.tsdb.path=/prometheus"
        - "--storage.tsdb.retention.time=10d"
        - "--web.console.libraries=/etc/prometheus/console_libraries"
        - "--web.console.templates=/etc/prometheus/consoles"
        resources:
          limits:
            cpu: 2000m
            memory: 2048Mi
          requests:
            cpu: 1000m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 9090
          initialDelaySeconds: 5
          timeoutSeconds: 10
        livenessProbe:
          httpGet:
            path: /-/healthy
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
        volumeMounts:
        - name: data
          mountPath: /prometheus
          subPath: prometheus
        - name: config
          mountPath: /etc/prometheus
        - name: prometheus-rules
          mountPath: /etc/prometheus/rules
      - name: configmap-reload
        image: jimmidyson/configmap-reload:v0.5.0
        imagePullPolicy: IfNotPresent
        args:
        - "--volume-dir=/etc/config"
        - "--webhook-url=http://localhost:9090/-/reload"
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 10m
            memory: 10Mi
        volumeMounts:
        - name: config
          mountPath: /etc/config
          readOnly: true
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: prometheus-data-pvc
      - name: prometheus-rules
        configMap:
          name: prometheus-rules
      - name: config
        configMap:
          name: prometheus-config
          
          
# 创建
# kubectl apply -f prometheus-deploy.yaml

# 验证
kubectl get pods -n monitor

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.

k8s 部署 prometheus及集群层面的监控_主机名_06

创建prometheus ingress实现外部域名访问

vi  prometheus-ingress.yaml   //加入下列内容


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: monitor
  name: prometheus-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: prometheus.kubernets.cn
    http:
      paths:
        - pathType: Prefix
          backend:
            service:
              name: prometheus
              port:
                number: 9090
          path: /


#创建
 kubectl apply -f prometheus-ingress.yaml 
 
#验证 如下

#验证访问测试如下图二

#浏览器输入域名进行验证，下图三：

k8s 部署 prometheus及集群层面的监控_主机名_07

k8s 部署 prometheus及集群层面的监控_Group_08

k8s 部署 prometheus及集群层面的监控_主机名_09

基于Prometheus监控集群的配置

KubeStateMetrics

kube-state-metrics 是一个 Kubernetes 组件，它通过查询 Kubernetes 的 API 服务器，收集关于 Kubernetes 中各种资源（如节点、pod、服务等）的状态信息，并将这些信息转换成 Prometheus 可以使用的指标

kube-state-metrics 主要功能

节点状态信息，如节点 CPU 和内存的使用情况、节点状态、节点标签等。
Pod 的状态信息，如 Pod 状态、容器状态、容器镜像信息、Pod 的标签和注释等。
Deployment、Daemonset、Statefulset 和 ReplicaSet 等控制器的状态信息，如副本数、副本状态、创建时间等。
Service 的状态信息，如服务类型、服务 IP 和端口等。
存储卷的状态信息，如存储卷类型、存储卷容量等。
Kubernetes 的 API 服务器状态信息，如 API 服务器的状态、请求次数、响应时间等。

通过 kube-state-metrics 可以方便的对 Kubernetes 集群进行监控，发现问题，以及提前预警。

部署KubeStateMetrics

包含ServiceAccount、ClusterRole、ClusterRoleBinding、Deployment、ConfigMap、Service 六类YAML文件

#  cat metrics.yaml   //加入下列内容


apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: monitor
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]
- apiGroups: ["networking.k8s.io", "extensions"]
  resources:
  - ingresses
  verbs: ["list", "watch"]
- apiGroups: ["storage.k8s.io"]
  resources:
  - storageclasses
  verbs: ["list", "watch"]
- apiGroups: ["certificates.k8s.io"]
  resources:
  - certificatesigningrequests
  verbs: ["list", "watch"]
- apiGroups: ["policy"]
  resources:
  - poddisruptionbudgets
  verbs: ["list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kube-state-metrics-resizer
  namespace: monitor
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["get"]
- apiGroups: ["extensions","apps"]
  resources:
  - deployments
  resourceNames: ["kube-state-metrics"]
  verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: monitor
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-state-metrics
  namespace: monitor
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kube-state-metrics-resizer
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: monitor

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitor
  labels:
    k8s-app: kube-state-metrics
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v1.3.0
spec:
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
      version: v1.3.0
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
        version: v1.3.0
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.4.2
        ports:
        - name: http-metrics            ## 用于公开kubernetes的指标数据的端口
          containerPort: 8080           
        - name: telemetry               ##用于公开自身kube-state-metrics的指标数据的端口
          containerPort: 8081
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: addon-resizer         ##addon-resizer 用来伸缩部署在集群内的 metrics-server, kube-state-metrics等监控组件
        image: mirrorgooglecontainers/addon-resizer:1.8.6
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 30Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        volumeMounts:
          - name: config-volume
            mountPath: /etc/config
        command:
          - /pod_nanny
          - --config-dir=/etc/config
          - --container=kube-state-metrics
          - --cpu=100m
          - --extra-cpu=1m
          - --memory=100Mi
          - --extra-memory=2Mi
          - --threshold=5
          - --deployment=kube-state-metrics
      volumes:
        - name: config-volume
          configMap:
            name: kube-state-metrics-config
---
# Config map for resource configuration.
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-state-metrics-config
  namespace: monitor
  labels:
    k8s-app: kube-state-metrics
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration

---
apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitor
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "kube-state-metrics"
  annotations:
    prometheus.io/scrape: 'true'
spec:
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics
    protocol: TCP
  - name: telemetry
    port: 8081
    targetPort: telemetry
    protocol: TCP
  selector:
    k8s-app: kube-state-metrics
    
    
#创建：
kubectl apply -f metrics.yaml

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
82.
83.
84.
85.
86.
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.
103.
104.
105.
106.
107.
108.
109.
110.
111.
112.
113.
114.
115.
116.
117.
118.
119.
120.
121.
122.
123.
124.
125.
126.
127.
128.
129.
130.
131.
132.
133.
134.
135.
136.
137.
138.
139.
140.
141.
142.
143.
144.
145.
146.
147.
148.
149.
150.
151.
152.
153.
154.
155.
156.
157.
158.
159.
160.
161.
162.
163.
164.
165.
166.
167.
168.
169.
170.
171.
172.
173.
174.
175.
176.
177.
178.
179.
180.
181.
182.
183.
184.
185.
186.
187.
188.
189.
190.
191.
192.
193.
194.
195.
196.
197.
198.
199.
200.
201.
202.
203.
204.
205.
206.
207.
208.
209.
210.
211.
212.
213.
214.
215.
216.
217.
218.
219.
220.
221.
222.
223.
224.
225.
226.
227.
228.
229.
230.
231.
232.
233.
234.
235.
236.
237.
238.
239.
240.
241.

新增 Kubernetes 集群架构监控

添加监控kube-apiserver

#在configmap-prometheus-01.yaml配置文件中最下面加入下列内容
vi  configmap-prometheus-01.yaml

    - job_name: kube-apiserver
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
        action: keep
        regex: default;kubernetes
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
        
#添加完成之后 apply应用
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

添加监控controller-manager

查看并修改controller-manager信息

kubectl describe pods kube-controller-manager-k8s-master -n kube-system

cat /etc/kubernetes/manifests/kube-controller-manager.yaml

#找到如下两个参数
--bind-address=127.0.0.1   //将127.0.0.1修改为0.0.0.0
--port=0                   //注释掉该参数

#编写prometheus的配置文件需要注意的是，他默认匹配到的是80端口，需要手动指定为10252端口

#vi configmap-prometheus-01.yaml    //在最后加入下列内容


    - job_name: kube-controller-manager
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_component]
        regex: kube-controller-manager
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:10252
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
        
#添加完成之后 apply应用
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

编写prometheus配置文件，需要注意的是，他默认匹配到的是80端口，需要手动指定为10252端口

添加监控scheduler

查看scheduler 的信息

kubectl describe pod kube-scheduler-k8s-master -n kube-system

与上方controller-manager配置一致需要将下面两个参数修改

cat /etc/kubernetes/manifests/kube-scheduler.yaml

#找到如下两个参数

--bind-address=127.0.0.1   //将127.0.0.1修改为0.0.0.0
--port=0                   //注释掉该参数

#修改完成之后还需要在configmap-prometheus-01.yaml文件下添加如下内容

vi configmap-prometheus-01.yaml  //在最下方添加如下内容

    - job_name: kube-scheduler
      kubernetes_sd_configs:
      - role: pod
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_component]
        regex: kube-scheduler
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:10251
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
        
#添加完成之后应用文件
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

添加监控kube-state-metrics

编写prometheus配置文件，需要注意的是，他默认匹配到的是8080和801两个端口，需要手动指定为8080端口;

vi configmap-prometheus-01.yaml  //在最下方添加如下内容
 
 
 - job_name: kube-state-metrics
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: kube-state-metrics
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:8080
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
        
#添加完成之后应用文件
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

添加监控coredns

编写prometheus配置文件，需要注意的是，他默认匹配到的是53端口，需要手动指定为9153端口

vi configmap-prometheus-01.yaml  //在最下方添加如下内容


    - job_name: coredns
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels:
          - __meta_kubernetes_service_label_k8s_app
        regex: kube-dns
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:9153
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: service
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
        
#添加完成之后应用文件
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

添加监控etcd

查看etcd 的信息

kubectl describe pods etcd-k8s-master -n kube-system

cat /etc/kubernetes/manifests/etcd.yam

#找到如下参数并修改

--listen-metrics-urls=http://127.0.0.1:2381   //将127.0.0.1修改为0.0.0.0


#然后修改配置文件 添加etcd的监控
vi configmap-prometheus-01.yaml  //在最下方添加如下内容


    - job_name: etcd
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_label_component
        regex: etcd
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        regex: (.+)
        target_label: __address__
        replacement: ${1}:2381
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
#添加完成之后应用文件
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

上面部分参数简介如下:

kubernetes_sd_configs: 设置发现模式为 Kubernetes 动态服务发现
kubernetes_sd_configs.role: 指定 Kubernetes 的服务发现模式，这里设置为 endpoints 的服务发现模式，该模式下会调用 kube-apiserver 中的接口获取指标数据。并且还限定只获取 kube-state-metrics 所在 - Namespace 的空间 kube-system 中的 Endpoints 信息
kubernetes_sd_configs.namespace: 指定只在配置的 Namespace 中进行 endpoints 服务发现
relabel_configs: 用于对采集的标签进行重新标记

热加载prometheus，使configmap配置文件生效（也可以等待prometheus的自动热加载）：

cAdvisor

cAdvisor 主要功能：

对容器资源的使用情况和性能进行监控。它以守护进程方式运行，用于收集、聚合、处理和导出正在运行容器的有关信息。
cAdvisor 本身就对 Docker 容器支持，并且还对其它类型的容器尽可能的提供支持，力求兼容与适配所有类型的容器。
Kubernetes 已经默认将其与 Kubelet 融合，所以我们无需再单独部署 cAdvisor 组件来暴露节点中容器运行的信息。

Prometheus 添加 cAdvisor 配置

由于 Kubelet 中已经默认集成 cAdvisor 组件，所以无需部署该组件。需要注意的是，他的指标采集地址为 /metrics/cadvisor，需要配置https访问，可以设置 insecure_skip_verify: true 跳过证书验证;

vi configmap-prometheus-01.yaml  //在最下方添加如下内容

    - job_name: kubelet
      metrics_path: /metrics/cadvisor
      scheme: https
      tls_config:
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        

#添加完成之后应用文件
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

node-exporter

Node Exporter 是 Prometheus 官方提供的一个节点资源采集组件，可以用于收集服务器节点的数据，如 CPU频率信息、磁盘IO统计、剩余可用内存等等。

部署创建：

由于是针对所有K8S-node节点，所以我们这边使用DaemonSet这种方式；

cat exporter.yaml   //加入下列内容


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitor
  labels:
    name: node-exporter
spec:
  selector:
    matchLabels:
     name: node-exporter
  template:
    metadata:
      labels:
        name: node-exporter
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100
        resources:
          requests:
            cpu: 0.15
        securityContext:
          privileged: true
        args:
        - --path.procfs
        - /host/proc
        - --path.sysfs
        - /host/sys
        - --collector.filesystem.ignored-mount-points
        - '"^/(sys|proc|dev|host|etc)($|/)"'
        volumeMounts:
        - name: dev
          mountPath: /host/dev
        - name: proc
          mountPath: /host/proc
        - name: sys
          mountPath: /host/sys
        - name: rootfs
          mountPath: /rootfs
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: dev
          hostPath:
            path: /dev
        - name: sys
          hostPath:
            path: /sys
        - name: rootfs
          hostPath:
            path: /
            
            
#应用
kubectl apply -f exporter.yaml

k8s 部署 prometheus及集群层面的监控_vim_10

新增 k8s-node 监控

在 configmap-prometheus-01.yaml 中新增采集 job：k8s-nodes

node_exporter也是每个node节点都运行，因此role使用node即可，默认address端口为10250，替换为9100即可；

- job_name: k8s-nodes
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):10250'
        replacement: '${1}:9100'
        target_label: __address__
        action: replace
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels: [__meta_kubernetes_endpoints_name]
        action: replace
        target_label: endpoint
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
        
        
 #添加完成之后应用文件
kubectl apply -f configmap-prometheus-01.yaml

#手动加载prometheus服务
curl -XPOST http://prometheus.kubernets.cn/-/reload

总结

kube-state-metrics：将 Kubernetes API 中的各种对象状态信息转化为 Prometheus 可以使用的监控指标数据。
cAdvisor：用于监视容器资源使用和性能的工具，它可以收集 CPU、内存、磁盘、网络和文件系统等方面的指标数据。
node-exporter：用于监控主机指标数据的收集器，它可以收集 CPU 负载、内存使用情况、磁盘空间、网络流量等各种指标数据。

这三种工具可以协同工作，为用户提供一个全面的 Kubernetes 监控方案，帮助用户更好地了解其 Kubernetes 集群和容器化应用程序的运行情况。

原创作者: cyfang 转载于: https://blog.51cto.com/cyfang/11802784