k8s学习(二十六) 使用prometheus监控常用资源对象

1、监控apiserver
apiserver 作为 Kubernetes 最核心的组件，当然他的监控也是非常有必要的，对于apiserver 的监控我们可以直接通过 kubernetes 的 Service 来获取：

[root@master prometheus]# kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   14d

修改prometheus-cm.yaml
添加endpoint类型的job，并过滤到只有apiserver服务

 - job_name: 'kubernetes-apiservers'
      kubernetes_sd_configs:
      - role: endpoints
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

[root@master prometheus]# gedit prometheus-cm.yaml 
[root@master prometheus]# kubectl delete -f prometheus-cm.yaml 
configmap "prometheus-config" deleted
[root@master prometheus]# kubectl apply -f prometheus-cm.yaml 
configmap/prometheus-config created
curl -X POST http://10.105.89.6:9090/-/reload

查看效果

2、监控service

上面的 apiserver 实际上是一种特殊的 Service，现在我们同样来配置一个任务用来专门发现普通类型的 Service。
在prometheus-cm.yaml中添加配置

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

注意我们这里在relabel_configs区域做了大量的配置，特别是第一个保留__meta_kubernetes_service_annotation_prometheus_io_scrape为true的才保留下来，这就是说要想自动发现集群中的 Service，就需要我们在 Service 的annotation区域添加prometheus.io/scrape=true的声明。

现在我们在之前创建的 redis 这个 Service 中添加上prometheus.io/scrape=true这个 annotation：

---
kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: kube-ops
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
spec:
  selector:
    app: redis
  ports:
  - name: redis
    port: 6379
    targetPort: 6379
  - name: prom
    port: 9121
    targetPort: 9121

kubectl apply –f promethues-redis.yaml
kubectl delete –f prometheus-cm.yaml
kubectl apply –f prometheus-cm.yaml
curl -X POST http://10.105.89.6:9090/-/reload

3、安装kube-state-metrics

1)下载kube-state-metrics
下载地址：https://github.com/kubernetes/kube-state-metrics/tree/release-1.8
已经下载好在”用到的包/kube-state-metrics-release-1.8.zip”

2. 下载kube-state-metrics镜像
   有网的机器上

docker pull  openshift/kube-state-metrics
docker save -o kube-state-metrics.tar openshift/kube-state-metrics

已经下载好了在”用到的包/kube-state-metrics.tar”

3. 加载kube-sate-metrics
   拷贝到主节点，

docker load -i kube-state-metrics.tar
docker tag openshift/kube-state-metrics 192.168.100.94:80/kube-state-metrics
docker push 192.168.100.94:80/kube-state-metrics

4. 解压kube-state-metrics-release-1.8.zip
   进入kubernetes目录，修改kube-state-metrics-deployment.yaml
   将image修改为192.168.100.94:80/kube-state-metrics

Kubectl aply -f .