Kubernetes可视化界面及监控安装

---

声明:这是我在大学毕业后进入第一家互联网工作学习的内容

---

背景

刚刚部署完压测环境kubernertes(以下简称k8s),测试人员需要对项目进行压测，我需要组装一个监控系统帮助测试人员更好地测试个项目性能，百度了很多方案，目前选择了Kubernetes-dashboard+heapster+influxdb+grafana这种方案。

要求说明

项目	版本号	targetPort	nodePort
kubernetes	1.15
kubernetes-dashboard	v1.10.1	9090	32666
heapster-amd64	v1.5.4	8082
heapster-grafana-amd64	v5.0.4	3000	31234
heapster-influxdb-amd64	v1.5.2	8086	30444

确保nodeport在整个k8s集群内唯一，版本号k8s和kubernetes-dashboard 有一些对应关系

kubernetes-dashboard的这个版本我没有做权限控制，比较麻烦，如果需要

heapster-grafana-amd64 是为了进一步监控每个容器的cpu、内存等更为具体的使用情况，如果你只需要kubernetes-dashboard的基础监控（页面上直观看到）那么只用装heapster-influxdb-amd64和heapster-amd64

简介

Kubernetes-dashboard

Kubernetes Dashboard 是 Kubernetes 集群的基于 Web 的通用 UI。它允许用户管理在群集中运行的应用程序并对其进行故障排除，以及管理群集本身。

由于Kubernetes API版本之间的重大更改，某些功能可能无法在仪表板中正常运行，这也导致了kubernetes-dashboard的兼容性问题比较严重，最好一个版本使用一个版本最稳定的dashboard。

比如kubernetes-dashboard 新版本：v2.0.0 兼容 Kubernetes 版本：1.18以上，对以下的版本不兼容。

如果想安装高版本可以看 Kubernetes 部署 Kubernetes-Dashboard v2.0.0

而本文kubernetes-dashboard：v1.10.1 兼容 kubernetes：1.15版本

Heapster

Heapster是容器集群监控和性能分析工具，天然的支持Kubernetes和CoreOS。

Kubernetes有个出名的监控agent—cAdvisor。在每个kubernetes Node上都会运行cAdvisor，它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime)。在较新的版本中，K8S已经将cAdvisor功能集成到kubelet组件中。每个Node节点可以直接进行web访问。

Heapster是一个收集者，Heapster可以收集Node节点上的cAdvisor数据，将每个Node上的cAdvisor的数据进行汇总，还可以按照kubernetes的资源类型来集合资源，比如Pod、Namespace，可以分别获取它们的CPU、内存、网络和磁盘的metric。默认的metric数据聚合时间间隔是1分钟。还可以把数据导入到第三方工具(如InfluxDB)。

Kubernetes原生dashboard的监控图表信息来自heapster。在Horizontal Pod Autoscaling中也用到了Heapster，HPA将Heapster作为Resource Metrics API，向其获取metric。

InfluxDB

InfluxDB是一个开源的时序数据库，使用GO语言开发，特别适合用于处理和分析资源监控数据这种时序相关数据。而InfluxDB自带的各种特殊函数如求标准差，随机取样数据，统计数据变化比等，使数据统计和实时分析变得十分方便。

时序数据库产品的发明都是为了解决传统关系型数据库在时序数据存储和分析上的不足和缺陷，这类产品被统一归类为时序数据库。针对时序数据的特点对写入、存储、查询等流程进行了优化，这些优化与时序数据的特点息息相关：

• 存储成本：
  利用时间递增、维度重复、指标平滑变化的特性，合理选择编码压缩算法，提高数据压缩比；
  通过预降精度，对历史数据做聚合，节省存储空间。

• 高并发写入：
  批量写入数据，降低网络开销；
  数据先写入内存，再周期性的dump为不可变的文件存储。

• 低查询延时，高查询并发：
  优化常见的查询模式，通过索引等技术降低查询延时；
  通过缓存、routing等技术提高查询并发。

Grafana

Grafana是一个跨平台的开源的度量分析和可视化工具，可以通过将采集的数据查询然后可视化的展示，并及时通知。它主要有以下六大特点：

• 展示方式：快速灵活的客户端图表，面板插件有许多不同方式的可视化指标和日志，官方库中具有丰富的仪表盘插件，比如热图、折线图、图表等多种展示方式；

• 数据源：Graphite，InfluxDB，OpenTSDB，Prometheus，Elasticsearch，CloudWatch和KairosDB等；

• 通知提醒：以可视方式定义最重要指标的警报规则，Grafana将不断计算并发送通知，在数据达到阈值时通过Slack、PagerDuty等获得通知；

• 混合展示：在同一图表中混合使用不同的数据源，可以基于每个查询指定数据源，甚至自定义数据源；

• 注释：使用来自不同数据源的丰富事件注释图表，将鼠标悬停在事件上会显示完整的事件元数据和标记；

• 过滤器：Ad-hoc过滤器允许动态创建新的键/值过滤器，这些过滤器会自动应用于使用该数据源的所有查询。

部署

下载安装

项目文件信息链接

一共有5个yaml文件及2个json文件

[root@vm-26-11 ~]# tree
├── kubernetes-dashboard
│   ├── grafana.yaml
│   ├── heapster-rbac.yaml
│   ├── heapster.yaml
│   ├── kubernetes-dashboard.yaml
│   └── influxdb.yaml

确认kubernetes-dashboard目录的yaml文件正确（json先不用管）

[root@vm-26-11 ~]# kubectl create -f kubernetes-dashboard

kubernetes-dashboard访问地址：

http://nodeip:32666/

配置grafana

grafana访问地址：http://nodeip:31234/

选择添加 Dashboard模板（上面下载的json文件）

kubernetes-node-statistics_rev1.json

kubernetes-pod-statistics_rev1.json

问题排查

下面有几个问题，我记录下当时的排坑操作

kubernetes-dashboard

大视报没有开权限控制，所以只用保证镜像号在国内可以找到就行，这里我用的事阿里云的镜像号
registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1

heapster

• 权限控制

E0409 06:05:27.448890 1 reflector.go:190] k8s.io/heapster/metrics/util/util.go:30: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list nodes at the cluster scope

报错原因是无权访问，这个需要创建heapster-rbac.yaml就行

• 端口不通

E0302 06:11:05.004391       1 manager.go:101] Error in scraping containers from kubelet:172.16.8.12:10255: failed to get all container stats from Kubelet URL 

"http://172.16.8.12:10255/stats/container/": Post http://172.16.8.12:10255/stats/container/: dial tcp 172.16.8.12:10255: getsockopt: connection refused

报错原因是10255端口不通（k8s默认使用10250作为kubelet端口）
需要将heapster的
- --source=kubernetes:https://kubernetes.default
改为
- --source=kubernetes:kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true

grafana

导入2个json文件后可以看到node、namespace的整体资源，但是看不了pod的资源，大视报显示的namespace的pod资源都是none。

我觉得很奇怪，但是又不太懂这个监控组件。
找了很久，在kubernetes-pod-statistics_rev1.json里找到了

        "query": "SHOW TAG VALUES FROM \"uptime\" WITH KEY=pod_name WHERE \"namespace_name\"='$ns'",
        "refresh": 1,
        "regex": "/^(.*)-\\d{9,}-\\w{5,}$/",

虽然我不知道作者为什么把pod的查找结果还给过滤一道（真心不懂）
但是把"regex": "/^(.*)-\d{9,}-\w{5,}$/"改成 “regex”: “” 保存，就可以看到pod的资源监控情况了。

总结

我花了接近一天的时间才慢慢把坑排完，看国内文章大部分都是转载，纯水文，关键时候还是看各种官方文档才能找出问题。不过最后还是按时完成任务，完美！

附件

如果下载很慢，可以直接粘贴下面的yaml文件。

kubernetes-dashboard.yaml

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.8.
#
# Example usage: kubectl create -f <this_file>

---
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    # Allows editing resource and makes sure it is created first.
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kubernetes-dashboard-certs
  namespace: kube-system
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
    # Allows editing resource and makes sure it is created first.
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kubernetes-dashboard-key-holder
  namespace: kube-system
type: Opaque
---
# ------------------- Dashboard Service Account ------------------- #

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-admin
  namespace: kube-system

---
# ------------------- Dashboard Role & Role Binding ------------------- #
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-admin
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard-admin
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: cluster-watcher
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - 'get'
  - 'list'
- nonResourceURLs:
  - '*'
  verbs:
  - 'get'
  - 'list'
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
  verbs: ["get", "update", "delete"]
  # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["kubernetes-dashboard-settings"]
  verbs: ["get", "update"]
  # Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
  resources: ["services"]
  resourceNames: ["heapster"]
  verbs: ["proxy"]
---
# ------------------- Dashboard Deployment ------------------- #
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      priorityClassName: system-cluster-critical
      containers:
      - name: kubernetes-dashboard
       # image: registry.cn-beijing.aliyuncs.com/kubernetes-cn/kubernetes-dashboard-amd64:v1.10.1
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.10.1
        resources:
          #limits:
          #  cpu: 100m
          #  memory: 300Mi
          requests:
            cpu: 50m
            memory: 200Mi
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          #- --auto-generate-certificates
        volumeMounts:
        - name: kubernetes-dashboard-certs
          mountPath: /certs
        - name: tmp-volume
          mountPath: /tmp
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      volumes:
      - name: kubernetes-dashboard-certs
        secret:
          secretName: kubernetes-dashboard-certs
      - name: tmp-volume
        emptyDir: {}
      serviceAccountName: kubernetes-dashboard-admin
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

---
# ------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  ports:
    - port: 443
      targetPort: 9090
  selector:
    k8s-app: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-external
  namespace: kube-system
spec:
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 32666
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard

influxdb.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-influxdb
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: influxdb
    spec:
      containers:
      - name: influxdb
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-influxdb-amd64:v1.5.2 
        volumeMounts:
        - mountPath: /data
          name: influxdb-storage
      volumes:
      - name: influxdb-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-influxdb
  name: monitoring-influxdb
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 8086
    targetPort: 8086
    nodePort: 30444
    name: http
  selector:
    k8s-app: influxdb

heapster.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true
        - --sink=influxdb:http://monitoring-influxdb:8086
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

heapster-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
subjects:
  - kind: ServiceAccount
    name: heapster
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

grafana.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: monitoring-grafana
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: grafana
    spec:
      containers:
      - name: grafana
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-grafana-amd64:v5.0.4
        ports:
        - containerPort: 3000
          protocol: TCP
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: ca-certificates
          readOnly: true
        - mountPath: /var
          name: grafana-storage
        env:
        - name: INFLUXDB_HOST
          value: monitoring-influxdb
        - name: GF_SERVER_HTTP_PORT
          value: "3000"
          # The following env variables are required to make Grafana accessible via
          # the kubernetes api-server proxy. On production clusters, we recommend
          # removing these env variables, setup auth for grafana, and expose the grafana
          # service using a LoadBalancer or a public IP.
        - name: GF_AUTH_BASIC_ENABLED
          value: "false"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          value: Admin
        - name: GF_SERVER_ROOT_URL
          # If you're only using the API Server proxy, set this value instead:
          # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
          value: /
      volumes:
      - name: ca-certificates
        hostPath:
          path: /etc/ssl/certs
      - name: grafana-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-grafana
  name: monitoring-grafana
  namespace: kube-system
spec:
  # In a production setup, we recommend accessing Grafana through an external Loadbalancer
  # or through a public IP.
  # type: LoadBalancer
  # You could also use NodePort to expose the service at a randomly-generated port
  type: NodePort
  ports:
  - port: 80
    targetPort: 3000
    nodePort: 31234
  selector:
    k8s-app: grafana

参考资料

Heapster-Github

在具有InfluxDB后端和Grafana UI的Kubernetes集群中运行Heapster

Grafana的Dashboards模板

kubernetes监控方案之：heapster+influxdb+grafana详解

时序数据库介绍和使用