简介
K8S从1.8版本开始，CPU、内存等资源的metrics信息可以通过 Metrics API来获取，用户可以直接获取这些metrics信息（例如通过执行kubect top命令），HPA使用这些metics信息来实现动态伸缩。本文介绍K8S集群基于metric server的HPA。在开始之前我们需要了解一下Metrics API和Metrics Server。
Metrics API：
1、通过Metrics API我们可以获取到指定node或者pod的当前资源使用情况，API本身不存储任何信息，所以我们不可能通过API来获取资源的历史使用情况。
2、Metrics API的获取路径位于：/apis/metrics.k8s.io/
3、获取Metrics API的前提条件是metrics server要在K8S集群中成功部署
4、更多的metrics资料请参考：https://github.com/kubernetes/metrics
Metrics server：
1、Metrics server是K8S集群资源使用情况的聚合器
2、从1.8版本开始，Metrics server默认可以通过kube-up.sh 脚本以deployment的方式进行部署，也可以通过yaml文件的方式进行部署
3、Metrics server收集所有node节点的metrics信息
heapster 已经被废弃了，后续版本中会使用 metrics-server代替,不过kubenetes如果是1.10之前的版本，使用Metrics Server 还是要和heapster 配合的。k8s v1.10版本之前仍然要通过heapster获取指标数据，否则即使部署了metrics-server，kubectl top这个指令仍然连接heapster
官网部署方法
git clone https://github.com/kubernetes-incubator/metrics-server
cd metrics-server
kubectl create -f deploy/1.8+/
kubectl -n kube-system get pods -l k8s-app=metrics-server
我的实际部署步骤
下载文件
[root@master01 1.8+]# ll
total 28
-rw-r--r-- 1 root root 384 Apr 9 14:03 aggregated-metrics-reader.yaml
-rw-r--r-- 1 root root 308 Apr 9 14:03 auth-delegator.yaml
-rw-r--r-- 1 root root 329 Apr 9 14:03 auth-reader.yaml
-rw-r--r-- 1 root root 298 Apr 9 14:03 metrics-apiservice.yaml
-rw-r--r-- 1 root root 1099 Apr 10 10:15 metrics-server-deployment.yaml
-rw-r--r-- 1 root root 249 Apr 9 14:03 metrics-server-service.yaml
-rw-r--r-- 1 root root 517 Apr 10 10:14 resource-reader.yaml
images
可以使用 registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1 感谢作者^^!
Dockerfile文件地址：https://github.com/minminmsn/k8s1.13/blob/master/metrics-server/Dockerfile
修改metrics-server-deployment.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1 ## 修改镜像地址
imagePullPolicy: Always
command: ##新增
- /metrics-server ##新增
- --metric-resolution=30s ##新增
- --kubelet-insecure-tls ##新增
- --kubelet-preferred-address-types=InternalIP ##新增
volumeMounts:
- name: tmp-dir
mountPath: /tmp
修改 resource-reader.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats ##新增
- namespaces ##新增
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
部署
kubectl create -f .
报错
I0109 05:55:43.708300 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
Error: cluster doesn't provide requestheader-client-ca-file
解决方法 开启聚合层，Enable apiserver flags，修改kube-apiserver配置，重启服务
[root@elasticsearch01 cfg]# tail /k8s/kubernetes/cfg/kube-apiserver
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/k8s/etcd/ssl/server-key.pem \
--requestheader-client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--requestheader-allowed-names=aggregator \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--proxy-client-cert-file=/k8s/kubernetes/ssl/kube-proxy.pem \
--proxy-client-key-file=/k8s/kubernetes/ssl/kube-proxy-key.pem"
(重点)修改kube-apiservice和kube-controller-manager.service开启聚合层
证书要准备好,我的证书（搭建k8s环境时候会生成证书，找到证书是难点。或者网上查找证书生成步骤、这里我不做多步骤啦）：
[root@master01 heapster]# ll /etc/kubernetes/ssl/*
-rw-r--r-- 1 kube kube 1415 Apr 10 15:34 /etc/kubernetes/ssl/ca.pem ##根证书
-rw------- 1 kube kube 1679 Apr 4 2018 /etc/kubernetes/ssl/kubernetes-client-proxy-key.pem ##proxy的key文件
-rw-r--r-- 1 kube kube 1562 Apr 4 2018 /etc/kubernetes/ssl/kubernetes-client-proxy.pem ##proxy的证书文件
注意证书的权限！！！ 我的是kube用户chown kube:kube /etc/kubernetes/ssl
将这三个证书分发到其他node节点 scp etc/kubernetes/ssl/* node01:/etc/kubernetes/ssl
kube-apiservice文件：
新增以下几行
--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-allowed-names= --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kubernetes-client-proxy.pem --proxy-client-key-file=/etc/kubernetes/ssl/kubernetes-client-proxy-key.pem --runtime-config=api/all=true --enable-aggregator-routing=true"
kube-controller-manager.service文件
新增一行：--horizontal-pod-autoscaler-use-rest-clients=true
systemctl daemon-reload
systemctl restart kube-apiserver.service
systemctl restart kube-controller-manager
确认启动成功后重新部署etrics-server0.3.1
cd 1.8+/
kubectl create -f .
kubectl get pod,svc -n kube-system
[root@master01 1.8+]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
metrics-server-57596fd464-ms2zp 1/1 Running 0 39m
部署heapster
kubenetes 1.10后废弃了，k8s v1.10版本之前仍然要通过heapster获取指标数据，否则即使部署了metrics-server，kubectl top这个指令仍然连接heapster
创建heapster.yaml
[root@master01 heapster]# cat heapster.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: heapster
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: heapster
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: heapster
namespace: kube-system
spec:
replicas: 1
template:
metadata:
labels:
task: monitoring
k8s-app: heapster
spec:
serviceAccountName: heapster
containers:
- name: heapster
image: harbor.reg/qijin_product/heapster-amd64:v1.4.3
imagePullPolicy: IfNotPresent
command:
- /heapster
# - --source=kubernetes:https://kubernetes.default
# - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
- --source=kubernetes:http://10.1.11.11:8080?inClusterConfig=false ##api-server 机器IP和端口
- --sink=influxdb:http://monitoring-influxdb:8086
---
apiVersion: v1
kind: Service
metadata:
labels:
task: monitoring
# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
# If you are NOT using this as an addon, you should comment out this line.
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: Heapster
name: heapster
namespace: kube-system
spec:
ports:
- port: 80
targetPort: 8082
selector:
k8s-app: heapster
如果要持久化数据还要安装influxdb,yaml文件去链接:https://pan.baidu.com/s/1WjCSZ9wQzJQ4Q0PCxXogZw 密码:6wtd下载
获取v1beta1.metrics.k8s.io并验证
[root@master01 1.8+]# kubectl get apiservice
NAME AGE
v1. 1y
v1.apps 1y
v1.authentication.k8s.io 1y
v1.authorization.k8s.io 1y
v1.autoscaling 1y
v1.batch 1y
v1.networking.k8s.io 1y
v1.rbac.authorization.k8s.io 1y
v1.storage.k8s.io 1y
v1alpha1.admissionregistration.k8s.io 1h
v1alpha1.rbac.authorization.k8s.io 1h
v1alpha1.scheduling.k8s.io 1h
v1alpha1.settings.k8s.io 1h
v1alpha1.storage.k8s.io 1h
v1beta1.admissionregistration.k8s.io 1y
v1beta1.apiextensions.k8s.io 1y
v1beta1.apps 1y
v1beta1.authentication.k8s.io 1y
v1beta1.authorization.k8s.io 1y
v1beta1.batch 1y
v1beta1.certificates.k8s.io 1y
v1beta1.events.k8s.io 1y
v1beta1.extensions 1y
v1beta1.metrics.k8s.io 40m ### 说明部署成功
v1beta1.policy 1y
v1beta1.rbac.authorization.k8s.io 1y
v1beta1.storage.k8s.io 1y
v1beta2.apps 1y
v2alpha1.batch 1h
v2beta1.autoscaling 1y
[root@master01 1.8+]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
creationTimestamp: 2019-04-10T08:40:19Z
name: v1beta1.metrics.k8s.io
resourceVersion: "41675073"
selfLink: /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io
uid: 459dc419-5b6c-11e9-bac9-005056b678c1
spec:
caBundle: null
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: 2019-04-10T08:40:24Z
message: all checks passed
reason: Passed
status: "True"
type: Available
通过kubectl工具测试获取metrics数据
yum -y install jq
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq
稍等一下查看pod的使用资源
[root@beta-k8s-master-1 ~]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
beta-k8s-node-1 131m 1% 7413Mi 46%
beta-k8s-node-2 198m 2% 11421Mi 72%
beta-k8s-node-3 219m 2% 10914Mi 69%
[root@beta-k8s-master-1 ~]# kubectl top pod
NAME CPU(cores) MEMORY(bytes)
frm-eureka-profile-a1-794995f789-fp5sd 18m 1731Mi
部署dashboard
下载地址：
链接:https://pan.baidu.com/s/18PTR648nYQQWEXBWasLamg 密码:gunr
最后上成果图
扫一扫
2万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
