目录
1.资源的限制类型
Kubernetes采用request和limit两种限制类型来对资源进行分配。
• request(资源需求):即运行Pod的节点必须满足运行Pod的最基本需求才能 运行Pod。
• limit(资源限额):即运行Pod期间,可能内存使用量会增加,那最多能使用多少内存,这就是资源限额。
资源类型:
• CPU 的单位是核心数,内存的单位是字节。
• 一个容器申请0.5个CPU,就相当于申请1个CPU的一半,你也可以加个后缀 m 表示千分之一的概念。比如说100m的CPU,100豪的CPU和0.1个CPU都是一样的。
内存单位:
• K、M、G、T、P、E #通常是以1000为换算标准的。
• Ki、Mi、Gi、Ti、Pi、Ei #通常是以1024为换算标准的。
2. 内存限制
[kubeadm@server1 ~]$ mkdir quota
[kubeadm@server1 ~]$ cd quota/
[kubeadm@server1 quota]$ vim demo.yml
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: stress
args:
- --vm
- "1"
- --vm-bytes
- 200M
resources:
requests:
memory: 50Mi
limits:
memory: 100Mi
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
memory-demo 0/1 ContainerCreating 0 5s
nfs-client-provisioner-96649cd96-pntbp 1/1 Running 1 26h
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
memory-demo 0/1 OOMKilled 2 33s
nfs-client-provisioner-96649cd96-pntbp 1/1 Running 1 26h
[kubeadm@server1 quota]$ kubectl describe pod memory-demo
Name: memory-demo
Namespace: default
Priority: 0
Node: server3/172.25.1.3
Start Time: Sat, 04 Jul 2020 22:43:14 +0800
Labels: <none>
Annotations: Status: Running
IP: 10.244.2.137
IPs:
IP: 10.244.2.137
Containers:
memory-demo:
Container ID: docker://d8745253bbf2a986d4a8bf7e1afd5aed1609e42601d4801d57feccb5424bcb21
Image: stress
Image ID: docker-pullable://stress@sha256:48a71454d405dbe1c756dd728cadeb577f429f61313ac62b413b52fbaa8a3b44
Port: <none>
Host Port: <none>
Args:
--vm
1
--vm-bytes
200M
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 1
Started: Sat, 04 Jul 2020 22:44:51 +0800
Finished: Sat, 04 Jul 2020 22:44:51 +0800
Ready: False
Restart Count: 4
Limits:
memory: 100Mi
Requests:
memory: 50Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5qqxc (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-5qqxc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5qqxc
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 104s default-scheduler Successfully assigned default/memory-demo to server3
Normal Created 51s (x4 over 96s) kubelet, server3 Created container memory-demo
Normal Started 51s (x4 over 96s) kubelet, server3 Started container memory-demo
Warning BackOff 22s (x7 over 94s) kubelet, server3 Back-off restarting failed container
Normal Pulling 7s (x5 over 103s) kubelet, server3 Pulling image "stress"
Normal Pulled 7s (x5 over 96s) kubelet, server3 Successfully pulled image "stress"
[kubeadm@server1 quota]$ kubectl delete -f demo.yml
pod "memory-demo" deleted
3. CPU限制
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: stress
args:
- -c
- "2"
resources:
requests:
cpu: 5
limits:
cpu: 10
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
memory-demo 0/1 Pending 0 3s
nfs-client-provisioner-96649cd96-zprsg 0/1 ContainerCreating 0 10h
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
memory-demo 0/1 Pending 0 7s
nfs-client-provisioner-96649cd96-zprsg 0/1 ContainerCreating 0 10h
[kubeadm@server1 quota]$ kubectl describe pod memory-demo
Name: memory-demo
Namespace: default
Priority: 0
Node: <none>
Labels: <none>
Annotations: Status: Pending
IP:
IPs: <none>
Containers:
memory-demo:
Image: stress
Port: <none>
Host Port: <none>
Args:
-c
2
Limits:
cpu: 10
Requests:
cpu: 5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5qqxc (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-5qqxc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5qqxc
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 16s (x2 over 16s) default-scheduler 0/3 nodes are available: 3 Insufficient cpu.
调度失败是因为申请的CPU资源超出集群节点所能提供的资源 但CPU 使用率过高,不会被杀死
[kubeadm@server1 quota]$ vim demo.yml
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: stress
args:
- -c
- "1"
resources:
requests:
memory: 50Mi
cpu: 1
limits:
memory: 300Mi
cpu: 10
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
memory-demo 1/1 Running 0 3s
[kubeadm@server1 quota]$ kubectl describe pod memory-demo
Name: memory-demo
Namespace: default
Priority: 0
Node: server3/172.25.1.3
Start Time: Sat, 04 Jul 2020 22:47:39 +0800
Labels: <none>
Annotations: Status: Running
IP: 10.244.2.138
IPs:
IP: 10.244.2.138
Containers:
memory-demo:
Container ID: docker://7487935a3be6f3b577786c33dc924e34b9f65644ac184886592bceeb5b6156d6
Image: stress
Image ID: docker-pullable://stress@sha256:48a71454d405dbe1c756dd728cadeb577f429f61313ac62b413b52fbaa8a3b44
Port: <none>
Host Port: <none>
Args:
-c
1
State: Running
Started: Sat, 04 Jul 2020 22:47:40 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 10
memory: 300Mi
Requests:
cpu: 1
memory: 50Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5qqxc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-5qqxc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5qqxc
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18s default-scheduler Successfully assigned default/memory-demo to server3
Normal Pulling 17s kubelet, server3 Pulling image "stress"
Normal Pulled 17s kubelet, server3 Successfully pulled image "stress"
Normal Created 17s kubelet, server3 Created container memory-demo
Normal Started 17s kubelet, server3 Started container memory-demo
4. 为namespace设置资源限制
[kubeadm@server1 quota]$ vim limits.yml
[kubeadm@server1 quota]$ cat limits.yml
apiVersion: v1
kind: LimitRange
metadata:
name: limitrange-memory
spec:
limits:
- default:
cpu: 0.5
memory: 512Mi
defaultRequest:
cpu: 0.1
memory: 256Mi
max:
cpu: 1
memory: 1Gi
min:
cpu: 0.1
memory: 100Mi
type: Container
[kubeadm@server1 quota]$ kubectl apply -f limits.yml
limitrange/limitrange-memory created
[kubeadm@server1 quota]$ kubectl get limitranges
NAME CREATED AT
limitrange-memory 2020-07-04T14:51:09Z
[kubeadm@server1 quota]$ kubectl describe limitranges limitrange-memory
Name: limitrange-memory
Namespace: default
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu 100m 1 100m 500m -
Container memory 100Mi 1Gi 256Mi 512Mi -
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
Error from server (Forbidden): error when creating "demo.yml": pods "memory-demo" is forbidden: [minimum memory usage per Container is 100Mi, but request is 50Mi, maximum cpu usage per Container is 1, but limit is 10]
[kubeadm@server1 quota]$ vim demo.yml
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: myapp:v1
resources:
requests:
memory: 100Mi
cpu: 0.2
limits:
memory: 300Mi
cpu: 1
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl delete -f demo.yml
pod "memory-demo" deleted
==注意:LimitRange 在 namespace 中施加的最小和最大内存限制只有在创建和更新 Pod 时才会被应用。改变 LimitRange 不会对之前创建的 Pod 造成影响。 ==
[kubeadm@server1 quota]$ vim demo.yml
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: myapp:v1
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl describe pod demo.yml
Error from server (NotFound): pods "demo.yml" not found
[kubeadm@server1 quota]$ kubectl describe pod memory-demo
Name: memory-demo
Namespace: default
Priority: 0
Node: server3/172.25.1.3
Start Time: Sat, 04 Jul 2020 22:55:49 +0800
Labels: <none>
Annotations: kubernetes.io/limit-ranger:
LimitRanger plugin set: cpu, memory request for container memory-demo; cpu, memory limit for container memory-demo
Status: Running
IP: 10.244.2.140
IPs:
IP: 10.244.2.140
Containers:
memory-demo:
Container ID: docker://70632378717f635ef59e376befa1d5925ac86c654b1feacc8ecea4ed25fe3428
Image: myapp:v1
Image ID: docker-pullable://myapp@sha256:9eeca44ba2d410e54fccc54cbe9c021802aa8b9836a0bcf3d3229354e4c8870e
Port: <none>
Host Port: <none>
State: Running
Started: Sat, 04 Jul 2020 22:55:50 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 512Mi
Requests:
cpu: 100m
memory: 256Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-5qqxc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-5qqxc:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-5qqxc
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19s default-scheduler Successfully assigned default/memory-demo to server3
Normal Pulled 18s kubelet, server3 Container image "myapp:v1" already present on machine
Normal Created 18s kubelet, server3 Created container memory-demo
Normal Started 18s kubelet, server3 Started container memory-demo
- 1
5. 为namespace设置资源配额
参考官网:https://kubernetes.io/zh/docs/concepts/policy/resource-quotas/
当多个用户或团队共享具有固定节点数目的集群时,人们会担心有人使用超过其基于公平原则所分配到的资源量。资源配额是帮助管理员解决这一问题的工具。
资源配额,通过 ResourceQuota 对象来定义,对每个命名空间的资源消耗总量提供限制。 它可以限制命名空间中某种类型的对象的总数目上限,也可以限制命令空间中的 Pod 可以使用的计算资源的总上限。
资源配额的工作方式如下:
不同的团队可以在不同的命名空间下工作,目前这是非约束性的,在未来的版本中可能会通过 ACL (Access Control List 访问控制列表) 来实现强制性约束。
集群管理员可以为每个命名空间创建一个或多个资源配额对象。
当用户在命名空间下创建资源(如 Pod、Service 等)时,Kubernetes 的配额系统会跟踪集群的资源使用情况,以确保使用的资源用量不超过资源配额中定义的硬性资源限额。
如果资源创建或者更新请求违反了配额约束,那么该请求会报错(HTTP 403 FORBIDDEN),并在消息中给出有可能违反的约束。
如果命名空间下的计算资源 (如 cpu 和 memory)的配额被启用,则用户必须为这些资源设定请求值(request)和约束值(limit),否则配额系统将拒绝 Pod 的创建。
提示: 可使用 LimitRanger 准入控制器来为没有设置计算资源需求的 Pod 设置默认值。更多示例可以参考https://kubernetes.io/docs/tasks/administer-cluster/quota-api-object/
[kubeadm@server1 quota]$ kubectl get limitranges
NAME CREATED AT
limitrange-memory 2020-07-06T22:15:14Z
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: myapp:v1
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
memory-demo 1/1 Running 0 9s
[kubeadm@server1 quota]$ cat quota.yml
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
pods: "2"
[kubeadm@server1 quota]$ kubectl apply -f quota.yml
resourcequota/mem-cpu-demo created
[kubeadm@server1 quota]$ kubectl describe resourcequotas
Name: mem-cpu-demo
Namespace: default
Resource Used Hard
-------- ---- ----
limits.cpu 500m 2
limits.memory 512Mi 2Gi
pods 1 2
requests.cpu 100m 1
requests.memory 256Mi 1Gi
创建的ResourceQuota对象将在default名字空间中添加以下限制:
• 每个容器必须设置内存请求(memory request),内存限额(memory limit),cpu请求(cpu request)和cpu限额(cpu limit)
[kubeadm@server1 quota]$ kubectl delete -f limits.yml
limitrange "limitrange-memory" deleted
[kubeadm@server1 quota]$ kubectl delete -f demo.yml
pod "memory-demo" deleted
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
Error from server (Forbidden): error when creating "demo.yml": pods "memory-demo" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
[kubeadm@server1 quota]$ kubectl describe resourcequotas
Name: mem-cpu-demo
Namespace: default
Resource Used Hard
-------- ---- ----
limits.cpu 0 2
limits.memory 0 2Gi
pods 0 2
requests.cpu 0 1
requests.memory 0 1Gi
[kubeadm@server1 quota]$ vim demo.yml
[kubeadm@server1 quota]$ cat demo.yml
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo
image: myapp:v1
resources:
requests:
memory: 100Mi
cpu: 0.2
limits:
memory: 300Mi
cpu: 1
[kubeadm@server1 quota]$ kubectl apply -f demo.yml
pod/memory-demo created
[kubeadm@server1 quota]$ kubectl describe resourcequotas
Name: mem-cpu-demo
Namespace: default
Resource Used Hard
-------- ---- ----
limits.cpu 1 2
limits.memory 300Mi 2Gi
requests.cpu 200m 1
requests.memory 100Mi 1Gi
为 Namespace 配置Pod配额:
[kubeadm@server1 quota]$ vim quota.yml
[kubeadm@server1 quota]$ cat quota.yml
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
pods: "2"
[kubeadm@server1 quota]$ kubectl apply -f quota.yml
resourcequota/mem-cpu-demo configured
[kubeadm@server1 quota]$ kubectl describe resourcequotas
Name: mem-cpu-demo
Namespace: default
Resource Used Hard
-------- ---- ----
limits.cpu 1 2
limits.memory 300Mi 2Gi
pods 1 2
requests.cpu 200m 1
requests.memory 100Mi 1Gi
[kubeadm@server1 quota]$ kubectl apply -f limits.yml
limitrange/limitrange-memory created
[kubeadm@server1 quota]$ kubectl get limitranges
NAME CREATED AT
limitrange-memory 2020-07-06T22:34:34Z
[kubeadm@server1 quota]$ kubectl run demo-1 --image=myapp:v1
pod/demo-1 created
[kubeadm@server1 quota]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
demo-1 1/1 Running 0 4s
memory-demo 1/1 Running 0 7m26s
[kubeadm@server1 quota]$ kubectl run demo-2 --image=myapp:v1
Error from server (Forbidden): pods "demo-2" is forbidden: exceeded quota: mem-cpu-demo, requested: pods=1, used: pods=2, limited: pods=2
[kubeadm@server1 quota]$ kubectl delete pod demo-1
pod "demo-1" deleted
[kubeadm@server1 quota]$ kubectl delete -f .
pod "memory-demo" deleted
limitrange "limitrange-memory" deleted
resourcequota "mem-cpu-demo" deleted