How to rightsize the Kubernetes resource limits

Kubernetes resource limits are always a tricky setting to tweak, since you have to find the sweet spot between having the limits too tight or too loose.

In this article, which is a continuation of the Kubernetes capacity planning series, you’ll learn how to set the right Kubernetes resource limits: from detecting the containers without any limit, to finding the right Kubernetes resource limits you should set in your cluster.

We are assuming that you’re using Prometheus for monitoring your Kubernetes cluster. That’s why every step in this guide is illustrated with example PromQL queries.

 

Detecting containers without Kubernetes resource limits

---

 The first step to set the right Kubernetes resource limits is to detect the containers without any limits.

Containers without Kubernetes resource limits can cause very critical consequences in your nodes. In the best case, the nodes will start evicting pods in order or scoring. They also will have performance issues due to CPU throttling. In the worst-case scenario, the nodes will die of memory starvation.

没有 Kubernetes 资源限制的容器可能会在您的节点中造成非常严重的后果。在最好的情况下，节点将开始按顺序或评分驱逐 Pod。由于 CPU 节流，它们也会出现性能问题。在最坏的情况下，节点将死于内存不足。

Find the containers without Kubernetes resource limits 

---

 Containers without CPU limits by namespace

sum by (namespace,pod,container)(count by (namespace,pod,container)(kube_pod_container_info{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"}))

kube_pod_container_info    Gauge    查询所有 Container 的信息。可以通过 sum() 函数获得集群中的所有 Container 数目。

kube_pod_container_resource_limits    Gauge    查询容器的资源限制量。允许通过标签筛选，查看容器具体的资源限制量。

Containers without memory limits by namespace

sum by (namespace,pod,container)(count by (namespace,pod,container)(kube_pod_container_info{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"}))

Found too many containers without Kubernetes resource limits?

---

Maybe you found several containers without Kubernetes resource limits. Let’s focus on the most dangerous ones for now. How? Easy, just find the top 10 containers that are using more resources and have no Kubernetes resource limits.

kube_pod_container_resource_requests	Gauge	查询容器的资源需求量。允许通过标签筛选，查看容器具体的资源需求量。
kube_pod_container_resource_limits	Gauge	查询容器的资源限制量。允许通过标签筛选，查看容器具体的资源限制量。

Top 10 containers without CPU limits, using more CPU 

topk(10,sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!=""}[5m])) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"})) 

 Top 10 containers without memory limits, using more memory

topk(10,sum by (namespace,pod,container)(container_memory_usage_bytes{container!=""}) unless sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"}))

Detecting containers with very tight Kubernetes resource limits

---

Detecting containers with very tight CPU limits

If a container is close to its CPU limit and needs to perform more CPU-demanding operations than usual, it will have a degraded performance due to CPU throttling.

如果容器接近其 CPU 限制并且需要执行比平时更多的 CPU 需求操作，则由于 CPU 节流，它的性能将下降。

使用此查询查找 CPU 使用率接近其限制的容器：

Use this query to find the containers whose CPU usage is close to its limits:

(sum by (namespace,pod,container)(rate(container_cpu_usage_seconds_total{container!=""}[5m])) / sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="cpu"})) > 0.8

 Detecting containers with very tight memory limits

If a container is close to its memory limit and overpasses it, it will be killed.

如果容器接近其内存限制并超过它，它将被杀死。

 This chart shows how a container increased its memory usage by a lot until it hit the limit and died

 Use this query to find the containers whose memory usage is close to its limits:

(sum by (namespace,pod,container)(container_memory_usage_bytes{container!=""}) / sum by (namespace,pod,container)(kube_pod_container_resource_limits{resource="memory"})) > 0.8

How to set the right Kubernetes resource limits?

---

 One way to do it is to study for a while the resource usage of the containers that we want to limit. To do this, we will focus our attention on the containers of the same kind and workload (being workload a deployment, daemonset, statefulset, etc.). Here, we have two strategies:

一种方法是研究一段时间我们要限制的容器的资源使用情况。要做到这一点，我们将重点放在同一的容器我们的注意力kind和workload（即工作量deployment，daemonset，statefulset等）。在这里，我们有两种策略：

Conservative

 We will select the value of the container that consumed the most in each moment. If we set the limit to that value, the containers won’t run out of resources.

我们将选择每个时刻消耗最多的容器的值。如果我们将限制设置为该值，容器将不会耗尽资源。 

max by (namespace,owner_name,container)((rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))

Finding the right memory limit, with the conservative strategy

max by (namespace,owner_name,container)((container_memory_usage_bytes{container!="POD",container!=""}) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))

Aggressive

We will select the quantile 99 as the limit. This will leave the 1% most consuming out of the limits. This is a good strategy if there are sparse anomalies or peaks that you do not want to support.

我们将选择quantile 99作为限制。这将使消耗最大的 1% 超出限制。如果存在您不想支持的稀疏异常或峰值，这是一个很好的策略。 

Finding the right CPU limit, with the aggressive strategy

quantile by (namespace,owner_name,container)(0.99,(rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))

 Finding the right memory limit, with the aggressive strategy

quantile by (namespace,owner_name,container)(0.99,(container_memory_usage_bytes{container!="POD",container!=""}) * on(namespace,pod) group_left(owner_name) avg by (namespace,pod,owner_name)(kube_pod_owner{owner_kind=~"DaemonSet|StatefulSet|Deployment"}))

Has the cluster enough capacity?

---

In Kubernetes, nodes assure that the pods scheduled in them have enough resources based on each pod’s container requests. This also means that the node is committing to give every container the amount of CPU and memory set within their limits.

在 Kubernetes 中，节点根据每个 Pod 的容器请求确保其中调度的 Pod 具有足够的资源。这也意味着节点承诺为每个容器提供在其限制范围内设置的 CPU 和内存量。 

Talking about containers with very loose limits is the same as talking about limit overcommit. This happens when the sum of all Kubernetes resource limits is bigger than the capacity of that resource.

谈论限制非常宽松的容器与谈论限制过度使用是一样的。当所有 Kubernetes 资源限制的总和大于该资源的容量时，就会发生这种情况

When you are overcommitting resources in your cluster, everything might run perfectly in normal conditions, but in high load scenarios, the containers could start consuming CPU and memory up to their limit. This will cause the node to start doing pod eviction, and in very critical situations, the node will die, due to the starvation of the available resources in the cluster.

当您过度使用集群中的资源时，一切都可能在正常条件下完美运行，但在高负载情况下，容器可能会开始消耗 CPU 和内存，直至达到极限。这将导致节点开始执行 pod eviction，并且在非常危急的情况下，由于集群中可用资源的匮乏，节点将死亡。 

 

Finding the overcommit of the cluster

We can check the overcommit percentage of our cluster on memory and CPU with this:

% memory overcommitted of the cluster 

kube_node_status_capacity

Gauge

查询节点的全部资源总数，包括：CPU、内存、Pods 等。允许通过标签筛选，查看节点具体的资源容量。

100 * sum(kube_pod_container_resource_limits{container!="",resource="memory"} ) / sum(kube_node_status_capacity{resource="memory"})

% CPU overcommitted of the cluster

100 * sum(kube_pod_container_resource_limits{container!="",resource="cpu"} ) / sum(kube_node_status_capacity{resource="cpu"})

Normally, not all the containers will be consuming all their resources at the same time, so having an overcommit of 100% is ideal from a resource point of view. On the other hand, this will incur extra costs in infrastructure that will never be used.

通常，并非所有容器都会同时消耗所有资源，因此从资源的角度来看，100% 的过量使用是理想的。另一方面，这将导致永远不会使用的基础设施的额外成本。 

To better adjust the capacity of your cluster, you can opt for a conservative strategy, assuring that the overcommit is under 125%, or an aggressive strategy if you let the overcommit reach 150% of the capacity of your cluster.

为了更好地调整集群容量，您可以选择保守策略，确保过度使用低于 125%，或者选择激进策略，如果您让过度使用达到集群容量的 150%。

Putting it all together

---

 In this article, you learned why it is key to understand Kubernetes limits and requests, how to detect inefficiencies in your cluster, and the different strategies we could follow to set the right Kubernetes resource limits.

If you want to dig deeper, you can learn more about limits and requests in Kubernetes, or how to rightsize the requests of your cluster.