资源限制简介
命名空间级别:
资源配额(ResourceQuota):用来定义某个命名空间下所有资源的使用限额,包括计算资源的配额,存储资源的配额,对象数量的配额
LimitRange:基于namespace的资源管理,配置默认值,包括pod和container的最小、最大和defaultLimit、defaultRequests等
容器级别:
配置ResourceQuota后,pod和container未设置compute resources,则会报错
配置ResourceQuota后,如果配置LimitRange,pod和container未设置compute resources,则走limitRange中配置的默认值,而不会报错
Ex:
为命名空间test创建ResourceQuota
apiVersion: v1
kind: ResourceQuota
metadata:
name: test-compute-resources
namespace: test
spec:
hard:
pods: "4"
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
查看
kubectl describe quota test-compute-resources --namespace=test
Name: test-compute-resources
Namespace: myspace
Resource Used Hard
-------- ---- ----
limits.cpu 0 2
limits.memory 0 2Gi
pods 0 4
requests.cpu 0 1
requests.memory 0 1Gi
为命名空间test创建LimitRange
apiVersion: v1
kind: LimitRange
metadata:
name: mylimits
namespace: test
spec:
limits:
- max:
cpu: "2"
memory: 1Gi
min:
cpu: 200m
memory: 6Mi
type: Pod
- default:
cpu: 300m
memory: 200Mi
defaultRequest:
cpu: 200m
memory: 100Mi
max:
cpu: "2"
memory: 1Gi
min:
cpu: 100m
memory: 3Mi
type: Container
查看:
kubectl describe limits mylimits --namespace=test
Name: mylimits
Namespace: test
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Pod cpu 200m 2 - - -
Pod memory 6Mi 1Gi - - -
Container cpu 100m 2 200m 300m -
Container memory 3Mi 1Gi 100Mi 200Mi -
不同计算资源类型的限制机制
计算资源类型:
CPU是kubernetes支持的可压缩资源,当资源紧缺时,会对容器限流
内存是kubernetes支持的不可压缩资源,当资源紧缺时,会根据pod的优先级,杀死优先级低的pod。
pod的优先级:
kubernetes将容器划分为3个QoS登记:
Guaranteed(完全可靠的): 设置limit=request
Burstable(弹性波动,较可靠的): 设置limit!=request
Best-Effort(尽力而为,不太可靠的): 未设置limit和request
上述三个等级优先级递减。
操作验证
配置ResourceQuota而未配置LimitRange情况:
已正常运行的未设置limit和request的pod: 无影响;但pod删除后无法创建新的pod;
创建未限制资源的pod: 无法创建
创建限制资源容器: 满足两个条件request <= limit 及 request <= node剩余资源。 不满足前述两个条件时,无法创建
命名空间ResourceQuota配置超过node剩余资源:
ResourceQuota可以创建,但当node剩余资源不足时,虽满足ResourceQuota条件,但仍无法创建新pod。
Rbac配合ResourceQuota,实现对租户的资源限制
创建用户
创建user:testrq
testrq-csr.json
{
"CN": "testrq",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "TianJin",
"L": "TianJin",
"O": "k8s",
"OU": "System"
}
]
}
userconfig.sh
for targetName in testrq; do
cfssl gencert --ca /root/ssl/k8s-root-ca.pem --ca-key /root/ssl/k8s-root-ca-key.pem --config /root/ssl/k8s-gencert.json --profile kubernetes $targetName-csr.json | cfssljson --bare $targetName
echo "Create $targetName kubeconfig..."
kubectl config set-cluster kubernetes --certificate-authority=/root/ssl/k8s-root-ca.pem --embed-certs=true --server=https://10.0.13.158:6443 --kubeconfig=$targetName.kubeconfig
kubectl config set-credentials $targetName --client-certificate=$targetName.pem --client-key=$targetName-key.pem --embed-certs=true --kubeconfig=$targetName.kubeconfig
kubectl config set-context kubernetes --cluster=kubernetes --user=$targetName --kubeconfig=$targetName.kubeconfig
kubectl config use-context kubernetes --kubeconfig=$targetName.kubeconfig
done
执行./userconfig.sh(注:脚本中设计的根证书等与k8s集群的一致)
可看到,当前文件夹生成testrq.kubeconfig,用于之后创建k8s资源。
用户与命名空间绑定
创建role及RoleBinding
testrq-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: testrq
namespace: test
rules:
- apiGroups:
- ""
resources:
- pods
- pods/attach
- pods/exec
- pods/portforward
- pods/proxy
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- persistentvolumeclaims
- replicationcontrollers
- replicationcontrollers/scale
- services
- services/proxy
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- secrets
- serviceaccounts
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- bindings
- events
- limitranges
- namespaces/status
- pods/log
- pods/status
- replicationcontrollers/status
- resourcequotas
- resourcequotas/status
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- apps
resources:
- daemonsets
- deployments
- deployments/rollback
- deployments/scale
- replicasets
- replicasets/scale
- statefulsets
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- batch
resources:
- cronjobs
- jobs
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- extensions
resources:
- daemonsets
- deployments
- deployments/rollback
- deployments/scale
- ingresses
- replicasets
- replicasets/scale
- replicationcontrollers/scale
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
testrq-rolebinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: testrq
namespace: test
subjects:
- kind: User
name: testrq # 目标用户
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: testrq # 角色信息
apiGroup: rbac.authorization.k8s.io
创建role及roleBinding
kubectl create -f testrq-role.yaml
kubectl create -f testrq-rolebinding.yaml
以上初步完成了限制某租户的资源使用,该租户只能在指定命名空间操作,且该命名空间设置了资源限制,即resourceQuota。
租户进行操作命令示例
kubectl create -f test.yaml --kubeconfig testrq.kubeconfig
一些思考
1 对于单副本pod,为避免资源紧缺时造成pod的误删,应该在估算计算资源之后设置limit=request,保证pod优先级最高。
2 对于多副本pod,因为有多个副本,停止一个副本不会造成太大影响,因此可以设置limit>request。 但对于处于高并发状态的多副本pod,停止一个pod会暂时造成业务的响应缓慢,这种情景下,我们可以配合Horizontal Pod Autoscaling(pod弹性伸缩),配置伸缩指标小于request,这样可以保证资源的使用还未达到request使用时,副本已扩展。此时或许会认为limit配置没什么作用了,但是当pod扩展到最大个数时,limit将发挥效果,保证资源使用不超过limit限制。
3 对于一些重要性很低,重启不影响的pod,可以不设置limit和request,以便更有效地使用集群资源。