Since Kubernetes v1.10, K8s considered to configure kubelet with configuration file, this has become a recommended approach because its simplifies node deployment and configuration management. in this article, I will describe how to use a configuration file to configure kubelet to do resource control.
how to pass a configuration file to kubelet
we need to use --config flag to set a path to a configure file to kubelet in kubelet service conf file, when system start kubelet, will pass this configuration to kubelet
when we initial Kubernetes cluster, this /var/lib/kubelet/config.yaml file will be generated by kubeadm tool with default value if we do not set any parameter for KubeletConfiguration in cluster configuration file. In below example, we give some parameters to enable resource management for K8s, all these parameters will write to config.yaml file
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
systemReserved:
cpu: 1
memory: 1Gi
ephemeral-storage: 10Gi
systemReservedCgroup: /system.slice
kubeReserved:
cpu: 1
memory: 2Gi
ephemeral-storage: 10Gi
kubeReservedCgroup: /system.slice/kubelet.service
enforceNodeAllocatable:
- pods
- kube-reserved
- system-reserved
evictionHard:
imagefs.available: 15%
memory.available: 500Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
Since we use linux CGroup(control groups) in this example to manage resource for Kubernetes, so we need to create some new groups for Kubelet under cgroup before we initial K8s cluster, you can put all these steps into kubelet service configuration file
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
(redhat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf)
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
add following:
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/pids/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/devices/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/memory/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/blkio/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/systemd/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/systemd/system.slice/docker.service
after we done above steps, then we can use below command to initize cluster:
kubeadm init --config=/etc/kubernetes/k8s-cluster-bi.yaml --upload-certs
the config.yaml will be generated:
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
cgroupsPerQOS: true
clusterDNS:
- 172.17.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable: - pods
- kube-reserved
- system-reserved
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 500Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 10
kubeAPIQPS: 5
kubeReserved:
cpu: 1000m
ephemeral-storage: 10Gi
memory: 2Gi
kubeReservedCgroup: /system.slice/kubelet.service
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 40
nodeStatusReportFrequency: 1m0s
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
systemReserved:
cpu: 1000m
ephemeral-storage: 10Gi
memory: 1Gi
systemReservedCgroup: /system.slice
volumeStatsAggPeriod: 1m0s
then you can use command to check other nodes resource allocatable number
kubectl describe node ppydalbik0101
Capacity:
cpu: 16
ephemeral-storage: 102821812Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65943252Ki
pods: 110
Allocatable:
cpu: 14
ephemeral-storage: 73285745303
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 62285524Ki
pods: 110