kubernetes node DiskPressure 压力

最新推荐文章于 2024-06-25 10:16:48 发布

qinzhao168

最新推荐文章于 2024-06-25 10:16:48 发布

阅读量1.7w

点赞数 2

分类专栏： kubetnetes 文章标签： kubernetes node diskPressure

本文链接：https://blog.csdn.net/qq_21816375/article/details/82905660

版权

kubetnetes 专栏收录该内容

106 篇文章 8 订阅

订阅专栏

问题描述：
k8s slave节点没有启动任何容器

[root@dev-slave-107 log]# docker ps 
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@dev-slave-107 log]# docker ps 
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

查看磁盘使用量

[root@dev-slave-107 ~]# df -h 
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G   42G  4.7G  91% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  637M  7.2G   9% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           770M     0  770M   0% /run/user/0

磁盘使用量达到了 91%

kubelet 上报了DiskPressure 状态为true

 - lastHeartbeatTime: 2018-09-30T01:41:32Z
    lastTransitionTime: 2018-09-30T01:34:01Z
    message: kubelet has disk pressure
    reason: KubeletHasDiskPressure
    status: "True"
    type: DiskPressure

查看哪个目录使用磁盘的大小du -sh ./*

进到某个目录使用

[root@dev-slave-107 log]# du -sh .
38G	.

发现日志message的日志多

[root@dev-slave-107 log]# ll -h 
total 38G
-rw-r--r-- 1 root root 9.4K Sep 29 17:46 boot.log
-rw------- 1 root utmp    0 Sep  1 03:26 btmp
-rw------- 1 root utmp    0 Aug  1 03:50 btmp-20180901
drwxr-xr-x 2 root root  12K Sep 29 17:17 containers
-rw------- 1 root root  17M Sep 30 09:51 cron
-rw------- 1 root root  74K Sep 30 03:06 dhclient
-rw-r--r-- 1 root root  39K Sep 29 17:46 dmesg
-rw-r--r-- 1 root root  39K Sep 29 15:42 dmesg.old
drwxr-xr-x 2 root root 4.0K Sep 29 17:12 filebeat
drwxr-xr-x 2 root root 4.0K Jun  7 21:11 filelog
-rw-r--r-- 1 root root 560K Sep 29 17:46 firewall
-rw------- 1 root root 183M Sep 30 09:51 maillog
-rw------- 1 root root  37G Sep 30 09:51 messages
-rw------- 1 root root 695K Sep 30 03:06 NetworkManager
drwxr-xr-x 2 root root 4.0K Sep 29 17:17 pods
drwxr-x--- 2 root root  20K Sep 29 03:55 qingcloud-flex-volume
-rw------- 1 root root  53K Sep 30 09:33 secure
-rw-r--r-- 1 root root  120 Sep 29 17:46 wpa_supplicant.log
-rw-rw-r-- 1 root utmp  49K Sep 30 09:31 wtmp
You have mail in /var/spool/mail/root

解决办法

[root@dev-slave-107 log]# echo "" >messages

查看磁盘使用量

[root@dev-slave-107 log]# df -h 
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G  5.0G   42G  11% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  637M  7.2G   9% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           770M     0  770M   0% /run/user/0

重启docker kubelet

systemctl restart docker kubelet

问题解决

root@dev-slave-107 log]# docker ps 
CONTAINER ID        IMAGE                                                                                                     COMMAND                  CREATED             STATUS              PORTS               NAMES
6283e10f3465        2ae65d218c78                                                                                              "/bin/bash /daemon.sh"   7 seconds ago       Up 6 seconds                            k8s_stateful-pod_stateful-new-pod-0_lijiaob_251a07dd-c3d6-11e8-98ff-52549da43ad9_0
1008fba8085d        harbor.enncloud.cn/paas/pause-amd64:3.1                                                                   "/pause"                 8 seconds ago       Up 7 seconds                            k8s_POD_stateful-new-pod-0_lijiaob_251a07dd-c3d6-11e8-98ff-52549da43ad9_0
925aef84f293        20282b8475a8                                                                                              "/install-cni.sh"        9 seconds ago       Up 7 seconds                            k8s_install-cni_calico-node-gd62v_kube-system_911a77c1-c453-11e8-98ff-52549da43ad9_0
b318f9048e50        c75dc27cbfd7                                                                                              "/blockplugin --nodei"   10 seconds ago      Up 8 seconds                            k8s_csi-qingcloud_csi-qingcloud-node-2vp78_kube-system_90bfc29c-c453-11e8-98ff-52549da43ad9_0
99c84abbeba3        0c4dc3ee8c07                                                                                              "/proxy --cluster-cid"   12 seconds ago      Up 9 seconds                            k8s_kube-proxy_kube-proxy-zzcgg_kube-system_90810274-c453-11e8-98ff-52549da43ad9_0