背景:
在生产环境中,linux操纵系统使用 df -h 命令后卡住不动,
df -h卡住的原因分析,是k8s集群的计算节点服务器,并且是使用nfs存储创建pvc存储的,
问题处理:
解决方法
1.使用如下strace命令追踪,查看 df-h 是在哪里卡住了,
先安装strace命令:
# yum install strace -y
再执行追踪命令
# strace df -h
执行结果出现如下:
有提示如下的内容,表示存在挂载异常的情况
stat("/var/lib/kubelet/pods/a09f830b-7598-4e02-9208-6cde8dc5a8ee/volumes/kubernetes.io~nfs/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93")
2.使用如下命令查看此目录的详细信息
# mount | grep "/var/lib/kubelet/pods/a09f830b-7598-4e02-9208-6cde8dc5a8ee/volumes/kubernetes.io~nfs/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93"
3.使用如下命令,将nfs闲时自动卸载
因为多次执行strace df -h,所有查询到多个结果了
# umount -l /var/lib/kubelet/pods/9e4779e5-6845-4eb9-8ec0-72f029947e65/volume-subpaths/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93/sh-gw-farmwater/0
# umount -l /var/lib/kubelet/pods/a09f830b-7598-4e02-9208-6cde8dc5a8ee/volumes/kubernetes.io~nfs/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93
# umount -l /var/lib/kubelet/pods/a09f830b-7598-4e02-9208-6cde8dc5a8ee/volume-subpaths/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93/sh-gw-jianghe/2
# umount -l /var/lib/kubelet/pods/6cc5f2fc-33b4-457f-b535-493d4b5b169f/volumes/kubernetes.io~nfs/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93
# umount -l /var/lib/kubelet/pods/6cc5f2fc-33b4-457f-b535-493d4b5b169f/volume-subpaths/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93/sh-gw-iot/2
# umount -l /var/lib/kubelet/pods/31258a9f-6136-43db-aa6d-12a03d959d5e/volumes/kubernetes.io~nfs/pvc-e51838d2-b1e5-4e10-8b3a-f7cbadad7f41
# umount -l /var/lib/kubelet/pods/86799b3b-6432-49c4-88c3-44e3903bc3f7/volumes/kubernetes.io~nfs/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93
# umount -l /var/lib/kubelet/pods/86799b3b-6432-49c4-88c3-44e3903bc3f7/volume-subpaths/pvc-958f3dfa-f5a9-404f-a9ce-3ac08bdc4a93/dfm-microsvr/0
4、没有任何问题的情况strace df -h执行的结果如下:
..................
s/kubernetes.io~secret/default-token-g4fq2
) = 164
close(1) = 0
munmap(0x7f23f7d65000, 4096) = 0
close(2) = 0
exit_group(0) = ?
+++ exited with 0 +++