linux磁盘假满(文件删除后没有被释放)解决

背景

今天，我们 df -h 查询磁盘使用情况，发现 use竟然达到了99%（下表为降下来之后），

[root@ecs-4c8g /]# df -h
Filesystem                                              Size  Used Avail Use% Mounted on
/dev/vda1                                                40G   12G   26G  32% /
devtmpfs                                                3.9G     0  3.9G   0% /dev
tmpfs                                                   3.9G     0  3.9G   0% /dev/shm
tmpfs                                                   3.9G  185M  3.7G   5% /run
tmpfs                                                   3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs                                                   783M     0  783M   0% /run/user/0

然后通过 du -sh *查看根目录下各个文件夹实际占用情况，发现所有加起来不过12G，而磁盘足有40G。

[root@ecs-4c8g /]# du -sh *
0	bin
133M	boot
200M	CloudResetPwdUpdateAgent
2.2M	CloudrResetPwdAgent
0	dev
38M	etc
6.2G	home
0	lib
0	lib64
16K	lost+found
4.0K	media
4.0K	mnt
4.0K	opt
du: cannot access ‘proc/14288/task/14288/fd/3’: No such file or directory
du: cannot access ‘proc/14288/task/14288/fdinfo/3’: No such file or directory
du: cannot access ‘proc/14288/fd/3’: No such file or directory
du: cannot access ‘proc/14288/fdinfo/3’: No such file or directory
0	proc
312K	root
185M	run
0	sbin
2.1G	shareZone
4.0K	srv
0	sys
484K	tmp
4.9G	usr
320M	var

lsof解决

1. 解决办法用 lsof | grep deleted命令查看是否还有进程在使用那些删了的文件

[root@ecs-4c8g /]# lsof | grep deleted 
mysqld_sa  9416          root    2u      CHR              136,5       0t0          8 /dev/pts/5 (deleted)
java      13482            es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482            es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13484      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13484      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13485      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13485      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13486      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13486      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13487      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13487      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13488      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13488      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13489      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13489      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13490      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13490      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13491      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13491      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13492      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13492      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13493      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13493      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13494      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13494      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13495      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13495      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)
java      13482 13496      es  173r     FIFO              253,1       0t0     525070 /tmp/elasticsearch.djB6Ww49/controller_log_13482 (deleted)
java      13482 13496      es  174w     FIFO              253,1       0t0     525077 /tmp/elasticsearch.djB6Ww49/controller_command_13482 (deleted)

2. 找到占用较大的进程，kill掉相关进程

[root@ecs-4c8g /]# kill -9 13482

3. df -h 查看效果

原理

linux里的文件被删除后，空间没有被释放是因为在Linux系统中，通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用)，那么进程将仍然可以读取该文件，磁盘空间也一直被占用。
状态为deleted为标记被删除，其实该文件并没有从磁盘中删除，类似windows下的回收站状态。
据称当有其他进程打开某文件时文件被删除，就会将该文件标记为deleted，并删除其目录节点。使用du查看时，因为没有该删除状态文件的节点信息，所以就不做统计，从而导致与df的结果不一致。
若要将deleted状态文件删除，则根据pid直接kill调相应进程即可。