今天遇到根目录磁盘空间占用100%,具体如下所示:
root@localhost:~$ df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 37G 35G 0 100% /
devtmpfs 63G 0 63G 0% /dev
tmpfs 63G 24K 63G 1% /dev/shm
tmpfs 63G 3.5G 60G 6% /run
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/sdb1 3.7T 354G 3.3T 10% /home
/dev/sda6 29G 2.6G 26G 10% /var/log
/dev/sda5 38G 33M 38G 1% /tmp
/dev/sda2 922M 143M 716M 17% /boot
tmpfs 13G 0 13G 0% /run/user/452
tmpfs 13G 0 13G 0% /run/user/0
tmpfs 13G 0 13G 0% /run/user/669
root@localhost:~$
通过du -h --max-depth=1 进行查看,发现每一个目录大小都不是很大,不应该报警。然后仔细查看size是37G,Used是35G,这两者相差2G,却报使用率是100%。猜测是其他进程占用了文件,但是文件并没有回收。
下面就要通过lsof命令进行查看,过滤已经被删除的文件:
root@localhost:~$ lsof | grep delete > a.txt
root@localhost:~$ vi a.txt
proftpd 103373 nobody 7r REG 8,3 1012 532255 /etc/group+ (deleted)
mysqld_sa 162952 root 2u CHR 136,0 0t0 3 /dev/pts/0 (deleted)
mysqld 163104 mysql 4u REG 8,5 0 201 /tmp/ibmoE7Gw (deleted)
mysqld 163104 mysql 5u REG 8,5 0 202 /tmp/ibETXoYu (deleted)
mysqld 163104 mysql 6u REG 8,5 0 203 /tmp/ibGcAGft (deleted)
mysqld 163104 mysql 7u REG 8,5 0 204 /tmp/ibkxiaQp (deleted)
mysqld 163104 mysql 11u REG 8,5 0 205 /tmp/ibYzcwbo (deleted)
mysqld 163104 mysql 67w REG 8,3 27084592587 1584726 /usr/local/mysql/data/influxdb.log (deleted)
mysqld 163104 86667 mysql 4u REG 8,5 0 201 /tmp/ibmoE7Gw (deleted)
mysqld 163104 86667 mysql 5u REG 8,5 0 202 /tmp/ibETXoYu (deleted)
mysqld 163104 86667 mysql 6u REG 8,5 0 203 /tmp/ibGcAGft (deleted)
mysqld 163104 86667 mysql 7u REG 8,5 0 204 /tmp/ibkxiaQp (deleted)
mysqld 163104 86667 mysql 11u REG 8,5 0 205 /tmp/ibYzcwbo (deleted)
mysqld 163104 86667 mysql 67w REG 8,3 27084592587 1584726 /usr/local/mysql/data/influxdb.log (deleted)
mysqld 163104 86844 mysql 4u REG 8,5 0 201 /tmp/ibmoE7Gw (deleted)
mysqld 163104 86844 mysql 5u REG 8,5 0 202 /tmp/ibETXoYu (deleted)
mysqld 163104 86844 mysql 6u REG 8,5 0 203 /tmp/ibGcAGft (deleted)
mysqld 163104 86844 mysql 7u REG 8,5 0 204 /tmp/ibkxiaQp (deleted)
mysqld 163104 86844 mysql 11u REG 8,5 0 205 /tmp/ibYzcwbo (deleted)
mysqld 163104 86844 mysql 67w REG 8,3 27084592587 1584726 /usr/local/mysql/data/influxdb.log (deleted)
mysqld 163104 86848 mysql 4u REG 8,5 0 201 /tmp/ibmoE7Gw (deleted)
mysqld 163104 86848 mysql 5u REG 8,5 0 202 /tmp/ibETXoYu (deleted)
果然发现:
1)有大文件占用,文件已经从磁盘删除掉了,但是文件句柄并没有回收
2)进程是mysql且mysql就是根目录中
解决方式:重启mysql进程
扩展:如果是线上环境,是比较麻烦的,毕竟服务进程不能随便重启