Resolution
Graceful shutdown of relevant process
First, obtain a list of deleted files which are still held open by applications:
$ lsof | egrep "deleted|COMMAND"
COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
ora 25575 8194 oracle oracle 33 REG 65,65 4294983680 31014933 /oradata/DATAPRE/file.dbf (deleted)
Note: check either the filesystem path within NAME field or the device number under DEVICE to match the filesystem of interest.
The lsof
output shows the process with pid 25575
has kept file /oradata/DATAPRE/file.dbf
open with file descriptor (fd) number 33
.
After a file has been identified, free the file used space by shutting down the affected process. If a graceful shutdown does not work, then issue the kill
command to forcefully stop it by referencing the PID
.
Truncate File Size
Alternatively, it is possible to force the system to de-allocate the space consumed by an in-use file by forcing the system to truncate the file via the proc
file system. This is an advanced technique and should only be carried out when the administrator is certain that this will cause no adverse effects to running processes. Applications may not be designed to deal elegantly with this situation and may produce inconsistent or undefined behavior when files that are in use are abruptly truncated in this manner.
$ echo > /proc/pid/fd/fd_number
For example, from the lsof
output above:
$ file /proc/25575/fd/33
/proc/25575/fd/33: broken symbolic link to `/oradata/DATAPRE/file.dbf (deleted)'
$ echo > /proc/25575/fd/33
The same reason will cause different disk usage from du
command and df
command, please refer to Why does df show bigger disk usage than du?
To identify the used file size (in blocks), use the command below:
# lsof -Fn -Fs |grep -B1 -i deleted | grep ^s | cut -c 2- | awk '{s+=$1} END {print s}'