Disk Space Mismatch Between df and du Commands

A disk usage check reveals low available space.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   48G  1.2G  98% /

Inspecting directory sizes with du shows a discrepancy, indicating space not accounted for in the filesystem hierarcyh.

# du -sh /* | sort -hr | head -5
12G /var
8G  /usr
3G  /home
2G  /opt
1G  /tmp

Identifying Processes with Open Deleted Files

The lsof command lists processes that retain handles to files marked as deleted.

# lsof +L1
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
java    24501  appuser 4w   REG   8,1   2.1G      0    123456 /app/logs/app.log (deleted)
nginx   1123   root    7w   REG   8,1   500M      0    234567 /var/log/nginx/access.log (deleted)

An alternative, more targeted query:

# lsof | grep -i deleted

Reclaiming Occupied Disk Space

Terminate the idantified processes to close their file handles and release space.

# kill -15 24501  # Send SIGTERM
# kill -9 1123    # Forcefully kill if necessary

After a moment, verify that space has been freed.

# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   36G   14G  72% /

Technical Explanation

In Linux, deleting a file (rm) removes its directory entry (unlinks it). However, if any process still has an open file descriptor to it, the inode and data blocks remain allocated untill all handles are closed. The storage is only freed for reuse by the system after the final reference is released.

Resolution Procedure

1. Identify processes holding open deleted files using lsof.
2. Reclaim space by either:
   • Stopping or restarting the specific processes.
   • Gracefully stopping the application service to allow the operating system to clean up resources.