Analyzing Disk Usage

When diagnosing storage issues, it is essential to identify which directories are consuming the most space. The following command sorts directories within /var by their human-readable size:

du -sh /var/* | sort -h

Output example:

0 /var/preserve
0 /var/run
0 /var/tmp
8.0K /var/db
16M /var/log
77M /var/spool
144M /var/lib
315M /var/cache

Simulating Low-Inode Filesystems

A "No space left on device" error can occur even when disk blocks are available if the filesystem runs out of inodes. To demonstrate this, we can create a small loopback filesystem.

Step 1: Creating a Disk Image

Create a 100KB file filled with zeros to act as a virtual block device:

dd if=/dev/zero of=/var/tmp/disk_image.img bs=1K count=100

Verification:

ls -lh /var/tmp/disk_image.img
file /var/tmp/disk_image.img

Step 2: Formatting the Image

Format the file with the ext4 filesystem. Note that this is a file, not a block device, so confirmation is required:

mkfs.ext4 /var/tmp/disk_image.img
# Proceed anyway? (y,n) y

Step 3: Mounting the Filesystem

mkdir -p /mnt/storage_test
mount -o loop /var/tmp/disk_image.img /mnt/storage_test
df -ih | grep storage_test

Step 4: Inode Exhaustion Test

Attempt to create multiple files. Although there is disk space, the inode limit (small on a 100KB partition) will be reached quickly:

cd /mnt/storage_test
touch data_file_{01..20}.txt
# Error: touch: cannot touch ‘data_file_06.txt’: No space left on device

Checking the usage reveals that while block space is available, inodes are exhausted:

df -h /mnt/storage_test
# Filesystem Size Used Avail Use% Mounted on
# /dev/loop0   93K  14K   72K  17% /mnt/storage_test
df -i /mnt/storage_test
# IUse% 69% (or 100% depending on limits)

Understanding File Deletion Mechanics

In Linux, a file is truly deleted only when two conditions are met:

1. The hard link count is zero (no directory entry points to it).
2. No process is holding the file open (file descriptor count is zero).

If a file is deleted with rm but a process still has it open, the disk space is not freed until the process closes the file or the process is terminated.

Troubleshooting "Deleted" Files Occupying Space

If df -h shows high usage but du -sh shows lower usage, deleted files held by processes are likely the cause. Use lsof to find them:

lsof | grep deleted

Output example:

rsyslogd  1234 root  6w  REG  8,3  4889144307  73298  /var/log/messages (deleted)

To free the space, restart the service holding the file handle:

systemctl restart rsyslog.service

Handling "Argument list too long" Errors

Commands like rm or touch fail when operating on a massive number of files because the argument list exceeds the shell's limit (ARG_MAX).

Bypassing Limits with xargs

Instead of passing all files directly, pipe the list to xargs, which executes the command in batches.

Creating a large number of files:

mkdir -p /tmp/batch_test
# Direct expansion fails:
# touch /tmp/batch_test/{1..500000} -> Argument list too long

# Using echo and xargs:
echo /tmp/batch_test/{1..500000} | xargs touch
ls /tmp/batch_test/ | wc -l
# 500000

Deleting a large number of files:

# Direct deletion fails:
# rm /tmp/batch_test/* -> Argument list too long

# Using find and xargs:
find /tmp/batch_test/ -type f | xargs rm -f

Alternatively, if the directory itself can be removed, simply deleting the directory is faster, provided permissions and ownership are noted for recreation.

Inode Exhaustion Scenarios

Creating many small files consumes inodes rapidly. A directory containing many files consumes metadata blocks itself.

mkdir small_files
touch small_files/file_{1..50000}
du -sh small_files/
# drwxr-xr-x 2 root root 1.2M ...
ls small_files/ | wc -l
# 50000

If the system reports "No space left on device" but df -h shows free space, check inode usage with df -i.

Managing Swap Space

When memory-intensive applications like Java run out of RAM, the system relies on swap. Adding swap space can prevent crashes.

Adding Swap Temporarily

1. Create a swap file:

dd if=/dev/zero of=/swapfile bs=1M count=512

2. Format as swap:

mkswap /swapfile

3. Secure permissions and activate:

chmod 600 /swapfile
swapon /swapfile

Making Swap Permanent

To persist swap acrosss reboots, add an entry to /etc/fstab:

/swapfile none swap sw 0 0

Alternatively, for certain environments, adding the swapon command to /etc/rc.local can be used, though /etc/fstab is the standard method.