'No Space Left on Device' With 50GB Free Means You're Out of Inodes, and df -i Shows It

The engineer, who posts as fock1e, was on call for a logging pipeline when rsyslog kept crashing with “cannot create ‘/var/log/syslog.1’: No space left on device.” The obvious check said otherwise: df -h showed 50GB free on the root filesystem and 11GB free on /var/log.

They name the trap directly. “A Linux filesystem tracks two resources, not one”: bytes, and inodes, which count “how many files can exist.” Every file, directory and symlink uses one inode, and when they run out the kernel returns ENOSPC, which shows up as the same “No space left on device” message a full disk gives.

The second command, df -i, which lists “inode information instead of block usage,” found it:

$ df -i
Filesystem      Inodes    IUsed IFree IUse% Mounted on
/dev/sda1      6553600  6553598     2  100% /
/dev/sda2      1310720  1310718     2  100% /var/log

Two free inodes on each filesystem. A count with find /var/log -type f | wc -l turned up 1,310,715 files, nearly all named like session_000001.log, and every one was 0 bytes. “Someone wrote a debug script, forgot to clean up, and it’s been creating empty session logs for months.”

The fix was deleting them with find, since plain rm with a wildcard fails when the argument list is too long:

find /var/log -type f -name 'session_*' -delete

It took about 30 seconds, df -i dropped to 1%, and rsyslog started again.

Freeing bytes won’t help here, and on ext4 neither will growing the inode count in place. The mke2fs manual says the filesystem gets “an inode for every bytes-per-inode bytes of space,” and warns that “it is not possible to change this ratio on a file system after it is created.” XFS works differently: its mkfs.xfs manual sets “the maximum percentage of space in the filesystem that can be allocated to inodes” rather than a fixed count.

Small-file pileups keep catching people because most dashboards show one percentage per disk. A server that looks half empty can be completely full, and the people who get paged are the ones who check bytes and stop there. One more line in the alert config, watching df -i, catches it before rsyslog does.