Disk full on a VPS: finding what ate the space when you can't even log in

Your monitoring pinged, or a deploy failed, and now SSH either refuses the connection or drops you the instant you authenticate. The panel throws 500s. MySQL is down with Can't create/write to file or Disk is full writing. One thing broke all of it: the root filesystem hit 100%. Writing .bash_history fails, PAM can't create a session file, sshd can't write its temp state, so the login that normally works now bounces you.

This is the recovery order that works, starting from the case where you can't get a normal shell at all. Every result tells you what to rule out.

First, get any shell at all

If SSH drops immediately, don't keep retrying it. A completely full / breaks interactive login because the session needs to write, and there is nowhere to write. Go to your provider's control panel and open the serial or VNC console. That path doesn't depend on sshd or on free disk the same way, so it still lets you in when SSH won't.

Log in as root on the console. If even that hangs, reboot into single-user or a rescue image from the panel and mount the disk from there. Once you have a prompt, you can diagnose.

One quick trick buys a working shell if commands themselves fail to run: clear a scratch file first.

: > /var/log/syslog        # truncate the biggest writable log in place
rm -f /var/tmp/* 2>/dev/null

That frees a few blocks so tools can allocate temp space. Now work the problem properly.

df -h versus df -i: two different "full"

Run both. They answer different questions.

df -h
df -i

df -h shows block usage. If the mount with your data shows 100% on Use%, you are out of space in the ordinary sense and you walk du next.

If df -h shows room free but writes still fail with No space left on device, check df -i. A filesystem has a fixed pool of inodes, one per file, directory and symlink, set at format time on ext4. When IUse% hits 100% you cannot create a single new file even with gigabytes of block space free. The usual cause is millions of tiny files: PHP session files, a mail queue, a cache that never prunes, or a container overlay. Inode exhaustion looks identical to a full disk from the application's point of view and is fixed differently. You don't truncate a 40 GB log; you find the directory holding the file count and delete the files.

To find where the inodes went:

du --inodes -xd1 / 2>/dev/null | sort -n | tail -20

The -x keeps it on one filesystem so it doesn't wander into /proc or a mount you don't care about. On older coreutils without --inodes, count by hand:

for d in /*; do echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"; done | sort -n

Delete the offending files, then confirm with df -i / and a touch /tmp/t && rm /tmp/t. If creating a file works again, the inode problem is solved and block usage was a red herring.

Walk du from the root down

When it's genuine block exhaustion, don't guess. Measure from / and follow the biggest number down one level at a time.

du -xh --max-depth=1 / 2>/dev/null | sort -h

The -x matters: without it you double-count bind mounts and chase size into pseudo-filesystems. Whichever directory is largest, descend into it:

du -xh --max-depth=1 /var 2>/dev/null | sort -h
du -xh --max-depth=1 /var/lib 2>/dev/null | sort -h

Three or four steps and you are standing on the thing that ate the disk. Nine times out of ten it lands in /var. Here is where to look and what each culprit needs.

journald: it defaults to 10% of the filesystem

On a persistent journal, systemd's SystemMaxUse defaults to 10% of the filesystem the journal sits on, with SystemKeepFree reserving 15%. On a small VPS 10% is a lot of log. Check and trim:

journalctl --disk-usage
journalctl --vacuum-size=200M
# or by age:
journalctl --vacuum-time=2d

Vacuum only deletes archived (rotated) journal files; the active file is never touched, so you reclaim less than the total if most of it is still being written. Set a hard ceiling in /etc/systemd/journald.conf with SystemMaxUse=200M and systemctl restart systemd-journald so it can't come back.

Old kernels and the apt cache

Every kernel upgrade keeps the old image in /boot and its modules in /usr/lib/modules. On a VPS with a tiny /boot this alone can wedge an upgrade. Clean safely:

apt-get clean                 # dumps /var/cache/apt/archives
apt-get autoremove --purge    # drops unused kernels, keeps the running one

Never rm a kernel by hand from /boot; let apt update the bootloader and initramfs with it.

Docker, when it's the whole story

If /var/lib/docker is the fat directory, start with docker system df to see images, containers, volumes and build cache separately. Pruning images is safe; pruning volumes deletes data a stopped container may still need. We wrote the careful version of this up separately in reclaiming Docker disk space safely, including why prune --volumes is the line to think twice about.

MySQL binlogs: purge from inside MySQL, never with rm

Binary logs live beside the data in /var/lib/mysql and grow forever under load. Since MySQL 8.0, binlog_expire_logs_seconds defaults to 2592000, which is 30 days. On a write-heavy box 30 days of binlogs can be larger than the data they describe. If MySQL is still up, purge through the server so the log index stays consistent and replication stays safe:

mysql> SHOW BINARY LOGS;
mysql> PURGE BINARY LOGS BEFORE NOW() - INTERVAL 2 DAY;

Then lower retention: SET GLOBAL binlog_expire_logs_seconds = 259200; for 3 days, and make it permanent in my.cnf. If you run replicas, confirm every replica's Read_Source_Log_Pos is past the boundary before you purge, or you will break them. Do not delete mysql-bin.* files with rm; that leaves the index pointing at files that no longer exist and MySQL may refuse to start.

The space du swears isn't there: deleted-but-open files

You deleted a 20 GB log, df -h didn't move, and du can't find anything that big. This is the one that wastes an hour if you don't know it. When a process has a file open and you rm it, the directory entry goes away, so du (which walks directory names) stops counting it. But the process still holds the file descriptor, so the kernel keeps the blocks allocated and df still counts them. The space comes back only when that process closes the handle or exits.

Find them with the deleted-handle selector:

lsof +L1

The NLINK column shows 0: no name on disk, still open. The classic offenders are a logging daemon or an app writing to a logfile that logrotate removed without signalling the process to reopen. Two ways out. Restart the process holding the handle, which releases the space instantly. Or, if you can't restart it yet, truncate through its file descriptor:

truncate -s 0 /proc/<pid>/fd/<fd>

That zeroes the open file in place and frees the blocks without killing the service. This is also why df and du disagree far more often than people expect.

Buying space from a console when nothing will start

If services won't come up because there is no room, free a block of space first, in this order, from the rescue shell:

• apt-get clean or dnf clean all — package caches are pure reclaim, zero risk.
• journalctl --vacuum-size=100M — fast and safe.
• Truncate the single biggest file in /var/log with : > /var/log/bigfile.log rather than deleting it, so an open handle doesn't strand the space.
• Clear any deleted-but-open handle from lsof +L1.

Reclaim 1-2 GB this way, then start MySQL or the panel, and only then do the proper cleanup with services running. Don't delete anything under /var/lib/mysql or a database data directory to free space. That is how a disk-full incident becomes a data-loss incident.

When you can't fix it from your side

Sometimes the honest answer is that the disk is simply too small for the workload. You purged logs, trimmed the journal, pruned Docker, and you are back to 95% within a day because the application genuinely needs more than the plan provides. No amount of du changes that. At that point cleanup is a holding action and the fix is more disk.

This is where resizing beats rebuilding. On UK VPS in Coventry you raise disk, RAM and vCPU from the client area and apply it with a single reboot, keeping your data and your IP address. You are not migrating to a new server, re-pointing DNS, or restoring from backup; the same volume comes back larger. If your users sit closer to the Nordics or mainland Europe, the same applies on our Netherlands VPS in Naaldwijk. A 20-minute resize is cheaper than the next 2am lockout.

Stop it recurring: rotation and a cheap alarm

The incident is the symptom. These are the controls that prevent it:

• journald ceiling. Set SystemMaxUse=200M (or less) in journald.conf. Don't rely on the 10% default on a small disk.
• logrotate that signals apps. Make sure each app's logrotate stanza uses copytruncate or sends a reopen signal, so rotation doesn't create the deleted-but-open problem above.
• binlog retention. Set binlog_expire_logs_seconds to what you actually need for point-in-time recovery, often 3-7 days, not 30.
• Docker on a schedule. A weekly docker image prune -af via cron keeps build cache from creeping.
• An alarm at 80%, not 100%. A one-line cron that mails you when df crosses 80% turns a 2am outage into a daytime task. Even df -h / | awk 'NR==2 && $5+0 > 80' in cron is enough to start.

The difference between a filesystem at 80% and one at 100% is the difference between a note in your inbox and a login you can't complete. Set the alarm below the cliff and you never read the rest of this article again at 2am.

Choosing the plan

If cleanup keeps buying you only a day or two, size up rather than fight it weekly. Pick disk headroom of at least 30% above steady-state on the UK VPS in Coventry, so logs, binlogs and an OS upgrade all have somewhere to go. Both there and on the Netherlands VPS in Naaldwijk the upgrade is one reboot with the same data and IP, and on-demand snapshots let you take a safety point before you resize. If the real constraint is predictable I/O under heavy database writes, a dedicated-CPU plan reserves physical cores and removes the other half of the pressure.

Sources

• MySQL Reference Manual: Binary Logging Options and Variables
• Find Deleted Files Still Holding Disk Space in Linux (df vs du)
• Fix "No Space Left on Device" When df Shows Free Space
• Linux: reducing disk usage of Systemd journal logs (2024-09)
• Mastering MySQL Binary Logs: Optimize Disk Space

Frequently asked questions