An Unattended Job Stopped With 11MB Free. du Counted Only 59MB; the Other 9GB Was Somewhere I Couldn't See
Two bash scripts that measure free space on the filesystem you are about to write to and stop an unattended agent run before it starts. Includes why du and df disagreed and how to spot a shortage you cannot fix.
I run unattended agent work at fixed times, morning and evening. One day it stopped at the same early step twice in a row.
The first log said "free: 70MB" and the second said "free: 178MB". Both were aborted by the environment check before the repository was fetched, so nothing was changed in the articles or the repository.
After reading the logs, I suspected my own working directory first. Running du on my home directory gave 3.5GB, and for a moment I thought I had found the culprit.
Here is the conclusion up front. What an unattended job should measure is not the size of your home directory but the free space of the filesystem you are about to write to. And before it stops, it should record whether any of the shortage is something you can clean up yourself.
This article keeps the full text of the two bash scripts I use for that check, along with the numbers they actually printed.
The numbers from the day it stopped twice
First, the logs and the values I measured afterwards.
When
Free on the write target
What happened
First run (morning)
70MB
Below the 800MB standard and below the 300MB floor. Ended without fetching or changing anything
Second run (evening)
178MB
175MB even after clearing caches. Aborted for being under 300MB
While writing this article
11MB
9,441MB of a 9,980MB partition used. Inodes at 64%
Free space had been shrinking little by little through the day. Both times the guard worked and stopped without leaving a half-written file. That "stop without writing" behavior is something I built on purpose into the entry-point script described later.
du said 3.5GB while df said 100%
Running du -sm on my home directory returned 3,501MB. The partition showed 9,441MB used, so I assumed the rest was elsewhere and looked at the breakdown. That is where the numbers explained themselves.
How I measured
Value
What it told me
du on the whole home directory
3,501MB
3,443MB of it was the contents of a mounted shared folder
df on the mount
309,758MB free
A different filesystem with about 300GB free. The shortage was not there
du limited to the same partition (-x)
60MB
What I actually keep there is the 59MB repository fetched for this article, plus a little
df on the partition
9,441MB used
Even after subtracting my 60MB, about 9,382MB cannot be explained
Most of the 3.5GB belonged to another filesystem. The du on my home directory was not measuring consumption on the partition that was running out.
So who was using the remaining 9GB? Looking at the top of the partition, I found about 290 per-session directories. Most belong to other runs. Permissions keep me from opening them, so du cannot count them either.
On this partition, then, no matter how much of my own directory I clean up, free space barely moves. I needed to know that before sending any instruction to clean up.
✦
Thank you for reading this far.
Continue Reading
What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.
WHAT YOU'LL LEARN
✦You'll be able to measure free space on the write target with one command before an unattended job starts, and stop without writing anything when it is short
✦You'll be able to tell, within minutes, how much of a full disk you can clean up yourself and how much you cannot, when du is small but df shows 100%
✦You'll be able to separate 'did not start' from 'failed' with an exit code, so the next morning's log tells you the cause the moment you open it
Secure payment via Stripe · Cancel anytime
✦
Unlock This Article
Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.
After this episode I narrowed the check at the entry of an unattended job to three decisions.
The first is to measure the write target. Not home, not /tmp, but the filesystem that holds the directory I am about to fetch into. Even when the destination does not exist yet, the script walks up to the nearest existing parent.
The second is two thresholds. Below 800MB it warns and continues; below 300MB it stops. These values are my own rule of thumb, chosen because a fetched repository plus temporary files plus logs should stay within a few hundred MB.
The third is how it reports a stop. "Did not start" is returned with its own exit code (20), separate from a job failure. And when most of the usage is invisible to me, the script writes that into the record, so that whoever opens the log the next morning can see at once that this is not a problem their own cleanup will fix.
The full text of disk_guard.sh
First, what it solves. From three arguments (write target, warn MB, stop MB), it measures free space and exits with 0, 10 or 20. If you add -- and a command after those, it runs that command only when the check passes.
The -P in df -Pm pins one filesystem to one line even when the device name is long. Without it, the line read by awk 'NR==2' can shift.
The while loop that walks up to a parent is for mornings when the clone destination does not exist yet. Pointing df at a path that does not exist returns an error and an empty value.
The verdict rests on AVAIL alone. unexplained_mb is not used in the decision, only in the note printed when the guard stops. Adding conditions would make it stop wrongly on more days.
exec "$@" hands the process over to the caller's command. A guard that lingers makes the exit code of what follows harder to read.
What it actually printed
I ran it on the real environment (11MB free), on the mount, and with relaxed thresholds plus a command.
The note on the stop line is in Japanese because I run this in my own environment; it says that most of the usage on this area is somewhere I cannot see, and that deleting my own working directory will not free space. Translate the string if you adopt the script.
Please ignore unexplained_mb=1594908 on the second run. On a large shared mount, everyone else's files all fall into "unexplained". The value only means something on a small, nearly private partition, which is why the note is printed only when the guard stops.
The entry-point wrapper, run_unattended.sh
The other script calls the guard and, when it stops, leaves the reason in a file. Because the file name contains STOPPED, a glance at the directory listing the next morning separates "failed" from "did not start".
With WORKDIR pointed at a location with room to spare, it printed start job (guard rc=0) and went on to the real job. Put the command that launches your agent where the final echo is.
Four pitfalls
Trusting a du number as it is. When a mounted, separate filesystem is included, as it was here, the figure has nothing to do with consumption on the partition that is running out. Use du -x so the measurement does not cross partitions.
Assuming a shortage is fixed by cleaning up. If the partition is shared with other runs, deleting your own files may not be enough. The guard deletes nothing, so an unattended run never touches files it has no permission over.
Not looking at inodes. Even with plenty of free space, writes fail once inodes run out. They were at 64% here, with room to spare, but recording this value before work that creates many small files is worth it.
Using a single threshold. A stop-only rule halts suddenly on a marginal day. If you accumulate the 800MB warnings in warn.log, you can notice a day before it stops.
What I have not verified
To be honest about it: I did not place this guard in front of the agent's own launch command and run it. What I confirmed is the guard on its own and the numbers from the day it stopped.
I also cannot see what fills the other 9GB of the partition with my permissions, so the cause of the fill is still unknown. The 800MB and 300MB thresholds come from two days of experience, and I do not know whether they carry over to other environments.
Next step
First, add disk_guard.sh as a single line at the entry of an unattended job you already run. Point WORKDIR at the real destination and just watch whether warn.log grows for a few days; that is enough to learn which days are risky in your environment.
If you also want to tidy how you keep logs, or how you stop without leaving half-written files, these articles may help.
On days when there is not enough room, being able to stop without writing anything is, I think, the smallest courtesy the person handing off the work can offer.
Share
Thank You for Reading
Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.