With scheduled agents, I now look for runs that never happened before I look for failures
My run ledger showed a 100% success rate while the evening slot had not fired for two weeks. Here is the reconciliation I now run against an expected-fire table, with measured notes on cron expansion, exit codes, and how the ledger itself gets written.
I had increased a wallpaper classification batch to twice a day, or so I believed, until the morning the processed counts refused to add up. The ledger looked fine. Every entry said OK. There were no errors anywhere.
The counts were still short. I read the timestamps one by one and finally saw it. Only the morning slot was there. The evening slot had not appeared once in two weeks.
Until then I had been measuring the health of my unattended work by the absence of failures. If nothing failed, I assumed things were fine. But a run that never happens is not recorded as a failure either. Nothing lands in the ledger, and nothing rings. All that remains is the fact that nothing happened.
Since then I have turned the direction of my monitoring around. I did not stop looking at what happened. I added a step before it: counting what was supposed to happen.
Failure alerts cannot reach a run that never started
Written out, the reason is obvious. A failure alert fires only once a process is alive. A run that never started has no process, and therefore nothing to raise the alert.
Antigravity's Remote Control push notifications behave the same way. They arrive when an agent completes a task, or when it needs more input from you — that is, only for runs that began. Those notifications have saved me plenty of evenings, but they are not a channel that can tell you "today it simply did not run."
Counting on my own ledger, three days of records gave me this:
The success rate is 6/6, a clean 100%. Read from the side of what was expected, the same ledger is 6/9, or 67%. Same data, different denominator, opposite conclusion.
Count the runs that should have happened, not the ones that succeeded. Putting that sentence at the top of my operations notes was the point where I started trusting my overnight work again.
Hold the expectation as a table, not as an expression
When I started digging, I suspected the cron expression first. It read 30 4,16 * * *. I could not find anything wrong with it. Validating the expression itself led nowhere.
So I stopped reading it and expanded it instead. A short function that opens up the minute and hour fields is enough.
# expect.py - expand a cron expression into the times it should fire# Day, month and weekday are assumed to be *. For run auditing I only needed# the minute and hour fields.from datetime import datetimedef parse_field(f, lo, hi): """Expand commas, ranges and steps into a set of values.""" out = set() for part in f.split(","): if part == "*": out |= set(range(lo, hi + 1)) continue if "/" in part: # */6 or 0-23/6 base, step = part.split("/") rng = range(lo, hi + 1) if base == "*" else range( int(base.split("-")[0]), int(base.split("-")[-1]) + 1) out |= set(v for v in rng if (v - min(rng)) % int(step) == 0) continue if "-" in part: # 9-17 a, b = part.split("-") out |= set(range(int(a), int(b) + 1)) continue out.add(int(part)) return sorted(out)def expected(expr, day): """Return the datetimes that should fire on this day, ascending.""" minute_field, hour_field = expr.split()[0], expr.split()[1] minutes = parse_field(minute_field, 0, 59) hours = parse_field(hour_field, 0, 23) return [day.replace(hour=H, minute=M, second=0, microsecond=0) for H in hours for M in minutes]
Running it on my machine:
'30 4,16 * * *' -> 2 per day ['04:30', '16:30']
'30 4 * * *' -> 1 per day ['04:30']
'30 16 * * *' -> 1 per day ['16:30']
'0 */6 * * *' -> 4 per day ['00:00', '06:00', '12:00', '18:00']
As an expression, it is two firings. The syntax was never the problem. What actually ran was one. How a given runner treats a multi-slot field turned out to be a separate question from whether the expression is valid — that was the moment it finally landed for me.
Anything I want to run twice a day now lives as two separate jobs. Whatever I gain by keeping the expression compact is worth less than the risk of one half going quiet.
How it is written
Expected by the expression
Failure mode
What I do now
30 4,16 * * *
2 per day
One slot can fire alone, silently
I avoid it
30 4 * * * and 30 16 * * *
1 per day each
A stopped slot shows up as a gap
I split it this way
0 */6 * * *
4 per day
Thinning is hard to notice
Reconciliation is mandatory
✦
Thank you for reading this far.
Continue Reading
What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.
WHAT YOU'LL LEARN
✦You will be able to catch runs that never started at all, through a path separate from your failure alerts
✦You will be able to add expected-versus-actual reconciliation on top of the schedules you already have, without touching them
✦You will be able to measure the gap between success rate and coverage in your own setup (the same ledger reads 6/6 and 6/9)
Secure payment via Stripe · Cancel anytime
✦
Unlock This Article
Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.
Write the ledger line when the run starts, not when it ends
Reconciliation needs a record from the side that actually ran. What mattered most here was writing it at the start rather than at the end. If you only write on completion, a run that died halfway becomes indistinguishable from a run that never began.
Writing that ledger burned me twice, in both cases by corrupting the record itself.
The first was exit codes. If you pipe a check through head or tail, only the last exit code in the pipeline survives.
# Measured: the same check flips its result once a pipe is involved$ grep -q NOPE /etc/hostname | head -1; echo $?0 # it failed, yet reports 0$ set -o pipefail; grep -q NOPE /etc/hostname | head -1; echo $?1 # pipefail brings it back$ grep -q NOPE /etc/hostname; echo $?1 # not piping at all is the safest
The status column in the ledger is decided by that exit code. In other words, one pipe is enough to record a failure as OK. I no longer send a command whose result I depend on through a pipe, not even for formatting.
The second was heredocs. Leave the delimiter unquoted and any backticks or dollar signs inside the body get expanded on the spot.
# Bad: <<EOF (unquoted)cat << EOF > ledger_bad.txtid=$RUN_ID status=OK started=`date -u +%s` note=`echo INJECTED`EOF# Output (what got executed, not what I meant to record)# id=run-2026-09-06 status=OK started=1788657291 note=INJECTED
Notice how note has been replaced. The ledger I was keeping for auditing purposes was rewriting itself the instant it was written — and I found out on the day I went back to read it looking for gaps.
Keep the reconciliation outside the run
With that in place, the audit itself is short. Line the expected-fire table up against the ledger, and report any slot with no record inside the grace window.
# audit.py - reconcile the expected-fire table against the run ledgerfrom datetime import datetime, timedeltaimport json, sysfrom expect import expectedGRACE = timedelta(minutes=20) # startup delay allowance (chosen below)def load_ledger(path): """Read JSONL, one record per line, sorted by start time.""" rows = [] for line in open(path, encoding="utf-8"): line = line.strip() if not line: continue r = json.loads(line) rows.append((datetime.fromisoformat(r["started_at"]), r)) return sorted(rows)def audit(schedules, ledger_path, days): rows = load_ledger(ledger_path) missing, matched = [], 0 for job, expr in schedules.items(): seen = [t for t, r in rows if r["job"] == job] for day in days: for exp in expected(expr, day): if exp > datetime.now(): # slots still ahead are not gaps continue if any(abs(t - exp) <= GRACE for t in seen): matched += 1 else: missing.append((job, exp)) return matched, missingif __name__ == "__main__": schedules = { "wallpaper-category": "30 4,16 * * *", "reference-refresh": "0 7 * * *", } days = [datetime(2026, 9, 4), datetime(2026, 9, 5), datetime(2026, 9, 6)] matched, missing = audit(schedules, "ledger.jsonl", days) print(f"matched runs: {matched}") print(f"missing runs: {len(missing)}") for job, t in missing: print(f" MISSING {job} @ {t:%Y-%m-%d %H:%M}") sys.exit(1 if missing else 0) # non-zero when anything is missing
Dates and slots, named. What took me two weeks to notice now fits in three lines.
I run this as a job separate from the ones it watches. Put it inside the same machinery and it goes quiet on exactly the day that machinery goes quiet. Mine runs once a day, early in the morning.
Choose the grace window from your measured startup delay
There is a reason GRACE is twenty minutes. I started at five and healthy runs began showing up as gaps. Agent startup carries real jitter from model initialisation and credential refresh.
The way I settle it is simple: measure the actual offsets in the ledger you already have. Mine, against an 04:30 slot, were 04:31:12, 04:30:41 and 04:33:05 — three minutes five seconds at worst. Twenty minutes gives that room to breathe.
# List the offsets against the expected time before fixing the windowpython3 - << 'EOF'import jsonfrom datetime import datetimefor line in open("ledger.jsonl", encoding="utf-8"): r = json.loads(line) t = datetime.fromisoformat(r["started_at"]) delay = t.minute * 60 + t.second - 30 * 60 # seconds past 04:30 print(f'{r["job"]:20} {t:%m-%d %H:%M:%S} delay {delay:>4}s')EOF
Open it too wide and the window bleeds into the neighbouring slot, so a gap stops registering as a gap. For a twice-daily job I keep a quarter of the interval as my ceiling.
What actually bit me in production was the clock
The first morning I ran the reconciliation, every single slot came back missing. I assumed the machinery had stopped. The cause was how the times were being held. The ledger was written in UTC because that is what the runner produced, while I had built the expected table in my local Japan time.
from datetime import datetime, timedeltafrom zoneinfo import ZoneInfoutc = datetime(2026, 9, 6, 4, 31, 12, tzinfo=ZoneInfo("UTC")) # a ledger linejst = datetime(2026, 9, 6, 4, 30, tzinfo=ZoneInfo("Asia/Tokyo")) # an expected slotprint(abs(utc - jst)) # 9:01:12print(abs(utc - jst) <= timedelta(minutes=20)) # False - everything looks missing# And if you simply drop the timezone to make the comparison workprint(abs(utc.replace(tzinfo=None) - jst.replace(tzinfo=None))) # 0:01:12
The awkward part is that these two mistakes fail in opposite directions. Compare them timezone-aware and you are nine hours off, so every slot reads as a gap. Drop the timezone because that felt easier and the difference looks like one minute twelve seconds, so every slot matches. The second one shows no symptom at all, which makes it far more dangerous.
The fix, once decided, is plain. Write started_at with its timezone attached, and state the same timezone when you build the expected table. In this situation I would recommend keeping the comparison on one side and pushing local time out to display only.
# Build the expected table with an explicit timezone toofrom zoneinfo import ZoneInfoTZ = ZoneInfo("Asia/Tokyo")days = [datetime(2026, 9, 4, tzinfo=TZ), datetime(2026, 9, 5, tzinfo=TZ)]
One more thing I only noticed after this went into production use: slots that sit near midnight, an 23:50 job for instance, slip through if the audit only looks at the current date. I now always include the previous day in the range.
Point the alert at the gap, not only at the failure
One last thing about how it rings. I did not remove my failure alerts. I left them in place and added gap alerts to a different destination. A failure is a signal to go read that run; a gap is a signal to doubt the machinery — different in kind, and easier to act on when they are not mixed together.
Pick a single scheduled job and add one ledger line at the moment it starts. The reconciliation can come later.
For me, having gaps surface in three lines has cut down the time I spend imagining what might be going on overnight. Thank you for reading this far.
Share
Thank You for Reading
Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.