Articles/Agents & Manager

◈ Agents & Manager/2026-09-06Advanced

With scheduled agents, I now look for runs that never happened before I look for failures

My run ledger showed a 100% success rate while the evening slot had not fired for two weeks. Here is the reconciliation I now run against an expected-fire table, with measured notes on cron expansion, exit codes, and how the ledger itself gets written.

Antigravity³⁶⁴ agents¹⁴⁰ scheduled runs⁵ cron² operations³¹

✦ Premium Article

I had increased a wallpaper classification batch to twice a day, or so I believed, until the morning the processed counts refused to add up. The ledger looked fine. Every entry said OK. There were no errors anywhere.

The counts were still short. I read the timestamps one by one and finally saw it. Only the morning slot was there. The evening slot had not appeared once in two weeks.

Until then I had been measuring the health of my unattended work by the absence of failures. If nothing failed, I assumed things were fine. But a run that never happens is not recorded as a failure either. Nothing lands in the ledger, and nothing rings. All that remains is the fact that nothing happened.

Since then I have turned the direction of my monitoring around. I did not stop looking at what happened. I added a step before it: counting what was supposed to happen.

Failure alerts cannot reach a run that never started

Written out, the reason is obvious. A failure alert fires only once a process is alive. A run that never started has no process, and therefore nothing to raise the alert.

Antigravity's Remote Control push notifications behave the same way. They arrive when an agent completes a task, or when it needs more input from you — that is, only for runs that began. Those notifications have saved me plenty of evenings, but they are not a channel that can tell you "today it simply did not run."

Counting on my own ledger, three days of records gave me this:

matched runs        : 6
missing runs        : 3
caught by failure-only alerting: 0

The success rate is 6/6, a clean 100%. Read from the side of what was expected, the same ledger is 6/9, or 67%. Same data, different denominator, opposite conclusion.

Count the runs that should have happened, not the ones that succeeded. Putting that sentence at the top of my operations notes was the point where I started trusting my overnight work again.

Hold the expectation as a table, not as an expression

When I started digging, I suspected the cron expression first. It read 30 4,16 * * *. I could not find anything wrong with it. Validating the expression itself led nowhere.

So I stopped reading it and expanded it instead. A short function that opens up the minute and hour fields is enough.

# expect.py - expand a cron expression into the times it should fire
# Day, month and weekday are assumed to be *. For run auditing I only needed
# the minute and hour fields.
from datetime import datetime
 
def parse_field(f, lo, hi):
    """Expand commas, ranges and steps into a set of values."""
    out = set()
    for part in f.split(","):
        if part == "*":
            out |= set(range(lo, hi + 1))
            continue
        if "/" in part:                       # */6 or 0-23/6
            base, step = part.split("/")
            rng = range(lo, hi + 1) if base == "*" else range(
                int(base.split("-")[0]), int(base.split("-")[-1]) + 1)
            out |= set(v for v in rng if (v - min(rng)) % int(step) == 0)
            continue
        if "-" in part:                       # 9-17
            a, b = part.split("-")
            out |= set(range(int(a), int(b) + 1))
            continue
        out.add(int(part))
    return sorted(out)
 
def expected(expr, day):
    """Return the datetimes that should fire on this day, ascending."""
    minute_field, hour_field = expr.split()[0], expr.split()[1]
    minutes = parse_field(minute_field, 0, 59)
    hours = parse_field(hour_field, 0, 23)
    return [day.replace(hour=H, minute=M, second=0, microsecond=0)
            for H in hours for M in minutes]

Running it on my machine:

'30 4,16 * * *'      -> 2 per day  ['04:30', '16:30']
'30 4 * * *'         -> 1 per day  ['04:30']
'30 16 * * *'        -> 1 per day  ['16:30']
'0 */6 * * *'        -> 4 per day  ['00:00', '06:00', '12:00', '18:00']

As an expression, it is two firings. The syntax was never the problem. What actually ran was one. How a given runner treats a multi-slot field turned out to be a separate question from whether the expression is valid — that was the moment it finally landed for me.

Anything I want to run twice a day now lives as two separate jobs. Whatever I gain by keeping the expression compact is worth less than the risk of one half going quiet.

How it is written	Expected by the expression	Failure mode	What I do now
`30 4,16 * * *`	2 per day	One slot can fire alone, silently	I avoid it
`30 4 * * ` and `30 16 * *`	1 per day each	A stopped slot shows up as a gap	I split it this way
`0 /6 * *`	4 per day	Thinning is hard to notice	Reconciliation is mandatory

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦You will be able to catch runs that never started at all, through a path separate from your failure alerts

✦You will be able to add expected-versus-actual reconciliation on top of the schedules you already have, without touching them

✦You will be able to measure the gap between success rate and coverage in your own setup (the same ledger reads 6/6 and 6/9)

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Write the ledger line when the run starts, not when it ends

Reconciliation needs a record from the side that actually ran. What mattered most here was writing it at the start rather than at the end. If you only write on completion, a run that died halfway becomes indistinguishable from a run that never began.

Writing that ledger burned me twice, in both cases by corrupting the record itself.

The first was exit codes. If you pipe a check through head or tail, only the last exit code in the pipeline survives.

# Measured: the same check flips its result once a pipe is involved
$ grep -q NOPE /etc/hostname | head -1; echo $?
0                                  # it failed, yet reports 0
 
$ set -o pipefail; grep -q NOPE /etc/hostname | head -1; echo $?
1                                  # pipefail brings it back
 
$ grep -q NOPE /etc/hostname; echo $?
1                                  # not piping at all is the safest

The status column in the ledger is decided by that exit code. In other words, one pipe is enough to record a failure as OK. I no longer send a command whose result I depend on through a pipe, not even for formatting.

The second was heredocs. Leave the delimiter unquoted and any backticks or dollar signs inside the body get expanded on the spot.

# Bad: <<EOF (unquoted)
cat << EOF > ledger_bad.txt
id=$RUN_ID status=OK started=`date -u +%s` note=`echo INJECTED`
EOF
 
# Output (what got executed, not what I meant to record)
# id=run-2026-09-06 status=OK started=1788657291 note=INJECTED

# Good: <<'EOF' (quoted), then substitute
cat << 'EOF' > ledger_good.txt
id=__RUN_ID__ status=OK started=__TS__ note=none
EOF
sed -i "s/__RUN_ID__/$RUN_ID/; s/__TS__/$(date -u +%s)/" ledger_good.txt
 
# Output
# id=run-2026-09-06 status=OK started=1757116800 note=none

Notice how note has been replaced. The ledger I was keeping for auditing purposes was rewriting itself the instant it was written — and I found out on the day I went back to read it looking for gaps.

Keep the reconciliation outside the run

With that in place, the audit itself is short. Line the expected-fire table up against the ledger, and report any slot with no record inside the grace window.

# audit.py - reconcile the expected-fire table against the run ledger
from datetime import datetime, timedelta
import json, sys
from expect import expected
 
GRACE = timedelta(minutes=20)      # startup delay allowance (chosen below)
 
def load_ledger(path):
    """Read JSONL, one record per line, sorted by start time."""
    rows = []
    for line in open(path, encoding="utf-8"):
        line = line.strip()
        if not line:
            continue
        r = json.loads(line)
        rows.append((datetime.fromisoformat(r["started_at"]), r))
    return sorted(rows)
 
def audit(schedules, ledger_path, days):
    rows = load_ledger(ledger_path)
    missing, matched = [], 0
    for job, expr in schedules.items():
        seen = [t for t, r in rows if r["job"] == job]
        for day in days:
            for exp in expected(expr, day):
                if exp > datetime.now():       # slots still ahead are not gaps
                    continue
                if any(abs(t - exp) <= GRACE for t in seen):
                    matched += 1
                else:
                    missing.append((job, exp))
    return matched, missing
 
if __name__ == "__main__":
    schedules = {
        "wallpaper-category": "30 4,16 * * *",
        "reference-refresh":  "0 7 * * *",
    }
    days = [datetime(2026, 9, 4), datetime(2026, 9, 5), datetime(2026, 9, 6)]
    matched, missing = audit(schedules, "ledger.jsonl", days)
    print(f"matched runs: {matched}")
    print(f"missing runs: {len(missing)}")
    for job, t in missing:
        print(f"  MISSING {job} @ {t:%Y-%m-%d %H:%M}")
    sys.exit(1 if missing else 0)   # non-zero when anything is missing

Against the ledger with the evening slot absent:

matched runs: 6
missing runs: 3
  MISSING wallpaper-category @ 2026-09-04 16:30
  MISSING wallpaper-category @ 2026-09-05 16:30
  MISSING wallpaper-category @ 2026-09-06 16:30
exit=1

Dates and slots, named. What took me two weeks to notice now fits in three lines.

I run this as a job separate from the ones it watches. Put it inside the same machinery and it goes quiet on exactly the day that machinery goes quiet. Mine runs once a day, early in the morning.

Choose the grace window from your measured startup delay

There is a reason GRACE is twenty minutes. I started at five and healthy runs began showing up as gaps. Agent startup carries real jitter from model initialisation and credential refresh.

The way I settle it is simple: measure the actual offsets in the ledger you already have. Mine, against an 04:30 slot, were 04:31:12, 04:30:41 and 04:33:05 — three minutes five seconds at worst. Twenty minutes gives that room to breathe.

# List the offsets against the expected time before fixing the window
python3 - << 'EOF'
import json
from datetime import datetime
for line in open("ledger.jsonl", encoding="utf-8"):
    r = json.loads(line)
    t = datetime.fromisoformat(r["started_at"])
    delay = t.minute * 60 + t.second - 30 * 60     # seconds past 04:30
    print(f'{r["job"]:20} {t:%m-%d %H:%M:%S}  delay {delay:>4}s')
EOF

Open it too wide and the window bleeds into the neighbouring slot, so a gap stops registering as a gap. For a twice-daily job I keep a quarter of the interval as my ceiling.

What actually bit me in production was the clock

The first morning I ran the reconciliation, every single slot came back missing. I assumed the machinery had stopped. The cause was how the times were being held. The ledger was written in UTC because that is what the runner produced, while I had built the expected table in my local Japan time.

from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
 
utc = datetime(2026, 9, 6, 4, 31, 12, tzinfo=ZoneInfo("UTC"))        # a ledger line
jst = datetime(2026, 9, 6, 4, 30,     tzinfo=ZoneInfo("Asia/Tokyo"))  # an expected slot
 
print(abs(utc - jst))                            # 9:01:12
print(abs(utc - jst) <= timedelta(minutes=20))   # False - everything looks missing
 
# And if you simply drop the timezone to make the comparison work
print(abs(utc.replace(tzinfo=None) - jst.replace(tzinfo=None)))   # 0:01:12

The awkward part is that these two mistakes fail in opposite directions. Compare them timezone-aware and you are nine hours off, so every slot reads as a gap. Drop the timezone because that felt easier and the difference looks like one minute twelve seconds, so every slot matches. The second one shows no symptom at all, which makes it far more dangerous.

The fix, once decided, is plain. Write started_at with its timezone attached, and state the same timezone when you build the expected table. In this situation I would recommend keeping the comparison on one side and pushing local time out to display only.

# Build the expected table with an explicit timezone too
from zoneinfo import ZoneInfo
TZ = ZoneInfo("Asia/Tokyo")
days = [datetime(2026, 9, 4, tzinfo=TZ), datetime(2026, 9, 5, tzinfo=TZ)]

One more thing I only noticed after this went into production use: slots that sit near midnight, an 23:50 job for instance, slip through if the audit only looks at the current date. I now always include the previous day in the range.

Point the alert at the gap, not only at the failure

One last thing about how it rings. I did not remove my failure alerts. I left them in place and added gap alerts to a different destination. A failure is a signal to go read that run; a gap is a signal to doubt the machinery — different in kind, and easier to act on when they are not mixed together.

Signal	What catches it	Where I look first
The run failed	Exit code and in-run exceptions	That run's log
The run vanished midway	A start line with no end line	Resources and timeouts
The run never happened	Reconciliation with the expected table	Schedule definition and the runner

Agent definitions and defaults shift quietly with every update, inheritance of surrounding configuration included, and the change itself does not announce itself. That is why I like keeping a guard at the entrance alongside the reconciliation at the exit — I wrote up the entrance side in the work-root guard I run before letting an agent near several repositories. For the case where a misspelled setting passes without a word, there is a custom tool that says any_of loses its constraint without ever erroring.

One step for tomorrow

Pick a single scheduled job and add one ledger line at the moment it starts. The reconciliation can come later.

For me, having gaps surface in three lines has cut down the time I spend imagining what might be going on overnight. Thank you for reading this far.

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.