◉ANTIGRAVITY LABJP
Articles/Agents & Manager
◈ Agents & Manager/2026-06-29Advanced

When Background Agents Run Twice — Stopping Double Execution with Leases and Fencing Tokens

The same scheduled job fires from two machines at once and they overwrite each other's output. Here is how to stop that failure mode at the root in Antigravity 2.0 background agents, using leases and fencing tokens, with working code.

background agents3Antigravity419distributed lockfencing tokenoperations33

✦ Premium Article

One morning, an artifact that should have been freshly generated was half-overwritten with stale content. The logs told the story: the same scheduled job had fired from two machines at nearly the same minute, and one started writing before the other had finished.

As an indie developer at Dolice Labs, I run several blog operations on background agents. The intent was redundancy — while one machine sleeps, another picks up the work. But in the instant both were awake, both grabbed the same job. This is not a "forgot to take the lock" story. It happens even when you do take the lock. Let me walk through why, and how to stop it.

Why mutual exclusion alone does not stop double execution

The intuition is simple: take a lock at the start of the job, release it at the end. In the world of background agents, that premise quietly collapses.

A process can hold a lock and then stall for a long time — a long garbage-collection pause, an OS wake from sleep, a slow model call. Meanwhile the lock's TTL expires and another machine legitimately acquires it. The first machine wakes up still believing it owns the lock and begins writing. At that moment there are two lock holders in the world.

So the problem is not acquiring exclusivity; it is being unable to guarantee continued possession. Miss this distinction and every patch — longer TTLs, more heartbeats — only lowers the probability instead of eliminating it.

The idea of a lease

Reframe the lock as a lease: time-bounded ownership that is always assumed to expire. The holder must explicitly renew it before it lapses. The moment renewal stops, ownership is considered surrendered automatically.

The key property is that each time a lease is granted, the issued fencing token increases monotonically. Every acquisition produces a strictly larger token. That lets you distinguish an "old owner" from a "new owner" right before a write, using nothing but a numeric comparison.

AspectPlain mutual-exclusion lockLease + fencing token
Recovery from a stallThe old holder writes anywayThe old token is rejected at the write site
Ownership decisionRelies on "I think I hold it"Decided mechanically by number size
Effect of clock skewBreaks if TTL judgment driftsOrder is set by the token, not the clock
Required assumptionEveryone behaves honestlyThe write target can verify the token
✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
✦Understand exactly why 'I acquired the lock, so I'm safe' breaks down (GC pauses, sleep/wake, clock skew) — and why a plain mutual-exclusion lock cannot prevent double execution
✦A complete implementation of double-execution prevention with a lease plus a monotonically increasing fencing token (acquire, renew, expire, and write-side verification). The bash and Python are copy-paste ready
✦From the real experience of running scheduled jobs across two Macs as an indie developer and watching output get corrupted, a clear rule for where the verification gate must live
Secure payment via Stripe · Cancel anytime
✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

Related Articles

◈ Agents & Manager2026-09-06
With scheduled agents, I now look for runs that never happened before I look for failures
My run ledger showed a 100% success rate while the evening slot had not fired for two weeks. Here is the reconciliation I now run against an expected-fire table, with measured notes on cron expansion, exit codes, and how the ledger itself gets written.
◈ Agents & Manager2026-04-26
Designing Antigravity Agent Traces That Tell You Why It Failed — Observability in Practice
How to add OpenTelemetry spans to your own agent loop so a failure can be read in a few lines. Covers span structure, what to keep and drop in attributes, a failure_class taxonomy, a retry table, three dashboard views, cost attributes, and PII handling — with code that runs.
◈ Agents & Manager2026-11-19
Before an Agent Reads Your Issues: Splitting GitHub Actions Into a Reader Job and a Writer Job
Anyone can write an issue body. When an agent reads it inside GitHub Actions, split the workflow into a reader job with no write access and a writer job that never runs the agent, keep the body out of the shell, and check the agent's proposal with a small Python script. Working code included.