Designing Antigravity Retry for Production — Idempotency, Backoff, Cost Ceilings, and Failure Triage

Exponential Backoff with Jitter — Don't Retry Into the Storm

"Failed? Retry immediately." is a recipe for attacking your own dependencies during an outage. LLM APIs in particular respond to pressure with rate limits, and an immediate retry just widens the wound. The well-known fix is exponential backoff, ideally combined with jitter to prevent synchronized reconnect storms. AWS's classic write-up on this pattern, Exponential Backoff and Jitter, is still the clearest summary.

// src/retry/backoff.ts
// Exponential backoff + full jitter + max attempts + max delay.
// Apply this only to errors we have classified as retry-eligible.
 
export type RetryOptions = {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  shouldRetry: (err: unknown, attempt: number) => boolean;
  onRetry?: (err: unknown, attempt: number, delayMs: number) => void;
};
 
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
 
export async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  opts: RetryOptions,
): Promise<T> {
  let lastError: unknown;
  for (let attempt = 1; attempt <= opts.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      if (attempt === opts.maxAttempts || !opts.shouldRetry(err, attempt)) {
        throw err;
      }
      // Full jitter: random in [0, base * 2^attempt], capped by maxDelayMs.
      const expo = Math.min(opts.baseDelayMs * 2 ** attempt, opts.maxDelayMs);
      const delay = Math.floor(Math.random() * expo);
      opts.onRetry?.(err, attempt, delay);
      await sleep(delay);
    }
  }
  throw lastError;
}
 
// Usage
await retryWithBackoff(
  () => fetch("https://api.example.com/data").then((r) => r.json()),
  {
    maxAttempts: 5,
    baseDelayMs: 500,
    maxDelayMs: 30_000,
    shouldRetry: (err) => isTransientError(err), // see "Failure Triage" below
    onRetry: (err, attempt, delay) => {
      console.log(`retry attempt=${attempt} delay=${delay}ms error=${String(err)}`);
    },
  },
);

The expected behavior is straightforward: success returns immediately, retry-eligible failures wait a jittered exponential delay and try again, and we throw once attempts are exhausted or the error becomes non-retryable. The onRetry hook is where you should wire logging and metrics — leaving it empty in production is how incidents become invisible.

Why I Reach for Full Jitter First

There are fancier jitter strategies — equal jitter, decorrelated jitter — and each has a fan base. For a personal Antigravity agent, full jitter (uniform random in [0, max]) is the right default because it is trivial to implement, spreads the reconnection thundering-herd the most, and is easy to reason about. Start there, measure, and only move to something more complex if you see a concrete problem the simpler approach cannot solve.

Measuring the Four Backoff Strategies Instead of Trusting Folklore

"Add jitter" is advice everyone repeats, including me a few paragraphs ago. But how much does it actually buy you? I did not have a satisfying answer until I simulated it, so here is a small harness you can run yourself. It models forty agents hitting a rate-limited endpoint at the same moment, using a virtual clock so two hundred runs finish in well under a second.

The setup: 40 concurrent workers, a token bucket with capacity 8 refilling one token every 250ms, baseDelayMs of 500, a 30-second cap, and maxAttempts of 6. Results are averaged over 200 seeded runs.

// retry-sim.mjs — compare backoff strategies on a virtual clock. No dependencies.
// Run with: node retry-sim.mjs
 
function mulberry32(a) {
  return function () {
    a |= 0; a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
 
function simulate(strategy, seed) {
  const rnd = mulberry32(seed);
  const WORKERS = 40, CAPACITY = 8, REFILL_MS = 250;
  const BASE = 500, MAX_DELAY = 30_000, MAX_ATTEMPTS = 6;
 
  let tokens = CAPACITY, lastRefill = 0;
  const queue = Array.from({ length: WORKERS }, (_, id) => ({ t: 0, id, attempt: 0 }));
  let attempts = 0, ok = 0, fail = 0;
  const completedAt = [];
 
  while (queue.length) {
    queue.sort((a, b) => a.t - b.t);
    const e = queue.shift();
 
    const refill = Math.floor((e.t - lastRefill) / REFILL_MS);
    if (refill > 0) {
      tokens = Math.min(CAPACITY, tokens + refill);
      lastRefill += refill * REFILL_MS;
    }
 
    e.attempt++; attempts++;
    if (tokens > 0) { tokens--; ok++; completedAt.push(e.t); continue; }
    if (e.attempt >= MAX_ATTEMPTS) { fail++; continue; }
 
    const exp = Math.min(MAX_DELAY, BASE * 2 ** (e.attempt - 1));
    const delay =
      strategy === "fixed" ? BASE
      : strategy === "exponential" ? exp
      : strategy === "equal-jitter" ? exp / 2 + rnd() * (exp / 2)
      : rnd() * exp; // full-jitter
    queue.push({ t: e.t + delay, id: e.id, attempt: e.attempt });
  }
 
  completedAt.sort((a, b) => a - b);
  return {
    attempts, ok, fail,
    median: completedAt[Math.floor(completedAt.length / 2)],
    p95: completedAt[Math.floor(completedAt.length * 0.95)],
  };
}
 
for (const s of ["fixed", "exponential", "equal-jitter", "full-jitter"]) {
  let A = 0, O = 0, F = 0, M = 0, P = 0;
  const RUNS = 200;
  for (let r = 0; r < RUNS; r++) {
    const x = simulate(s, 1000 + r);
    A += x.attempts; O += x.ok; F += x.fail; M += x.median; P += x.p95;
  }
  console.log(
    `${s.padEnd(14)} attempts=${(A / RUNS).toFixed(1)} completion=${((100 * O) / (O + F)).toFixed(1)}% ` +
    `median=${(M / RUNS / 1000).toFixed(2)}s p95=${(P / RUNS / 1000).toFixed(2)}s`,
  );
}

Here is what came out on my machine. Median and p95 cover only the tasks that actually completed.

The number worth staring at is the 45% completion rate for fixed delay. Its median of 0.50s looks like the fastest row in the table, but that median only describes the lucky half that squeezed through early. The other half burned all six attempts at 500ms intervals and never got a token. If you watch average latency on a dashboard, that entire population is invisible to you — they show up as errors somewhere else, hours later, in a support ticket.

Plain exponential backoff recovers the completion rate to 95%, but its p95 stretches to 15.50s, because every worker wakes up at the same instant and recreates the same congestion it just backed off from. Equal jitter trades a slightly later median for a perfect completion rate. Full jitter keeps the tail shortest at 10.15s.

That tail is why full jitter is my personal default for solo projects: it is the difference a user actually feels. But the honest reading of this table is that neither is universally correct. An overnight batch where only "everything finished" matters is better served by equal jitter's completeness. An interactive task running in the background is better served by full jitter's short tail. The point is to pick based on whether your workload cares more about completion rate or worst-case wait — and to have a number in front of you when you decide.

Swap WORKERS and REFILL_MS for the limits of whatever API you are actually calling, and the harness becomes a model of your own environment. Ten minutes with it turns questions like "should maxAttempts be 6 or 8?" from a shrug into a decision you can defend in code review.

Cost Ceilings — Catch the Runaway Before It Drains Your Account

A runaway agent burning tokens overnight is not a hypothetical. I have lived through a "5× expected spend" morning and it was entirely my fault for not putting a ceiling in place. Because Antigravity agents call external LLM APIs, the right unit for the ceiling is not "number of retries" — it is "tokens spent."

The design is three ideas:

• Declare a token budget per task up front
• Track cumulative usage on every retry and abort on overrun
• Trigger alerts and drop back to the last checkpoint when the ceiling hits

// src/retry/budget.ts
// Per-task token budget wired into the agent loop.
// Extract `usage` from each LLM response and abort when the total crosses the ceiling.
 
export class BudgetExceeded extends Error {
  constructor(
    public readonly used: number,
    public readonly budget: number,
  ) {
    super(`budget exceeded: used=${used} budget=${budget}`);
  }
}
 
export class TokenBudget {
  private used = 0;
 
  constructor(
    private readonly budget: number,
    private readonly onWarn?: (used: number, budget: number) => void,
  ) {}
 
  track(usage: { prompt_tokens: number; completion_tokens: number }) {
    this.used += usage.prompt_tokens + usage.completion_tokens;
    if (this.used > this.budget) {
      throw new BudgetExceeded(this.used, this.budget);
    }
    // Soft alert at 80% — hook into Slack, PagerDuty, or a metrics counter.
    if (this.used > this.budget * 0.8) {
      this.onWarn?.(this.used, this.budget);
    }
  }
 
  remaining(): number {
    return Math.max(0, this.budget - this.used);
  }
}
 
// Usage inside the agent loop
const budget = new TokenBudget(200_000, (used, total) => {
  console.warn(`budget at 80%: ${used}/${total}`);
});
 
for (const step of plannedSteps) {
  try {
    const response = await callLLM(step.prompt);
    budget.track(response.usage);
    applyResult(response);
  } catch (err) {
    if (err instanceof BudgetExceeded) {
      await notifyHuman("budget exceeded, human review required", { err });
      await saveCheckpoint(step);
      throw err; // stop the whole loop
    }
    throw err;
  }
}

Expected behavior: once usage crosses 80% of the ceiling a warning fires, once it crosses 100% the loop aborts, and a checkpoint plus alert reaches a human. The worst-case blast radius is now capped at one budget multiple rather than "as much as the credit card allows."

How to Size a Budget Sanely

If you are doing this for the first time, these rules of thumb work well:

• Small tasks (single-file edits, doc updates): 10,000 to 30,000 tokens
• Medium tasks (multi-file refactors): 50,000 to 150,000 tokens
• Large tasks (feature scaffolds, migrations): 300,000 to 500,000 tokens

If you are tempted to go higher, that is usually a signal that the task is too big to hand to an agent in one shot. Either split it with Planning Mode or take it yourself. The splitting heuristic is covered in more depth in my piece on Antigravity Planning Mode strategies for large projects, which pairs nicely with the budgets above.

Failure Triage — Not All Errors Deserve a Retry

The biggest cause of runaway retries is treating every failure identically. In reality, agent errors fall into three buckets:

• Transient: network blips, rate limits, timeouts — almost always recover on retry
• Permanent: auth failures, permission errors, validation failures, unknown model names — the same result every time
• Ambiguous: 5xx errors, JSON parse failures, partial completions — retry under a tight cap, then escalate

Encoding this triage into code makes the "should we retry" decision unambiguous.

// src/retry/triage.ts
// Classify an error as transient, permanent, or ambiguous.
// Grow this function over time — every new error shape you see should add a rule.
 
export type ErrorClass = "transient" | "permanent" | "ambiguous";
 
export function classifyError(err: unknown): ErrorClass {
  if (!(err instanceof Error)) return "ambiguous";
  const msg = err.message.toLowerCase();
 
  // --- Auth, permission, and input errors are permanent ---
  if (/unauthorized|forbidden|invalid_api_key|authentication/.test(msg)) {
    return "permanent";
  }
  if (/validation|invalid request|bad request|schema/.test(msg)) {
    return "permanent";
  }
  if (/model_not_found|unknown model/.test(msg)) {
    return "permanent";
  }
 
  // --- Rate limits and transient network errors are retryable ---
  if (/rate.?limit|too many requests|429/.test(msg)) {
    return "transient";
  }
  if (/timeout|econnreset|etimedout|network/.test(msg)) {
    return "transient";
  }
  if (/503|service unavailable|overloaded/.test(msg)) {
    return "transient";
  }
 
  // --- Ambiguous ones get exactly one retry ---
  if (/500|internal server|json|parse/.test(msg)) {
    return "ambiguous";
  }
 
  return "ambiguous";
}
 
// Plugging the triage into the retry loop's shouldRetry
export function shouldRetryFromTriage(err: unknown, attempt: number): boolean {
  const klass = classifyError(err);
  if (klass === "permanent") return false;
  if (klass === "transient") return attempt < 5;
  if (klass === "ambiguous") return attempt < 3; // cap 3 — measured below
  return false;
}

This function is never done. Every new failure mode is an excuse to add another regex or two. The reward for keeping it up to date is a triage tuned to your real production traffic — no off-the-shelf classifier will match that.

Measuring What Triage Actually Buys You

That permanent errors are wasted spend is obvious enough. How much you save, and what you give up in exchange, is not — and I had never checked. My ambiguous limit sat at "retry once" for months purely because it sounded prudent.

So I simulated it. The population is 500 tasks that all failed on their first attempt, split into transient 60%, ambiguous 25%, permanent 15%, at 3,700 tokens per attempt (3,000 prompt + 700 completion). A regex classifier is never perfect, so the model misses 10% of permanent errors and 5% of transient ones, with every miss falling through to ambiguous.

// triage-cost.mjs — how triage and the ambiguous cap move cost and completion
// Run with: node triage-cost.mjs (no dependencies)
 
function mulberry32(a) {
  return function () {
    a |= 0; a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
 
const TASKS = 500;
const TOKENS = 3_700;                                   // one attempt = 3,000 prompt + 700 completion
const MIX = { transient: 0.60, ambiguous: 0.25, permanent: 0.15 };
const RECALL = { permanent: 0.90, transient: 0.95, ambiguous: 1.0 }; // what the regex triage misses
const RECOVER = { transient: 0.65, ambiguous: 0.35, permanent: 0 };  // recovery chance per attempt
 
const drawKind = (rnd) => {
  const x = rnd();
  if (x < MIX.transient) return "transient";
  if (x < MIX.transient + MIX.ambiguous) return "ambiguous";
  return "permanent";
};
const classify = (kind, rnd) => (rnd() < RECALL[kind] ? kind : "ambiguous");
 
function cap(policy, klass) {
  if (policy.ambiguous === null) return 5;              // no triage: five attempts for everything
  if (klass === "permanent") return 1;
  if (klass === "transient") return 5;
  return policy.ambiguous;
}
 
function simulate(policy, seed) {
  const rnd = mulberry32(seed);
  let tokens = 0, wasted = 0, ok = 0, attempts = 0;
  for (let i = 0; i < TASKS; i++) {
    const kind = drawKind(rnd);
    const klass = classify(kind, rnd);
    const limit = cap(policy, klass);
    let spent = 0, done = false;
    for (let a = 1; a <= limit; a++) {
      spent += TOKENS; attempts++;
      if (a > 1 && rnd() < RECOVER[kind]) { done = true; break; }
    }
    tokens += spent;
    if (done) ok++; else wasted += spent;                // spend on tasks that never succeeded
  }
  return { tokens, wasted, ok, attempts };
}
 
const POLICIES = [
  { name: "no triage", ambiguous: null },
  { name: "triage + ambiguous cap 2", ambiguous: 2 },
  { name: "triage + ambiguous cap 3", ambiguous: 3 },
  { name: "triage + ambiguous cap 4", ambiguous: 4 },
];
const RUNS = 300;
for (const p of POLICIES) {
  let T = 0, W = 0, O = 0, A = 0;
  for (let r = 0; r < RUNS; r++) {
    const x = simulate(p, 2000 + r);
    T += x.tokens; W += x.wasted; O += x.ok; A += x.attempts;
  }
  const tokens = T / RUNS, wasted = W / RUNS, ok = O / RUNS, att = A / RUNS;
  console.log(
    `${p.name.padEnd(26)} tokens=${Math.round(tokens).toLocaleString().padStart(9)} ` +
    `wasted=${((100 * wasted) / tokens).toFixed(1)}% ` +
    `completed=${((100 * ok) / TASKS).toFixed(1)}% ` +
    `per-success=${Math.round(tokens / ok).toLocaleString()}tok ` +
    `attempts=${(att / TASKS).toFixed(2)}`,
  );
}

Here is what came back on my machine. "Per success" is total tokens divided by completed tasks, which is the number closest to what actually lands on the invoice.

The first number that stops you is the 33.0% wasted share with no triage. A third of everything you spend goes to tasks that never finish, and most of that is permanent errors being politely asked five times whether the API key has started working yet.

What I did not expect was how much the ambiguous cap matters on its own. Capping it at 2 — a single retry — cuts total tokens by 30.2%, but completion falls 12.7 points with it. Loosening the cap to 3 brings completion back to 73.2%, and cost per success actually improves slightly, from 11,950 to 11,856 tokens. A cap of 4 lands in the same place.

The saving comes from dismissing permanent errors on the first attempt, not from starving ambiguous ones. Cut the attempts that cannot succeed, not the attempt count. Squeeze too hard and the tokens you thought you saved leave with the tasks that were one attempt away from passing.

I have moved the ambiguous limit in shouldRetryFromTriage from 2 to 3 because of this run. Your own failure mix and recovery rates will differ, so I would rather you refill MIX and RECOVER from your production logs and run it yourself than take my defaults on faith. Change the population and the right default changes with it.

Checkpoints — Retry Without Redoing the First 95 Steps

Even with good triage and a hard budget, losing a 100-step agent task on step 95 is a waste of time and money. Checkpoints fix this by persisting "we got this far" so retry can resume rather than restart. Antigravity's Manager Surface does some of this automatically, but any agent you build yourself needs explicit support.

Three design rules:

• Persist "already-completed step IDs" after each success
• On retry, start from the first incomplete step — never from zero
• Side-effecting steps must still run through the idempotency guard above

// src/retry/checkpoint.ts
// Smallest useful "resumable agent" loop.
// Swap fs for KV, Firestore, or any durable store in production.
 
import fs from "node:fs/promises";
 
type Step = { id: string; run: () => Promise<void> };
 
type Checkpoint = {
  completed: string[];
  updatedAt: string;
};
 
export async function runResumable(
  steps: Step[],
  checkpointPath: string,
): Promise<void> {
  let cp: Checkpoint;
  try {
    cp = JSON.parse(await fs.readFile(checkpointPath, "utf-8"));
  } catch {
    cp = { completed: [], updatedAt: new Date().toISOString() };
  }
 
  for (const step of steps) {
    if (cp.completed.includes(step.id)) {
      console.log(`skip already-completed step: ${step.id}`);
      continue;
    }
    try {
      await step.run();
      cp.completed.push(step.id);
      cp.updatedAt = new Date().toISOString();
      await fs.writeFile(checkpointPath, JSON.stringify(cp, null, 2));
    } catch (err) {
      // Flush the checkpoint on failure so the next run resumes here.
      await fs.writeFile(checkpointPath, JSON.stringify(cp, null, 2));
      throw err;
    }
  }
}

After a failure, the next invocation should skip every completed step and pick up where things went wrong. For larger agent plans this is the single most valuable change you can make — retry cost approaches zero as the checkpoint matures. For a deeper walkthrough of the pattern inside Antigravity itself, see Antigravity Checkpoints and Rollback Mastery.

Observability — Retry in the Dark Is Retry You Cannot Trust

Retries run where nobody is watching, which makes observability non-optional. The minimum viable kit:

• Structured logs: attempt number, delay, error class, tokens consumed
• Metrics: retry.count, retry.success, retry.fail, retry.budget.exceeded
• Alerts: budget exceeded, consecutive failures, any permanent classification
• Distributed traces: one span per attempt, linked to the outer Antigravity run

You can get most of this from Langfuse, Helicone, or your own OpenTelemetry pipeline. For small personal projects I run Cloudflare Workers Analytics Engine plus a Slack Webhook and call it done. The goal is never a beautiful dashboard — it is being able to notice a runaway within five minutes of it starting. For a hands-on walkthrough of wiring this into Antigravity specifically, see Antigravity × Langfuse Observability Runbook.

Team Hygiene — Write a RETRY_POLICY.md Before You Write the Retry

Even if you are a solo developer, "Retry by vibes" is a mistake. Drop a RETRY_POLICY.md at the root of the project and fill in five sections:

• maxAttempts per task category
• baseDelayMs, maxDelayMs, and jitter strategy
• Token budgets per task size (small, medium, large)
• Triage rules — which error messages count as permanent
• Notification routing and human runbook for budget-exceeded or permanent errors

Three Failure Modes You Will Probably Hit

Even with all four pillars in place, there are three recurring traps I keep falling into. Calling them out may save you a night.

1. Forgetting the Idempotency Guard on a New Side Effect

The most common production incident. Adding a new side-effecting action — send_email, create_payment, post_to_slack — without wrapping it in the guard leads to duplicate events the moment a retry fires. A useful habit: every new action needs a unit test that calls it twice and asserts the inner body ran exactly once.

2. Retrying Permanent Errors Until the Budget Is Gone

Auth failures, permission errors, and invalid model names never recover from retries. If they are classified as transient, the agent happily loops forever with billing. Make sure classifyError has CI coverage on unauthorized, forbidden, and invalid_api_key and that shouldRetry returns false for them.

3. A Budget Breach That Never Pages Anyone

The subtle one. You wired onWarn to a Slack webhook that has since expired, or to a dev channel nobody watches. Every alert pipeline needs a heartbeat. A simple "always-warn" dummy task that fires every 24 hours is the cheapest way to verify the notification chain is still alive.

4. Forgetting That Partial Completions Are a Triage Category Too

This one took me embarrassingly long to notice. The agent runs, returns something that looks like a valid response, but actually only solved half the task — say, it wrote the new function but forgot to update the tests, or it deleted the old file without creating the new one. If you treat "valid-looking response" as "success," the retry loop never kicks in, and the failure propagates silently downstream. The fix is to add a lightweight post-condition check for every step — even a single assert on expected output shape — and treat a failed check as an ambiguous-class error. One extra line of validation per step is cheap insurance against this shape of mistake.

A Walkthrough: Wiring the Pieces Together

It helps to see the whole system in one place, because the individual snippets above can feel disconnected. Here is a minimal end-to-end agent loop that uses all four pillars at once. I have deliberately kept it to what fits on a single screen.

// src/agents/resilient-loop.ts
// The four pillars composed: idempotency, backoff, budget, triage, plus checkpoints.
 
import { IdempotencyGuard } from "../retry/idempotency";
import { retryWithBackoff } from "../retry/backoff";
import { TokenBudget } from "../retry/budget";
import { shouldRetryFromTriage, classifyError } from "../retry/triage";
import { runResumable } from "../retry/checkpoint";
 
const guard = new IdempotencyGuard();
const budget = new TokenBudget(200_000, (used, total) => {
  console.warn(`budget at 80%: ${used}/${total}`);
});
 
type TaskStep = { id: string; action: string; payload: unknown };
 
async function runStep(step: TaskStep) {
  return retryWithBackoff(
    () =>
      guard.run(step.action, step.payload, async () => {
        const response = await callLLM({ action: step.action, payload: step.payload });
        budget.track(response.usage);
        return response;
      }),
    {
      maxAttempts: 5,
      baseDelayMs: 500,
      maxDelayMs: 30_000,
      shouldRetry: (err, attempt) => {
        // Permanent errors and budget breaches escape the retry loop.
        if (classifyError(err) === "permanent") return false;
        if (err instanceof Error && err.message.includes("budget exceeded")) return false;
        return shouldRetryFromTriage(err, attempt);
      },
      onRetry: (err, attempt, delay) => {
        console.log(`[${step.id}] retry ${attempt} in ${delay}ms: ${String(err)}`);
      },
    },
  );
}
 
export async function runTask(steps: TaskStep[]) {
  const adapted = steps.map((s) => ({ id: s.id, run: () => runStep(s).then(() => {}) }));
  await runResumable(adapted, `.agent/checkpoint-${Date.now()}.json`);
}

Reading this top to bottom, you can see each pillar in its own layer: the guard prevents double side effects, the backoff handles transient failures, the budget prevents cost runaway, the triage decides what is worth retrying, and the resumable wrapper lets us pick up exactly where we left off. No single piece is exotic. What makes it production-grade is that they are all present, and that a team reading the file can trace each decision back to an explicit rule.

How the System Behaves Under Real Failures

Spend a few minutes imagining the loop under concrete failure scenarios. A 429 rate limit arrives mid-task: triage says transient, backoff waits with jitter, the next attempt succeeds, the budget ticks up. An auth token has rotated: triage says permanent, the loop throws immediately, nothing is retried, a human wakes up to a single clean error. A runaway chain of "the model keeps producing slightly different malformed JSON": triage says ambiguous, a couple of retries are allowed, then we give up — the budget never gets anywhere near the ceiling. A truly massive prompt accidentally gets committed: the budget ceiling trips, a checkpoint drops, a Slack alert fires, and the loop halts at the end of the step we were in. Every one of these scenarios is boring, predictable, and — most importantly — bounded.

What Changes Once Retry Is Designed, Not Improvised

There is a quiet benefit to all of this that I did not expect when I first started treating retry as a system. The act of writing the policy and the code forces you to think about your agent's blast radius. You start asking: what is the worst this action can do? Is my ceiling actually sized correctly for the job I am asking it to do? Do I actually trust this prompt enough to let it loose overnight? Those questions surface design issues long before they become incidents. In a real sense, the best bug you can find is the one you find while writing the retry policy, because that is the cheapest possible moment to fix it.

The second shift is psychological. Once I had the four pillars in place across my own projects, overnight agent runs stopped being a source of anxiety. I still check the logs in the morning, but I check them the way you check a thermostat — not the way you check a suspected leak. That change is worth the weekend it takes to implement, several times over.

Four pillars, a handful of code snippets, and a short policy document. That is the whole retry system. If you only adopt one of them today, pick the idempotency guard and wrap a single side-effecting action in your Antigravity agent with it. The duplicate-email incident — the one that hurts the most when it happens — becomes dramatically less likely. From there, the rest of the architecture layers on naturally: backoff, budget, triage, checkpoint. In a week or two of incremental work, your agent reliability will not feel the same, and the retry policy you wrote on day one becomes the living document your future self and your teammates rely on every time you extend the agent's responsibilities.

Antigravity, designed properly, is an agent you can genuinely leave running overnight. The steering is still our responsibility, but once the retry architecture is in place it becomes quiet, patient, and — most importantly — boring in the best way. Boring is what reliability looks like. Boring is what sleep looks like. And in a field where "the agent did something unexpected" still makes headlines, boring is a competitive advantage worth engineering for.

If you take away nothing else, take away this: treat every Retry button press as a signal that something in your design is missing, and fix the design rather than the symptom. The button is for debugging; the four pillars are for production. Once you have internalized that distinction, the rest of this article will feel less like a recipe and more like a checklist you already knew you needed.