◉ANTIGRAVITY LABJP
Articles/Agents & Manager
◈ Agents & Manager/2026-03-30Advanced

Antigravity × Durable Execution: Designing Fault-Tolerant Long-Running AI Tasks

An implementation guide for putting Antigravity agents to work on long-running jobs using Durable Execution — covering checkpointing, idempotency, and automatic retries, plus a compare-and-swap checkpoint store, fault-injection testing, and seven production pitfalls the official docs never spell out.

antigravity456durable-execution2ai-agent18typescript28fault-toleranceworkflow57
✦ Premium Article

One morning the pipeline that aggregates yesterday's AdMob revenue died ten minutes from the finish line — an hour and a half of work, gone. I gave up on the spot and re-kicked the same job from my phone on the train into the office. I run a family of wallpaper and ambient apps as an indie developer, and every morning their revenue gets reconciled between AdMob, App Store Connect, and Google Play. Before I introduced Durable Execution, the "almost made it" failure mode was just brutal: AdMob rate limits, Cloudflare Workers timeouts, the occasional Supabase blip — each of them meant starting from scratch.

Durable Execution is what turns "a crash shouldn't cost you the whole run" from a wish into something the code actually enforces. Since I started handing long-running jobs to Antigravity agents, I can no longer run nightly work without it.

This guide starts from the minimal implementation, then builds out what production actually demands: a checkpoint store that detects concurrent runs, a test that breaks resumption on purpose to prove it works, and seven pitfalls the official docs never spell out — all in code I run every day.

The Three Principles That Make Durable Execution Work

Before any code, the three principles. Everything else falls out of these — and they map cleanly onto Antigravity agent design too.

Checkpointing for State Persistence

Every time a workflow step completes successfully, its result is saved to durable storage — a database, a queue, Cloudflare KV, anything that survives a restart. If the process crashes, the next run resumes from the most recent checkpoint instead of replaying every step. For my AdMob aggregation pipeline, this single change shrank average recovery time from 47 minutes to 4 minutes — roughly an 11x improvement.

Idempotency by Design

Operations must produce the same result whether they're called once or ten times. Without this, retries lead to duplicate writes, duplicate emails, double-charged customers. Missing idempotency in payment or notification paths is the kind of bug you only catch by getting publicly embarrassed.

Automatic Retry with Backoff

Transient failures (network timeouts, rate limits) should be retried with exponential backoff. Permanent failures should bubble up. The distinction matters: for AdMob, UNAVAILABLE and RESOURCE_EXHAUSTED are retryable, but PERMISSION_DENIED should fail fast — don't burn quota on something that will never succeed.

Implementing the Minimal Durable Workflow in Antigravity

Let's build a TypeScript Durable Execution skeleton that fetches data from an external API, transforms it, and persists the result.

// durable-workflow.ts
// Minimal implementation of the Durable Execution pattern
 
interface WorkflowState {
  currentStep: string;
  completedSteps: string[];
  data: Record<string, unknown>;
  retryCount: number;
  lastCheckpoint: string;
}
 
class DurableWorkflow {
  private state: WorkflowState;
  private storePath: string;
 
  constructor(workflowId: string) {
    this.storePath = `./workflow-state/${workflowId}.json`;
    this.state = this.loadState();
  }
 
  // Restore state from a checkpoint
  private loadState(): WorkflowState {
    try {
      const fs = require("fs");
      if (fs.existsSync(this.storePath)) {
        const saved = JSON.parse(fs.readFileSync(this.storePath, "utf-8"));
        console.log(`Resuming from checkpoint: ${saved.lastCheckpoint}`);
        return saved;
      }
    } catch (e) {
      console.warn("No checkpoint found, starting fresh");
    }
    return {
      currentStep: "init",
      completedSteps: [],
      data: {},
      retryCount: 0,
      lastCheckpoint: "",
    };
  }
 
  // Persist a checkpoint
  private saveCheckpoint(stepName: string): void {
    const fs = require("fs");
    const path = require("path");
    fs.mkdirSync(path.dirname(this.storePath), { recursive: true });
    this.state.lastCheckpoint = stepName;
    this.state.completedSteps.push(stepName);
    fs.writeFileSync(this.storePath, JSON.stringify(this.state, null, 2));
    console.log(`Checkpoint saved: ${stepName}`);
  }
 
  // Run a step idempotently
  async executeStep<T>(
    stepName: string,
    fn: () => Promise<T>,
    maxRetries = 3
  ): Promise<T> {
    // Skip completed steps
    if (this.state.completedSteps.includes(stepName)) {
      console.log(`Skipping completed step: ${stepName}`);
      return this.state.data[stepName] as T;
    }
 
    this.state.currentStep = stepName;
 
    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        const result = await fn();
        this.state.data[stepName] = result;
        this.saveCheckpoint(stepName);
        return result;
      } catch (error) {
        const waitMs = Math.min(1000 * Math.pow(2, attempt), 30000);
        console.error(
          `Step "${stepName}" failed (attempt ${attempt}/${maxRetries}):`,
          error
        );
        if (attempt === maxRetries) throw error;
        console.log(`Retrying in ${waitMs}ms...`);
        await new Promise((r) => setTimeout(r, waitMs));
      }
    }
    throw new Error(`Step "${stepName}" exhausted all retries`);
  }
}
 
// Example: a multi-step data transformation pipeline
async function runDataPipeline() {
  const workflow = new DurableWorkflow("data-pipeline-001");
 
  // Step 1: fetch (resumes from checkpoint on failure)
  const rawData = await workflow.executeStep("fetch-data", async () => {
    const res = await fetch("https://api.example.com/large-dataset");
    return res.json();
    // Expected output: { records: [...], total: 10000 }
  });
 
  // Step 2: transform
  const transformed = await workflow.executeStep("transform", async () => {
    return rawData.records.map((r: any) => ({
      id: r.id,
      normalized: r.value.toLowerCase().trim(),
      processedAt: new Date().toISOString(),
    }));
    // Expected output: [{ id: "1", normalized: "...", processedAt: "..." }, ...]
  });
 
  // Step 3: batch save
  const result = await workflow.executeStep("save-batch", async () => {
    const batchSize = 100;
    let saved = 0;
    for (let i = 0; i < transformed.length; i += batchSize) {
      const batch = transformed.slice(i, i + batchSize);
      await saveBatch(batch); // idempotent upsert
      saved += batch.length;
    }
    return { totalSaved: saved };
    // Expected output: { totalSaved: 10000 }
  });
 
  console.log("Pipeline complete:", result);
}
 
async function saveBatch(batch: any[]) {
  // Idempotent upsert keyed by record ID
  console.log(`Saving batch of ${batch.length} records`);
}
 
runDataPipeline().catch(console.error);

Each step saves a checkpoint to the filesystem on success. When the process restarts, completed steps are skipped automatically and execution resumes from where it stopped. In a serverless environment local FS is ephemeral — and there is a second, nastier problem lurking behind that one. The next section deals with both.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
✦Three real AdMob pipeline incidents from a 04:30 nightly job — recovery time cut from 47 minutes to 4 minutes
✦A compare-and-swap checkpoint store that detects concurrent runs (Cloudflare KV implementation, plus how to handle LeaseLostError) and a fault-injection test that mechanically proves resume correctness
✦Seven production pitfalls the official docs never spell out (checkpoint bloat, double notifications, rate-limit blast radius, partial recovery, time drift, cold start, monitoring blind spots)
✦A four-step prompt template for getting Antigravity to generate durable workflows that actually survive real failure modes
Secure payment via Stripe · Cancel anytime
✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $15 for lifetime access
View Membership →

Related Articles

◈ Agents & Manager2026-03-26
Antigravity Multi-Agent Workflow Guide — Accelerate Development with Multiple AIs
Learn how to use Antigravity's multi-agent features to dramatically speed up your development workflow. From Manager Surface basics to advanced agent coordination patterns.
◈ Agents & Manager2026-09-04
Snapshot Your Agents' Effective Scope So You Notice the Day a Default Changes
Antigravity 1.1.25 made Markdown-defined custom agents inherit the surrounding skills, rules, and subagents by default. Here is the snapshot-and-diff routine I now use to check whether an agent I meant to keep narrow quietly got wider after an update.
◈ Agents & Manager2026-07-15
The File Is Right There in ls, and Your Agent Still Can't Open It
The agent says the file does not exist. Your terminal says it does. After three days of blaming cloud sync, the answer turned out to be that one voiced consonant mark was never a single character. Detection script and a three-layer gate included.
📚RECOMMENDED BOOKS
Build a Large Language Model (From Scratch)
Sebastian Raschka
LLM Dev
Prompt Engineering for LLMs
Berryman & Ziegler
Prompting
AI Engineering
Chip Huyen
AI Eng
* Contains affiliate links