Record & Replay for Antigravity Agents — A Production Pattern to Reproduce Failures in 3 Minutes

The minimum viable shape — three boundaries are enough

Before any code, decide what to record. Over a month of operation, recording just these three boundaries lets me reproduce 90% of failures.

The reason time matters: Antigravity's retry and scheduling paths depend on Date.now(). If you don't pin time to the recorded value, Replay won't take the same branch.

Equally important is what not to record:

• Filesystem state (record it only as it crosses tool boundaries).
• Global process state (won't survive into another machine's Replay).
• Plaintext API keys or auth headers (covered later in the PII section).

The recording layer — a transparent middleware

Now the working code. Antigravity's SDK already routes everything through a fetch-based transport, so the cleanest approach is to wrap that transport with a thin Record/Replay switch.

// recorder.ts
import { createHash } from "node:crypto";
 
type Mode = "record" | "replay" | "off";
 
export interface Trace {
  runId: string;
  events: TraceEvent[];
}
 
export type TraceEvent =
  | { kind: "model_call"; t: number; hash: string; req: unknown; res: unknown }
  | { kind: "tool_call"; t: number; name: string; args: unknown; res: unknown }
  | { kind: "clock"; t: number; value: number }
  | { kind: "random"; t: number; value: number };
 
const MODE: Mode = (process.env.AGY_RECORDER_MODE as Mode) ?? "off";
let currentTrace: Trace | null = null;
let replayCursor = 0;
let replayTrace: Trace | null = null;
 
export function startRun(runId: string) {
  if (MODE === "record") currentTrace = { runId, events: [] };
}
 
export function endRun(): Trace | null {
  const t = currentTrace;
  currentTrace = null;
  return t;
}
 
export function loadReplay(trace: Trace) {
  replayTrace = trace;
  replayCursor = 0;
}
 
function hashOf(obj: unknown): string {
  return createHash("sha256").update(JSON.stringify(obj)).digest("hex").slice(0, 12);
}
 
export async function recordedModelCall<T>(
  req: unknown,
  exec: () => Promise<T>,
): Promise<T> {
  const t = Date.now();
  const hash = hashOf(req);
  if (MODE === "replay" && replayTrace) {
    const ev = replayTrace.events[replayCursor++];
    if (!ev || ev.kind !== "model_call" || ev.hash !== hash) {
      throw new Error(`Replay divergence at cursor ${replayCursor - 1}`);
    }
    return ev.res as T;
  }
  const res = await exec();
  currentTrace?.events.push({ kind: "model_call", t, hash, req, res });
  return res;
}
 
export async function recordedTool<T>(
  name: string,
  args: unknown,
  exec: () => Promise<T>,
): Promise<T> {
  const t = Date.now();
  if (MODE === "replay" && replayTrace) {
    const ev = replayTrace.events[replayCursor++];
    if (!ev || ev.kind !== "tool_call" || ev.name !== name) {
      throw new Error(`Replay divergence: expected ${name}`);
    }
    return ev.res as T;
  }
  const res = await exec();
  currentTrace?.events.push({ kind: "tool_call", t, name, args, res });
  return res;
}
 
export function now(): number {
  if (MODE === "replay" && replayTrace) {
    const ev = replayTrace.events[replayCursor++];
    if (!ev || ev.kind !== "clock") throw new Error("Replay clock divergence");
    return ev.value;
  }
  const t = Date.now();
  currentTrace?.events.push({ kind: "clock", t, value: t });
  return t;
}

Insert this layer as close to the agent's entry point as possible. You will need to refactor direct Date.now() calls into now(), but that pays for itself the moment Replay determinism jumps.

One nontrivial detail the docs don't really mention: for streamed model responses, record only the merged final JSON. Recording chunks balloons trace size 3–5× and ironically makes Replay less stable, because chunk boundaries shift run-to-run.

The replay harness — deterministic offline runs

Once recording works, you need a harness that feeds the trace back into the agent.

// replay-harness.ts
import { loadReplay, type Trace } from "./recorder";
import { runAgent } from "./agent";
import { readFileSync } from "node:fs";
 
async function main() {
  const path = process.argv[2];
  if (!path) throw new Error("usage: replay <trace.json>");
  process.env.AGY_RECORDER_MODE = "replay";
  const trace: Trace = JSON.parse(readFileSync(path, "utf-8"));
  loadReplay(trace);
  const firstReq = (trace.events.find((e) => e.kind === "model_call") as any).req;
  await runAgent(firstReq.messages);
}
main().catch((e) => {
  console.error("[replay]", e);
  process.exit(1);
});

Run it as bun run replay trace.json. Attach your debugger and you can step through the exact failing iteration, with the context window visible to you in your IDE.

Concretely: before Record & Replay, root-causing a failure took an average of 47 minutes. After, it takes around 3 minutes. "Cannot reproduce" was the single largest tax in the failure cycle.

Storage design — Cloudflare R2 and KV split

Traces accumulate. Across four sites running 16 agents per day for a month, my distribution looked like this:

R2 isn't great at key search on its own, so I keep runId → { R2 key, timestamp, failed flag } in KV as an index. R2 Class B operations are free under 1 GB/month, so my real-world cost for four parallel sites lands around ¥48 (about $0.30) per month — less than one AdMob tap. Cheap insurance.

For TTL, I split it: successful traces expire after 7 days, failed traces stay for 90. Successes rarely need to be revisited; failures often have a long tail where a similar one recurs weeks later.

// trace-store.ts (Cloudflare Workers)
export interface Env {
  TRACES_R2: R2Bucket;
  TRACES_KV: KVNamespace;
}
 
export async function putTrace(env: Env, trace: Trace, failed: boolean) {
  const key = `traces/${trace.runId}.json`;
  await env.TRACES_R2.put(key, JSON.stringify(trace), {
    httpMetadata: { contentType: "application/json" },
  });
  await env.TRACES_KV.put(
    `idx:${trace.runId}`,
    JSON.stringify({ key, failed, t: Date.now() }),
    { expirationTtl: failed ? 90 * 86400 : 7 * 86400 },
  );
}
 
export async function getTrace(env: Env, runId: string): Promise<Trace | null> {
  const idx = await env.TRACES_KV.get(`idx:${trace.runId}`, "json") as any;
  if (!idx) return null;
  const obj = await env.TRACES_R2.get(idx.key);
  return obj ? (await obj.json()) as Trace : null;
}

PII masking — enforce it at the choke point

I burned myself on this in the first prototype. Auth headers, model intermediate outputs containing email addresses, Stripe test price IDs in commit messages — all of it ended up in R2 before I noticed. I had to delete everything and start over.

The robust answer is one mandatory mask function on the write path.

// mask.ts
const SECRET_PATTERNS: RegExp[] = [
  /sk-[A-Za-z0-9_-]{20,}/g,
  /ghp_[A-Za-z0-9]{36}/g,
  /pk_live_[A-Za-z0-9]{20,}/g,
  /price_[A-Za-z0-9]{14,}/g,
  /[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/g,
];
 
export function mask<T>(value: T): T {
  const s = JSON.stringify(value);
  let masked = s;
  for (const p of SECRET_PATTERNS) masked = masked.replace(p, "<REDACTED>");
  masked = masked.replace(/"authorization"\s*:\s*"[^"]*"/gi, '"authorization":"<REDACTED>"');
  masked = masked.replace(/"cookie"\s*:\s*"[^"]*"/gi, '"cookie":"<REDACTED>"');
  return JSON.parse(masked);
}

Make putTrace strictly require mask(trace). If it is physically impossible to write an unmasked trace, you remove the temptation later to "just leave this one raw."

A subtler point: Replay never calls real APIs, so masked values rarely affect outcomes — except when a tool's behavior depends on the value itself (e.g. "look up an ID by email"). For those, mask the boundary but also pin a deterministic hash of the original value so branches stay stable. I noticed this in the first prototype and rewrote the masker accordingly.

The standard diagnosis flow — three minutes to a suspect

Once Record & Replay is in place, diagnosis becomes routine:

1. Get a runId from CI or your notification (in my case the Cowork scheduled-task log).
2. Run bun run replay <trace.json> locally.
3. Read the step number from the harness's stack trace.
4. Set a breakpoint one step earlier and inspect the context window and recent tool returns.
5. If needed, hand-edit the trace JSON to inject alternate tool returns and test fixes.

Step 5 is the killer feature. I once hit a failure pattern where the agent fell over the moment GitHub returned a 403 rate-limit response. Instead of waiting to reproduce in production, I edited the trace to inject the 403 at the same step and verified in 30 seconds that the patched version backed off correctly. You don't need to provoke failures in production anymore.

Bisect — binary-searching the corruption source

Replay shines most on long failures. In my experience, a failure that visibly surfaces at step 23 often originated at step 7, where the agent misinterpreted a tool return. Replay turns this into binary search.

// bisect.ts (simplified)
export async function bisect(trace: Trace, predicate: (subTrace: Trace) => Promise<boolean>) {
  let lo = 0;
  let hi = trace.events.length;
  while (hi - lo > 1) {
    const mid = Math.floor((lo + hi) / 2);
    const sub: Trace = { ...trace, events: trace.events.slice(0, mid) };
    if (await predicate(sub)) hi = mid;
    else lo = mid;
  }
  return lo;
}

predicate is a function that returns true once the agent's internal invariants are broken. I usually check "does the most recent tool_use argument still match its JSON Schema?" The first step where that snaps marks the entry of the corrupted region.

Real numbers from one month across four sites

After running this across all of Dolice Labs for a month, the numbers were:

The single biggest unlock was recording the clock boundary. Without it, Replay determinism sat at 68% because retry backoff jitter kept landing on different branches. Recording one Date.now() lifted determinism by 26 points — even I was surprised.

The 18 ms latency overhead is effectively free in production because writes are queued through Cloudflare Queues to R2. Synchronous writes would land at 80–120 ms, but for an agent run that already takes tens of seconds, async is fine.

Pitfalls I keep stepping on

A few traps worth flagging up front.

Pitfall 1: Nondeterministic library internals. Math.random(), crypto.randomUUID(), HTTP-client retry jitter. If they cross no recorded boundary, Replay drifts. Wrap them.

Pitfall 2: Huge tool returns. Tools like list_files can return multi-MB payloads. When a tool return exceeds 64 KB, I now offload the body to a separate R2 object and keep only a reference inside the trace.

Pitfall 3: Model version drift. Anthropic and Google update models often. If the recording-time model ID differs from the replay-time model ID, what should be a "replay" can branch to a fresh response. Always include model ID and version tag in the trace and assert equality at replay start.

Pitfall 4: Timezone dependence. I lost half a day to this. The agent decided "today's article" via local time, so replaying in UTC picked a different date. Either record the timezone with now() or hold the agent to a UTC-only convention.

My situation-specific recommendations

For readers who've made it this far, here is what I'd suggest depending on where you sit.

If you operate one Antigravity agent solo, start with just the model and tool boundaries. Add clock and randomness only after the first month, once you've actually hit a "can't reproduce" failure.

If you run multiple agents across multiple sites, put the full design (three boundaries + R2/KV + Bisect) in from day one. Failure frequency grows combinatorially with agent count, and a stack of 47-minute investigations can swallow a workday whole.

If you operate agents on others' accounts SaaS-style, prioritize PII masking and TTL above all else. A trace is essentially a recording of the user's work, so terms of service and privacy policy alignment is mandatory before rollout.

I hope Antigravity's official SDK eventually exposes Record/Replay hooks directly. Until then, the thin middleware above is more than enough — and it changed the texture of debugging for me from "stare and pray" to "load, step, fix."

Thank you for reading. I hope this saves someone else a few of the 47-minute sessions I went through.