Supervising Long-Running Antigravity Agents — Watchdog and Tiered Recovery

Heartbeat protocol

A simple "I was alive N seconds ago" heartbeat turned out to be insufficient. The unresponsive mode freezes right before an LLM call, so the timestamp at that exact moment still gets updated. You need both stage and progress (a monotonically increasing counter) to tell them apart.

// agent/heartbeat.ts
import { writeFileSync, renameSync } from "fs";
import { join } from "path";
 
type Stage =
  | "planning"
  | "tool_call"
  | "llm_response"
  | "post_process";
 
interface Heartbeat {
  ts: number;            // epoch ms
  stage: Stage;
  step_idx: number;      // monotonic
  context_tokens: number;
  session_id: string;
  last_tool: string | null;
}
 
const HB_PATH = join(process.env.STATE_DIR!, "heartbeat.json");
 
export function beat(h: Omit<Heartbeat, "ts">): void {
  const payload: Heartbeat = { ...h, ts: Date.now() };
  const tmp = HB_PATH + ".tmp";
  writeFileSync(tmp, JSON.stringify(payload));
  renameSync(tmp, HB_PATH); // atomic on POSIX
}

Two points to highlight. First, step_idx is a monotonically increasing counter, and that is the field you check for progress. The agent can sit in the same stage for 60 seconds and still be healthy as long as step_idx is climbing. If it sits still, you start suspecting the unresponsive mode. Second, the write must be atomic via tmp then rename. I once had the supervisor read a half-written JSON and trigger a recovery on a phantom failure.

Watchdog timer and verdict logic

The supervisor reads the heartbeat every 5 seconds and classifies the state.

// supervisor/watchdog.ts
import { readFileSync } from "fs";
 
interface Verdict {
  state: "healthy" | "stalled" | "session_dead" | "context_bloat";
  reason: string;
}
 
const STALL_MS = 90_000;          // 90s without progress -> stall
const TOKEN_LIMIT_RATIO = 0.92;   // 92% of token budget
 
let lastIdx = -1;
let lastIdxAt = Date.now();
 
export function evaluate(hbPath: string, tokenLimit: number): Verdict {
  const hb = JSON.parse(readFileSync(hbPath, "utf8"));
  const now = Date.now();
 
  // (1) Stale file -> session is dead
  if (now - hb.ts > 30_000) {
    return { state: "session_dead", reason: `hb age ${now - hb.ts}ms` };
  }
 
  // (2) step_idx not advancing -> stalled
  if (hb.step_idx !== lastIdx) {
    lastIdx = hb.step_idx;
    lastIdxAt = now;
  } else if (now - lastIdxAt > STALL_MS) {
    return { state: "stalled", reason: `step ${hb.step_idx} for ${now - lastIdxAt}ms` };
  }
 
  // (3) Context near limit -> bloat risk
  if (hb.context_tokens / tokenLimit > TOKEN_LIMIT_RATIO) {
    return {
      state: "context_bloat",
      reason: `tokens ${hb.context_tokens}/${tokenLimit}`,
    };
  }
 
  return { state: "healthy", reason: "ok" };
}

The 90-second stall threshold deserves a note. I started at 30 seconds and hit 10+ false positives per day. The AdMob reporting API had a median latency of 22 seconds and a P95 of 71 seconds, so a value with some headroom above P95 was the practical floor. If you accept that revenue-related external APIs are slow and assign per-stage thresholds, accuracy improves further.

Tiered recovery (escalation tiers)

Judgment is useless without action. The key is to avoid jumping straight to heavy interventions. I recommend three tiers.

Tier 1: soft retry (lightweight, idempotent)

When stall is detected, re-inject the current tool call while making the agent aware that the previous attempt produced zero progress.

async function softRetry(agentBus: AgentBus): Promise<boolean> {
  await agentBus.send({
    type: "supervisor_nudge",
    msg: "Previous tool call produced no progress for 90s. Retry the same operation or pick a different approach.",
  });
  return waitForProgress(15_000); // success if step_idx moves within 15s
}

This succeeds 58% of the time across the eight-week dataset. Most are transient LLM-side hiccups, and Tier 1 alone clears more than half of all incidents.

Tier 2: context reset (medium weight, state preserving)

If soft retry does not work, suspect either context_bloat or session_dead and reset. Specifically, replace long historical tool outputs with summaries, then restart the agent. A trick: do the summarization in a small separate LLM call. Asking the main agent to summarize itself led to recursive bloat several times in my testing.

Tier 2 succeeds 29% of the time. Together with Tier 1 you cover 87% via automation.

Tier 3: agent rotation (heaviest, last resort)

If context reset still fails, spawn a new workspace session, hand over the heartbeat and a state snapshot, and resume. Tier 3 is the point where a human should be notified. In my setup Tier 3 triggers a Pushover alert. Tier 3 fires on 13% of incidents, and almost all of those turned out to be external API outages (AdMob or Crashlytics) rather than agent-side faults.

Recovery tier results (8 weeks, 142 incidents)

The remaining 13% is genuinely external (API outages, network partitions) and requires human intervention. Chasing 100% auto-recovery makes the watchdog logic balloon and ironically introduces its own bugs. I recommend drawing the line at 87% and letting alerts handle the rest.

Three things that bit me

A few traps I want to spare you.

(1) Forgetting atomic heartbeat writes

I started with a plain writeFileSync and the supervisor read a partially written JSON, declared session_dead, and triggered a phantom recovery. The tmp-then-rename pattern is mandatory. Worth noting: on WSL2, rename occasionally returns EBUSY, so I keep up to three retries on the supervisor side.

(2) Letting the main agent summarize its own context

If you ask a bloated agent to summarize its own context, it takes on the summarization task while still bloated and the loop deepens. Push the summarization out to a small dedicated LLM (I use Gemini Flash) outside the main agent.

(3) Pushover alert reentrancy

If Tier 3 fires multiple times within a minute, the human gets carpet-bombed with notifications. Always rate-limit alerts at the supervisor (I do "same kind of alert at most once per 10 minutes"). I forgot this on day one and received 47 notifications in two hours.

When this design is worth the cost

I recommend the full watchdog plus tiered recovery stack if any of the following applies:

• A single run takes 30+ minutes and you let it run overnight
• The agent depends on external APIs (AdMob, App Store Connect, Crashlytics) that occasionally lag
• One agent spans multiple apps or sites
• You want fully automated supervision as a solo indie developer

If your task finishes within five minutes, the cost of building the supervisor outweighs the benefit. Just retry blindly. I do not put a watchdog on Antigravity Lab article generation (5 to 8 minutes per run). The full design is only on the 30+ minute AdMob optimization job and the Claude in Chrome plus Crashlytics auto-triage (averaging 90 minutes).

Next step

Drop just heartbeat.ts into an existing agent first. Writing is side-effect free if you do nothing with the data yet. Collect one week of stall traces and look at which mode dominates your workload, then build the supervisor around what you actually see. That avoids a lot of speculative code.

Hope this helps you ship something that survives the night. Thanks for reading.