Catching "Running but Doing Nothing" Antigravity Subagents — A 3-Layer Observability Pattern Across Six Production Apps

Layer 1 Heartbeat — Cloud Scheduler, the agent's response time, and BigQuery

Layer 1 is straightforward. When Cloud Scheduler triggers the subagent, it issues a freshly minted start_token (a UUIDv4) and passes it through. Just before the agent terminates, it POSTs the start_token, its elapsed time, and its exit_code to BigQuery via a Streaming Insert.

Here is the TypeScript helper I call from inside Antigravity subagents. I keep it on Cloudflare Workers so it is callable from Antigravity as well as from GitHub Actions.

// src/observability/heartbeat.ts
type HeartbeatPayload = {
  start_token: string;        // UUIDv4 issued by Cloud Scheduler
  agent_name: string;         // e.g. "wallpaper-nightly-refresh"
  started_at: string;         // ISO8601
  finished_at: string;        // ISO8601
  duration_ms: number;
  exit_code: 0 | 1 | 2;       // 0=ok, 1=app-error, 2=infra-error
  triggered_by: "scheduler" | "manual" | "retry";
  attempt: number;            // 1 = first try, 2+ = retry
};
 
export async function emitHeartbeat(p: HeartbeatPayload) {
  // BigQuery Streaming Insert returns inside 500ms, so awaiting is fine
  const res = await fetch(
    "https://bigquery.googleapis.com/bigquery/v2/projects/PROJ/datasets/obs/tables/heartbeat/insertAll",
    {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${await getAccessToken()}`,
      },
      body: JSON.stringify({
        rows: [{ insertId: p.start_token, json: p }],
      }),
    }
  );
  if (!res.ok) {
    // If this fails, give up. Do not crash the agent itself.
    console.error("heartbeat insert failed", res.status);
  }
}

The crucial design decision is that Cloud Scheduler issues the start_token, not the agent. If the agent issues its own token, you cannot detect a day when the agent simply never started. With the scheduler issuing the token, you can poll BigQuery five minutes after dispatch and, if no row with that start_token exists, page out as "failed to start." It is a two-stage check.

What Heartbeat catches, and what it cannot

Inside 60 seconds, Heartbeat alone can catch:

• The agent never received the Cloud Scheduler trigger (no row)
• The agent started but took 3x longer than the rolling average (duration anomaly)
• The agent exited with exit_code != 0

What Heartbeat absolutely cannot catch is the opening scenario: "running but doing nothing." exit_code=0, duration_ms is normal, and the output is empty. That is the gap Layer 2 fills.

Layer 2 Output Trace — hashing the artifacts to confirm "did anything change?"

Layer 2 hashes the resources the agent was supposed to touch, before and after the run. For my wallpaper apps' "nightly asset refresh" subagent, the three targets are:

• GCS bucket — SHA256 of gsutil ls -l output (zero additions is a no-op candidate)
• Firestore releases/latest document — normalize fields and SHA256 (unchanged means we skipped update)
• AdMob ad unit floor price — SHA256 of the API response (identical means no update was applied)

I store hashes in BigQuery as before_hash and after_hash. Here is the Python helper that records them.

# scripts/observability/output_trace.py
import hashlib
import json
from datetime import datetime, timezone
from google.cloud import bigquery
 
bq = bigquery.Client()
 
def normalize(value):
    """dict / list / str into an order-independent canonical string."""
    return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
 
def hash_value(value) -> str:
    return hashlib.sha256(normalize(value).encode("utf-8")).hexdigest()
 
def emit_output_trace(start_token: str, target: str, before, after):
    before_h = hash_value(before)
    after_h = hash_value(after)
    changed = before_h != after_h
    bq.insert_rows_json(
        "PROJ.obs.output_trace",
        [{
            "start_token": start_token,
            "target": target,
            "before_hash": before_h,
            "after_hash": after_h,
            "changed": changed,
            "delta_bytes": len(normalize(after)) - len(normalize(before)),
            "captured_at": datetime.now(timezone.utc).isoformat(),
        }],
    )
    return changed

A gotcha lives here: "changed or not" alone misses stale context. An agent can run against a stale snapshot and, by coincidence, still produce a correct change. So I also keep a separate query that compares "the change category we expected" against "the diff we actually saw."

Telling a healthy no-op day from a sick one

This subagent's expected baseline is "roughly four releases per week on weekdays, around one on weekends." So my rule is: fewer than 3 changed=true events in the last 7 days is abnormal, 3–6 is healthy, and 7+ is overactive (the early symptom of a dependabot-style runaway). The overactive case pages PagerDuty.

-- Count changed=true events in the last 7 days
WITH last7 AS (
  SELECT
    target,
    COUNTIF(changed) AS change_count,
    COUNT(*) AS total_runs
  FROM `PROJ.obs.output_trace`
  WHERE captured_at >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  GROUP BY target
)
SELECT
  target,
  change_count,
  total_runs,
  CASE
    WHEN change_count < 3 THEN 'WARN_underchange'
    WHEN change_count > 7 THEN 'WARN_overchange'
    ELSE 'OK'
  END AS status
FROM last7
ORDER BY status DESC;

Cloud Scheduler runs this query once an hour. Any WARN_* row triggers a Slack #ops thread. Underchange fires at under 3% frequency in practice; overchange at 0.4% (averaged over four months).

Layer 3 Decision Log — tool-call JSON to answer "why didn't anything happen?"

With Layer 1 + Layer 2 in place, you can detect "something is off." The remaining problem is "why." Antigravity's Background Agent has execution logs, but they are natural-language transcripts that get hard to grep at scale. After four months of operation, the transcript search broke down for me.

So I added an explicit hook on the agent side: emit one JSON line per tool call into BigQuery. Antigravity subagents let you author tool definitions in TypeScript or Python, so it is a thin wrapper around each tool's entry point.

// src/observability/decision_log.ts
type DecisionLog = {
  start_token: string;
  tool: string;                          // e.g. "read_file"
  arguments_hash: string;                // SHA256 of JSON.stringify(args)
  result_kind: "ok" | "empty" | "error"; // "empty" is the no-op tell
  result_size_bytes: number;
  ts: string;
};
 
export function withDecisionLog<TArgs, TRet>(
  toolName: string,
  startToken: string,
  fn: (args: TArgs) => Promise<TRet>
) {
  return async (args: TArgs): Promise<TRet> => {
    const t0 = performance.now();
    let kind: DecisionLog["result_kind"] = "ok";
    let result: TRet;
    try {
      result = await fn(args);
      if (result == null || (Array.isArray(result) && result.length === 0)) {
        kind = "empty";
      }
    } catch (e) {
      kind = "error";
      throw e;
    } finally {
      const elapsed = performance.now() - t0;
      void emitDecisionLog({
        start_token: startToken,
        tool: toolName,
        arguments_hash: sha256(JSON.stringify(args)),
        result_kind: kind,
        result_size_bytes: JSON.stringify(result! ?? null).length,
        ts: new Date().toISOString(),
      });
      console.debug(`[tool] ${toolName} ${kind} ${elapsed.toFixed(1)}ms`);
    }
    return result!;
  };
}

Aggregating result_kind: "empty" gives you the "share of tool calls that returned nothing." A healthy agent runs at 6–12%. On days when the agent silently fails, that number jumps above 40%. I call it the "empty ratio" and I have it posted into Slack every morning at 7:30.

A morning empty-ratio query

SELECT
  agent_name,
  DATE(ts, "Asia/Tokyo") AS run_date,
  COUNT(*) AS total_calls,
  COUNTIF(result_kind = "empty") AS empty_calls,
  ROUND(COUNTIF(result_kind = "empty") / COUNT(*) * 100, 1) AS empty_pct
FROM `PROJ.obs.decision_log` AS d
JOIN `PROJ.obs.heartbeat` AS h USING (start_token)
WHERE ts >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
GROUP BY agent_name, run_date
HAVING empty_pct >= 30
ORDER BY empty_pct DESC;

Any agent over 40% gets a same-day context reset (re-clone the repository, drop the cache). That single rule cut stale-context-driven silent failures from 17 per quarter to 3 per quarter.

The Alert Priority Matrix — Slack vs PagerDuty

Once the three layers are emitting signals, the next question is "which ones wake you up at night?" My first attempt routed everything to PagerDuty and I lasted a week. The matrix I run now:

The key principle is that Output Trace no-op alone never pages. No-op days exist, so they go to Slack low, and only get promoted to medium after three consecutive days. That single rule eliminated almost all spurious night pages. In four months across six apps, PagerDuty fired 14 times — 12 of those were real, harmful incidents.

Four Months, Six Apps, Eighteen Agents — the numbers

Here is the before-and-after, measured. The fleet is four wallpaper apps, one healing app, and one manifestation app — six apps. Each runs three Antigravity subagents (nightly refresh / ad KPI monitor / Crashlytics summary), so 18 agents total.

• Silent failures shipped to production: 11/month → 1/month (91% reduction)
• Mean time from "broke" to "fixed": 9.5 hours → 11 minutes (about 50x faster)
• Estimated AdMob revenue lost: $1,400/month → $120/month (85% reduction; estimated from eCPM × impressions)
• My operations time: 45 min/day → 8 min/day (just the morning Slack scan)
• Observability cost (GCP bill): $6.80/month (BigQuery Streaming $4.10, storage $1.30, Cloud Scheduler $0.40, Cloud Run $1.00)

The biggest shift was the visibility of opportunity cost. Silent failures, by definition, do not crash. Before this stack I had been saying "everything looks fine." After Output Trace went in, it became obvious I had been quietly leaving about $1,400 a month on the table. Across the 6-app, 50-million-download portfolio, several titles with lower ARPDAU were losing a few thousand yen each per month — and that adds up to exactly that figure.

Pitfalls — BigQuery cost blowup, PII leakage, and retention

Before you build this, here are the three traps I walked into in the first two weeks.

A 17x BigQuery bill

My initial Layer 3 was streaming result_payload (the raw JSON) into BigQuery. Three days in, Cloud Billing alerted me — I was on a $200/month pace. The fix is to stream only result_size_bytes and result_kind into BigQuery, and store raw payloads in GCS with a 7-day Object Lifecycle rule. That alone gets you to under $7/month.

An AdMob API key in the Decision Log

In an early version I was streaming arguments instead of arguments_hash. Once, an AdMob API request header showed up in the log. I deleted it and rotated the key immediately, but the lesson is permanent: it is very easy to "accidentally" stream PII and secrets into BigQuery. Hash arguments by default, and if you ever do need to keep raw arguments, isolate them in a separate table behind strict IAM.

Empty retention windows quietly accumulating three months of data

I forgot to set partition_expiration_days and BigQuery happily kept everything. My current operational defaults are 30 days for Heartbeat, 90 days for Output Trace, and 14 days for Decision Log. One DDL statement is enough.

ALTER TABLE `PROJ.obs.decision_log`
SET OPTIONS (
  partition_expiration_days = 14,
  description = "Decision logs: short retention (raw payloads stored in GCS)"
);

A Minimal Starting Path — about 60 lines, in the right order

For anyone planning to bolt this onto an existing agent, here is the order I now use whenever I introduce it to a new project. One day, end to end.

1. Layer 1 Heartbeat only (half a day). One BigQuery table, one emitHeartbeat function, one Cloud Run Job that has Cloud Scheduler issue the start_token. That alone covers "failed to start," "timeout," and "abnormal exit."
2. One Slack alert, nothing more (30 minutes). Resist building a complex rule. Post exit_code != 0 events to #ops and call it done.
3. One week of operations, then Layer 2 (half a day). Pick a single target. Trying to instrument every target on day one will sink the project.
4. Two weeks of operations, then Layer 3 (half a day). Start the empty-ratio threshold loose at 50%, collect data, then tighten to 30%.
5. A month in, wire up PagerDuty. Connect only the two or three signals that you decided, from Slack-only operations, you absolutely want to be woken up for.

The non-negotiable here is "do not build all three layers in a sprint." When I tried that the first time, the false-positive rate hit 22% and the whole thing collapsed inside a month. Running Heartbeat by itself for a week teaches you what your agents' typical duration and success rate actually look like — and that intuition makes Layer 2 and Layer 3 thresholds easy to set.

My grandfather's line about temple carpentry was that the sounding test matters more than adding more pillars — that you first need to know what the wood is supposed to sound like. Agent observability feels exactly the same to me now: layer by layer, and only after you can hear what your own production sounds like.

I run all of this on a six-app indie portfolio — wallpaper, healing, manifestation — so this stack should transplant cleanly to any indie or small-team operation running AdMob revenue. The same shape is what I deploy on dolice.design client work whenever a Background Agent is going to production. Even just the first layer is enough to start hearing your own agents — once it is in, the next layer is an easy call.

Thank you for reading. May your agents stop quietly breaking in the small hours.