Designing Antigravity Agent Traces That Tell You Why It Failed — Observability in Practice

Attribute design — what to keep, what to drop

I shape attribute decisions around two questions: "will this be queryable later?" and "is the storage cost tolerable?" The split I run is:

On agent.run:

• task.id / task.kind — unique ID and category
• agent.policy — policy name and version
• agent.tools_available — array of tool names usable on this task
• result.status — success / failure / aborted
• result.failure_class — failure classification (below)
• cost.input_tokens / cost.output_tokens — token usage for the whole task
• cost.usd_estimate — estimated USD spend

On agent.cycle:

• cycle.index
• cycle.action_type — what the cycle chose to do
• cycle.token_count — LLM tokens consumed
• cycle.duration_ms — total cycle time
• cycle.terminated — clean-termination flag

On agent.tool_call:

• tool.name
• tool.input_size — bytes or element count
• tool.output_size
• tool.error_kind — error classification when failing
• tool.retry_count — retries inside this call

What I deliberately leave out is raw input and output text. Putting it in traces blows up storage costs and invites accidental PII leaks. I store raw I/O in object storage (Cloudflare R2 in my setup) keyed by trace ID, and put only the storage key onto the span.

def record_io(span, payload, kind):
    # Stash raw I/O in object storage; only the key goes on the span.
    key = f"{span.context.trace_id:032x}/{span.context.span_id:016x}/{kind}"
    r2_client.put_object(Bucket="agent-io", Key=key, Body=payload)
    span.set_attribute(f"io.{kind}.storage_key", key)
    span.set_attribute(f"io.{kind}.size_bytes", len(payload))

Once you have this in place, you find a trace in Grafana, copy io.input.storage_key, and pull the raw input in one query. Splitting "search → drill in" into two steps beats stuffing long strings onto spans. R2 storage is $0.015/GB/month; six gigabytes of raw I/O per month costs under $0.10.

Failure tags — result.failure_class

Lumping every agent failure into "error" is observability surrendering. I tag them like this:

• tool_unavailable — external API down or auth expired
• tool_misuse — bad input handed to the tool (prompt design issue)
• model_refusal — LLM declined for safety reasons
• model_hallucination — LLM tried to call a tool/function that does not exist
• cycle_budget_exceeded — exceeded max cycles
• token_budget_exceeded — token budget exceeded
• validation_failed — tool output failed schema validation
• unknown_runtime_error — anything else

This taxonomy grows over time. I would not chase completeness on day one; I started with the five most frequent classes from a month of logs and dropped everything else into unknown_runtime_error, then revised monthly.

Once tags exist, dashboards can carve out "show me only tool_unavailable and validation_failed" — and that view directly informs retry policy and the wording of user-facing notifications.

Mapping failure_class to retry policy

The single biggest payoff of a failure taxonomy is that the policy of "how do we treat each failure" collapses into a table. Mine looks like this:

In Python I encode it as a dataclass-driven registry:

from dataclasses import dataclass
from typing import Optional
 
@dataclass(frozen=True)
class RetryPolicy:
    max_retries: int
    backoff_strategy: str  # "exponential" | "immediate" | "fixed"
    notify_user: bool
    ticket_threshold: Optional[int] = None
 
RETRY_POLICIES = {
    "tool_unavailable": RetryPolicy(3, "exponential", False, ticket_threshold=10),
    "tool_misuse":      RetryPolicy(0, "immediate",   True),
    "model_refusal":    RetryPolicy(0, "immediate",   True),
    "model_hallucination": RetryPolicy(1, "immediate", True),
    "cycle_budget_exceeded": RetryPolicy(0, "immediate", True),
    "token_budget_exceeded": RetryPolicy(0, "immediate", True),
    "validation_failed": RetryPolicy(2, "immediate",  True),
    "unknown_runtime_error": RetryPolicy(1, "fixed",  True),
}

The point is to keep design decisions in one place. When you add a new failure_class, you only update this table — behavior across the whole agent shifts in lockstep. Once retry logic scatters across the codebase, the operational story falls apart fast.

Dashboard design — three views that pay for themselves

Once traces are flowing into Grafana or Honeycomb or similar, these three views are what I build first. They cut my incident response time roughly in half — concretely, MTTR went from 47 min to 11 min.

View 1 — failure trace list, last 24 hours. Spans where result.status = failure, grouped by result.failure_class. You see at a glance "what is breaking right now." When an Antigravity update changes behavior, a sudden surge in model_hallucination shows up here before users start writing in.

In LogQL on Loki:

sum by (failure_class) (
  count_over_time(
    {service="antigravity-agent"}
      | json
      | result_status="failure"
      [1h]
  )
)

View 2 — cycle count histogram. Take the maximum cycle.index per successful agent.run and bin it. When tasks that used to finish in three cycles start needing six, you are watching prompt design quietly degrade. After model upgrades I check this every day for the first week.

In Honeycomb:

WHERE name = "agent.run" AND result.status = "success"
VISUALIZE HEATMAP(cycle.index_max)
GROUP BY task.kind
TIME 24h

View 3 — per-tool error rate over time. For each tool.name, plot the percentage where tool.error_kind is non-null over time. A specific external API silently degrading shows up here. Twice now I have caught a payment provider's API drifting downward week over week and reached out to support before users noticed.

sum by (tool_name) (rate(agent_tool_call_total{error_kind!=""}[1h]))
/
sum by (tool_name) (rate(agent_tool_call_total[1h]))

There is no off-the-shelf Grafana template for these, but with the attributes above in place, an hour or two in the query editor is usually enough.

Cost visibility — one chart that shows "what is eating the wallet"

Agents drift into expensive-by-surprise pretty easily, so putting cost attributes on traces speeds up operational decisions. I attach cost.input_tokens, cost.output_tokens, and cost.usd_estimate to every agent.run and watch task-kind cost over time:

sum by (task_kind) (
  rate(agent_run_cost_usd[1h])
)

Within a week of putting this together, I could see that one specific task kind (internal-doc summarization) was burning 62% of total spend. Adding context compression — a small preprocessing pass that summarizes long quotations before they hit the agent — cut that kind's spend by 38%.

Whether you can close a "notice → improve" loop inside a week comes down to your measurement resolution.

Handling PII and audit-grade logging

The data agents touch is often personal. I treat any field that could contain PII (email, payment info, user-authored text) as never written to span attributes. Instead I wrap it:

1. Anything with possible PII is wrapped in a SecureBlob type and written to a dedicated store.
2. Spans only carry pii.storage_key and pii.kind.
3. During debugging, going from trace ID to the PII store requires a separate permission grant.

This is a minimal posture for GDPR / Japanese APPI, but operationally it structurally prevents "oops, that ended up in logs." It is a half-day implementation: one wrapper type plus enforced usage.

@dataclass
class SecureBlob:
    payload: bytes
    kind: str  # "email" | "phone" | "user_text"
 
    def __repr__(self):
        # Stop str() / repr() leaking into logs.
        return f"<SecureBlob kind={self.kind} size={len(self.payload)}>"
 
    def store(self, span) -> str:
        key = f"pii/{secrets.token_hex(16)}"
        secure_store.put(key, self.payload)
        span.set_attribute("pii.storage_key", key)
        span.set_attribute("pii.kind", self.kind)
        return key

What six months of operations actually exposed

Four things became visible after running this for six months that I want to record.

Six in ten failures were tool_unavailable, and most of those concentrated on a single external API. Once visible, this turned into a concrete fix: a health check before that API call, with a fallback path. Before, all I had was the vague feeling that "things fail a lot."

model_hallucination doubled the week after model updates. Once I could see this, I baked a re-evaluation ritual into the model-update process and started updating prompts before the new model went live in production.

One specific task type used pathologically many cycles. Diagnosis: insufficient initial context, so the agent burned cycles gathering info it needed up front. A context refactor brought average cycle count from 4.2 to 2.8 and trimmed token consumption by 35%.

MTTR went from 47 to 11 minutes, which over the long term is the biggest win. Saving ~30 minutes per incident at 20 incidents a month is more than 10 hours a month. Observability is one of those areas where the first hour of investment is the highest-ROI hour you will spend.

Recommended adoption order

For anyone bootstrapping observability for the first time, here is the order I would take if I started over.

1. Add the agent.cycle span only (30 minutes) — drop a single tracer.start_as_current_span("agent.cycle") line into the loop. Cycle boundaries alone change how the system looks operationally.
2. Define five failure classes (1 hour) — pick the five most common failure modes from a recent month of logs and start tagging result.failure_class.
3. Build dashboard view 1 (1 hour) — the failure trace list. Just that view reshapes incident response.
4. Add cost attributes (30 minutes) — put cost.usd_estimate on agent.run and watch spend by task kind.

Three hours of investment gets you well past halfway to "the trace tells me why it failed." The other half is the polish that one month of operation gives you.

Observability is heavy to start, but committing one focused hour to span structure and failure tags, then implementing one cycle at a time, gets you to "the trace tells me why it failed" within a month. For your next agent build, start by adding a single agent.cycle span — the rest of the design will follow. I hope this is useful for anyone working on the same problem.