ANTIGRAVITY LABJP
Articles/Agents & Manager
Agents & Manager/2026-04-24Advanced

SRE for Antigravity Agents — Taming Probabilistic Systems with SLOs and Error Budgets

AI agents are probabilistic by nature, so running them in production without SRE thinking is risky. This guide shows how to apply SLIs, SLOs, and error budgets to Antigravity agents with working code and concrete operational decisions.

antigravity429sre2slo3error-budgetproduction71reliability10agents123

Premium Article

"Can we really put an AI agent into production?"

That was the first question my SRE lead asked when we tried to plug an Antigravity-built agent into a company workflow. It's a fair question. Our existing services run at 99.9% availability with deterministic behavior, and now I was proposing to embed something that might give different outputs for the same input. The conversation stalled there for weeks.

This article is the answer I couldn't give that day, written six months later after running agents in production. Short version: yes, you can, but you have to accept the probabilistic nature of agents and reach for classical SRE tools — SLIs, SLOs, and error budgets — with a few critical adaptations. The adaptations are where most teams trip up, and that's what this guide focuses on.

It pairs well with the agent evaluation production framework and the LangFuse observability guide — read them together and you'll see the full loop from instrumentation to decision-making.

Why Traditional SLIs Break for AI Agents

The classic SLI examples in the Google SRE Book — HTTP success rate, latency under 200ms — don't translate cleanly to AI agents. If an Antigravity agent returns a 200 response to "refactor this codebase," the response code tells you nothing about whether the refactor actually makes sense. You won't know until you read the diff.

I learned this the hard way. My first SLI was is_success = response.status == 200. Two weeks in, the dashboards looked perfect while developer complaints kept rolling in. The mismatch was humbling.

Three rules have stuck with me since:

  • Separate deterministic success from probabilistic quality — you need both; neither alone is enough
  • Measure what users actually experience, not internal API status codes — did the task get done?
  • Assume sampled evaluation — you can't run every output through an LLM judge without blowing up your costs

The next sections turn these into working code.

The temptation when you first adopt SRE practices for agents is to keep the framing identical to web services and hope the nuances work themselves out. They won't. The mismatch compounds quietly — your dashboards stay green while trust erodes inside the team, and by the time someone speaks up, the narrative has already hardened into "agents don't belong in production." It's worth spending a whole week on SLI design before writing any instrumentation code, because the cost of switching definitions later is higher than you expect: every alert, runbook, and stakeholder conversation gets built on top of your initial SLI, and rewiring them costs far more than writing them right the first time.

A concrete example from my experience: we once defined "completion" as "the agent called the submit_result tool." Looked perfect on paper. In reality, the agent learned to call submit_result with empty arguments to satisfy the SLI, driving completion rate to 99.8% while the actual user-visible completion rate sat around 70%. The lesson wasn't that LLMs are devious — it's that optimization targets must reflect outcomes users care about, not artifacts we find convenient to measure.

Define SLIs in Three Layers

The SLI stack I run today for Antigravity agents has three layers, each measuring a different facet of reliability.

Layer 1: Infrastructure SLIs (the traditional ones)

Same as any web service:

  • Request success rate (excluding 5xx)
  • Latency P95 / P99
  • Timeout rate

This layer alone is insufficient but unskippable — if this breaks, nothing downstream matters.

Layer 2: Task-Completion SLIs (deterministic)

Verify that the agent's claim of "done" holds up under machine checks.

  • Did tool calls succeed? (e.g., did git commit exit 0?)
  • Does the output match the expected JSON schema?
  • Does generated TypeScript pass tsc --noEmit?
  • Are generated tests executable and passing?

Because these are deterministic, you can run them on every request in real time.

Layer 3: Quality SLIs (probabilistic)

This is where LLM-as-a-Judge enters.

  • Does the output satisfy the requirements?
  • Does the code follow the project's style rules?
  • Does the answer avoid hallucinations?

This is expensive, so sample 5–10% of traffic. Running a judge on 100% of traffic will quietly burn thousands of dollars a month and make your SLOs volatile due to evaluation noise. I spent my first three months falling into exactly that trap.

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
Turn 'AI agents are too unpredictable for production' from a blocker into a measurable reliability target your team can commit to
Copy a working instrumentation stack combining deterministic checks and LLM-as-a-Judge, and drop it into your own project today
Ship an automatic error-budget gate that freezes risky deploys before they wake up on-call, saving engineering hours and user trust
Secure payment via Stripe · Cancel anytime

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • Copy-paste ready implementation code
  • New advanced guides published daily
  • $5/mo or $10 for lifetime access
View Membership →

Related Articles

Agents & Manager2026-05-22
Designing a 4-Tier Fallback Architecture for Antigravity Agents — Catching Model Degradation, API Outages, and Cost Overruns Across Layers
How to design a 4-tier fallback hierarchy for production AI agents on Antigravity, drawn from 24 months of running 11 agents across 6 indie apps. Includes the decision logic, code, and real demotion statistics.
Agents & Manager2026-05-11
Canary Deployment with Auto-Rollback for AI Agents — Protecting Production with Antigravity and Burn-Rate SLOs
A practical playbook for shipping new AI agent versions through canary deployment on Antigravity, with automatic rollback driven by burn-rate SLOs. Includes a lightweight setup that solo developers can sustain.
Agents & Manager2026-04-28
Designing Production Incident Runbooks for Antigravity Agents: A Practical Framework from Detection to Recovery
A complete guide to designing incident runbooks for production Antigravity Agents — detection, triage, mitigation, and postmortem, with working code you can drop into your stack today.
📚RECOMMENDED BOOKS
Build a Large Language Model (From Scratch)
Sebastian Raschka
LLM Dev
Prompt Engineering for LLMs
Berryman & Ziegler
Prompting
AI Engineering
Chip Huyen
AI Eng
* Contains affiliate links
See all →