SRE for Antigravity Agents — Taming Probabilistic Systems with SLOs and Error Budgets

Instrumenting Layers 1 and 2

This is TypeScript on the server side of an Antigravity agent, emitting OpenTelemetry metrics that Prometheus aggregates.

// agent-sli.ts
// Purpose: instrument agent SLIs
// Key idea: layer 1 (infra) and layer 2 (task completion) in one place
 
import { metrics, trace } from "@opentelemetry/api";
 
const meter = metrics.getMeter("antigravity-agent", "1.0.0");
 
// Layer 1: infrastructure SLIs
const requestCounter = meter.createCounter("agent_requests_total", {
  description: "Total agent requests",
});
const requestDuration = meter.createHistogram("agent_request_duration_ms", {
  description: "Agent request duration",
  unit: "ms",
});
 
// Layer 2: task-completion SLIs
const taskCompletionCounter = meter.createCounter("agent_task_completions_total", {
  description: "Outcome of the task-completion check",
});
 
export interface SliContext {
  agentId: string;
  taskType: string;
}
 
export interface TaskVerification {
  completed: boolean;
  reason?: string;
}
 
export async function instrumentAgentRun<T>(
  ctx: SliContext,
  run: () => Promise<T>,
  verify: (result: T) => Promise<TaskVerification>,
): Promise<T> {
  const span = trace.getTracer("agent").startSpan(`agent.${ctx.taskType}`);
  const start = Date.now();
 
  try {
    const result = await run();
 
    // Layer 1: succeeded as infrastructure
    requestCounter.add(1, { outcome: "ok", task_type: ctx.taskType });
    requestDuration.record(Date.now() - start, { task_type: ctx.taskType });
 
    // Layer 2: does the result hold up as a completed task?
    const verification = await verify(result);
    taskCompletionCounter.add(1, {
      outcome: verification.completed ? "completed" : "incomplete",
      reason: verification.reason ?? "none",
      task_type: ctx.taskType,
    });
    span.setAttributes({
      "agent.task_completed": verification.completed,
      "agent.verification_reason": verification.reason ?? "",
    });
 
    return result;
  } catch (err) {
    requestCounter.add(1, { outcome: "error", task_type: ctx.taskType });
    taskCompletionCounter.add(1, {
      outcome: "error",
      reason: err instanceof Error ? err.name : "unknown",
      task_type: ctx.taskType,
    });
    span.recordException(err as Error);
    throw err;
  } finally {
    span.end();
  }
}
 
// Example verification for a code-generation task. The type checker is injected.
export async function verifyCodeGenerationResult(
  result: { files: Array<{ path: string; content: string }> },
  deps: { typecheck: (files: Array<{ path: string; content: string }>) => Promise<boolean> },
): Promise<TaskVerification> {
  if (!Array.isArray(result.files) || result.files.length === 0) {
    return { completed: false, reason: "empty_output" };
  }
  if (result.files.some((f) => f.content.trim() === "")) {
    return { completed: false, reason: "empty_file" };
  }
  const tsFiles = result.files.filter((f) => f.path.endsWith(".ts"));
  if (tsFiles.length > 0 && !(await deps.typecheck(tsFiles))) {
    return { completed: false, reason: "typecheck_failed" };
  }
  return { completed: true };
}

The typecheck function can simply write the files to a temp directory and run tsc --noEmit. I once nearly settled for "does the content contain the word export," which lets near-empty output through. Start with a completion check that's slightly too strict; it's easier to loosen later.

Here is the completion-rate SLI in PromQL.

# Task-completion SLI (last 28 days)
sum(rate(agent_task_completions_total{outcome="completed"}[28d]))
  /
sum(rate(agent_task_completions_total[28d]))

Write one verify function per task type. "Done" means something different for code generation, doc generation, and research reports, so keep them pluggable.

Instrumenting Layer 3 (LLM-as-a-Judge)

Quality SLIs run as a separate job. For cost, latency, and reliability alike, keeping them off the request path is safer.

// quality-judge.ts
// Purpose: score sampled outputs with an LLM judge
// Key idea: if the judge fails, production must not
 
import { GoogleGenAI } from "@google/genai";
 
const judge = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
 
const JUDGE_PROMPT = `
You evaluate the output quality of an AI agent.
Score the item below on three axes from 1 to 5 and answer in JSON.
 
- correctness: does it satisfy the task requirements?
- safety: any security problems (hard-coded secrets, SQL injection, etc.)?
- style: does it follow the project's coding conventions?
 
Task: {task}
User input: {input}
Agent output: {output}
 
Return: {"correctness": 1-5, "safety": 1-5, "style": 1-5, "reason": "..."}
`;
 
interface QualityScore {
  correctness: number;
  safety: number;
  style: number;
  reason: string;
}
 
export async function evaluateQuality(
  task: string,
  input: string,
  output: string,
): Promise<QualityScore | null> {
  // Use function replacers so "$&" and similar in the input cannot corrupt the prompt
  const prompt = JUDGE_PROMPT
    .replace("{task}", () => task)
    .replace("{input}", () => input.slice(0, 4000))
    .replace("{output}", () => output.slice(0, 8000));
 
  try {
    const response = await judge.models.generateContent({
      model: "gemini-2.5-flash",
      contents: prompt,
      config: {
        responseMimeType: "application/json",
        temperature: 0.1, // keep scoring stable
      },
    });
 
    const text = response.text;
    if (!text) {
      console.warn("judge returned empty response");
      return null;
    }
 
    const parsed = JSON.parse(text) as QualityScore;
    const scores = [parsed.correctness, parsed.safety, parsed.style];
    if (!scores.every((v) => typeof v === "number" && v >= 1 && v <= 5)) {
      console.warn("judge returned invalid scores", parsed);
      return null;
    }
    return parsed;
  } catch (err) {
    // A judge failure is not an agent failure. Log it and move on.
    console.error("judge failed:", err);
    return null;
  }
}

Plain random sampling isn't enough either, because rare task types never accumulate evaluations. Guarantee a floor per type.

// sampler.ts
// Purpose: stratified sampling with a minimum per task type
 
const evaluatedCount = new Map<string, number>();
const MIN_PER_TYPE_PER_DAY = 5;
const BASE_RATE = 0.05;
 
export function shouldSample(taskType: string): boolean {
  const done = evaluatedCount.get(taskType) ?? 0;
  const pick = done < MIN_PER_TYPE_PER_DAY || Math.random() < BASE_RATE;
  if (pick) evaluatedCount.set(taskType, done + 1);
  return pick;
}
 
// Call at the start of the daily job
export function resetDailyCounters(): void {
  evaluatedCount.clear();
}

People ask whether the judge's scores can be trusted. My answer is "not fully." The judge is a coarse sieve for large volumes; final quality calls get corrected by a human. Even one hour a week of reading what the judge scored low, and measuring how often your scores agree with it, goes a long way. Track that agreement rate as a meta-SLI.

Choosing SLOs Without Emotion

Once the SLIs exist, set the SLOs. "Just say 99.9%" is the trap. Work backwards from the failure the business can tolerate.

Here is an example for a code-generation agent. These numbers illustrate the reasoning; they are not values to adopt as they are.

I don't put 99.9% on layer 2 because a target you can't reach only wears out whoever runs it. Start slightly looser than what you measure over one or two weeks of real traffic, then draw the line at "a number we can keep."

An Error-Budget Gate

An SLO implies its flip side, the error budget. A 92% monthly SLO leaves 8% of failures you can afford.

The budget's value is that decisions change with what's left: ship freely while it's plentiful, hold risky changes when it's nearly gone. This code automates that switch.

// error-budget-gate.ts
// Purpose: decide whether to deploy or experiment, based on remaining budget
// Key idea: leave the call to the numbers, not to a debate
 
export interface ErrorBudgetStatus {
  consumedPercentage: number; // 0 or more; above 100 means overspent
  remainingPercentage: number; // never below 0
  recommendation: "proceed" | "proceed_with_caution" | "freeze";
  reason: string;
}
 
const QUERY = `
  sum(rate(agent_task_completions_total{outcome="completed"}[28d]))
    /
  sum(rate(agent_task_completions_total[28d]))
`;
 
export async function checkErrorBudget(
  promUrl: string,
  sloTarget: number, // e.g. 0.92
): Promise<ErrorBudgetStatus> {
  const res = await fetch(
    `${promUrl}/api/v1/query?query=${encodeURIComponent(QUERY)}`,
    { signal: AbortSignal.timeout(5000) },
  );
 
  if (!res.ok) {
    // How to treat a Prometheus outage is a policy choice; some teams should freeze
    return {
      consumedPercentage: 0,
      remainingPercentage: 100,
      recommendation: "proceed_with_caution",
      reason: `prometheus query failed: ${res.status}`,
    };
  }
 
  const data = (await res.json()) as {
    data: { result: Array<{ value: [number, string] }> };
  };
 
  // An empty result (just instrumented, no traffic) or NaN must not read as "0% used"
  const raw = data.data.result[0]?.value[1];
  const completionRate = raw === undefined ? NaN : parseFloat(raw);
  if (Number.isNaN(completionRate)) {
    return {
      consumedPercentage: 0,
      remainingPercentage: 100,
      recommendation: "proceed_with_caution",
      reason: "no data in window — cannot evaluate budget",
    };
  }
 
  const allowedFailure = 1 - sloTarget; // e.g. 0.08
  const actualFailure = 1 - completionRate;
  const consumed = (actualFailure / allowedFailure) * 100;
  const remaining = Math.max(0, 100 - consumed);
 
  if (consumed < 50) {
    return { consumedPercentage: consumed, remainingPercentage: remaining,
      recommendation: "proceed", reason: "budget has plenty of headroom" };
  }
  if (consumed < 90) {
    return { consumedPercentage: consumed, remainingPercentage: remaining,
      recommendation: "proceed_with_caution",
      reason: "budget is being consumed — risky changes require review" };
  }
  return { consumedPercentage: consumed, remainingPercentage: remaining,
    recommendation: "freeze",
    reason: "error budget nearly exhausted — freeze non-critical deploys" };
}

Keep the caller in its own file. This assumes an ESM runner such as tsx.

// check-budget-cli.ts
import { checkErrorBudget } from "./error-budget-gate";
 
const status = await checkErrorBudget(
  process.env.PROM_URL ?? "http://localhost:9090",
  0.92,
);
console.log(JSON.stringify(status, null, 2));
 
if (status.recommendation === "freeze") {
  console.error("Error budget exhausted. Deploy blocked.");
  process.exit(1);
}

Call it from GitHub Actions and fail the job on freeze, and the mood of operations changes. Nobody has to wake up at night and plead for a freeze; the pipeline stops first.

"Prometheus is down, so proceed with caution" is a real judgment call. I judged that agents halting on a short outage costs more than a bad deploy slipping through that window. Your answer may differ. Write the reasoning down as an ADR, so the person you'll be in six months can question the line.

Burn-Rate Alerts That Can Actually Fire

Alert design is where agent SRE differs most. An alert that fires the instant a failure happens wears the on-call out. The Google SRE Workbook's multi-window, multi-burn-rate approach applies here.

Burn rate is how many times faster than the sustainable pace you're spending the budget. A burn rate of 1 uses the budget up exactly at the end of the period.

Here is the trap I noticed while checking the math. The Workbook's standard example — burn rate 14.4 over one hour, 6 over six hours — assumes an SLO around 99.9%. Carry it over to a 92% agent SLO and this happens:

With an allowed failure rate of 8%, the failure ratio can never exceed 100%. So the highest burn rate that can ever fire is 1 ÷ 0.08 = 12.5, and a 14.4x alert will never page you however long you wait. Your dashboard stays green and the alert stays silent — the hardest kind of failure to notice.

You can verify this in a few lines of Node. Rebuild this table every time you change an SLO.

// burn-table.mjs — node burn-table.mjs 0.92
const slo = Number(process.argv[2] ?? 0.92);
const budget = 1 - slo;
const monthHours = 720;
 
console.log(`max burn rate = ${(1 / budget).toFixed(2)}`);
for (const [name, burn, windowH] of [["page", 6, 1], ["ticket", 3, 6], ["slow", 1, 72]]) {
  const failure = burn * budget;
  const used = ((burn * windowH) / monthHours) * 100;
  console.log(`${name}: burn=${burn} window=${windowH}h failure>${(failure * 100).toFixed(1)}% budget=${used.toFixed(2)}%`
    + (failure >= 1 ? "  <- unreachable" : ""));
}

Run with SLO 0.92, it prints the same numbers as the table: a maximum of 12.50, page at 48.0%, ticket at 24.0%, slow at 8.0%.

Now the alert itself. Fire only when both a short and a long window exceed the threshold. The long window confirms the spend is real; the short one confirms it is still happening. First, record the per-window failure ratios.

# agent-slo-rules.yml (excerpt; define 1h, 6h, 30m, and 3d the same way)
groups:
  - name: agent-slo-recording
    rules:
      - record: agent:task_failure_ratio:rate5m
        expr: |
          1 - (
            sum(rate(agent_task_completions_total{outcome="completed"}[5m]))
            /
            sum(rate(agent_task_completions_total[5m]))
          )
      - record: agent:task_failure_ratio:rate1h
        expr: |
          1 - (
            sum(rate(agent_task_completions_total{outcome="completed"}[1h]))
            /
            sum(rate(agent_task_completions_total[1h]))
          )
 
  - name: agent-slo-alerts
    rules:
      - alert: AgentErrorBudgetBurnPage
        expr: |
          agent:task_failure_ratio:rate1h > (6 * 0.08)
          and
          agent:task_failure_ratio:rate5m > (6 * 0.08)
          and
          sum(rate(agent_task_completions_total[1h])) > 0.01
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "At this pace the budget is gone in about 5 days (SLO 92%)"

The trailing > 0.01 is a traffic floor against false alarms in quiet hours. An agent handling about 36 requests an hour will see its 5-minute failure ratio swing wildly on a single failure. The lower your volume, the more this floor and your window lengths matter.

One more note. The budget math above assumes 30 days. If your SLO window is 28 days, convert the consumed-budget figures with 28 days; when the window and the burn rate drift apart, the alert stops matching the promise you made.

For the first month, I'd set the thresholds looser (page at 4 instead of 6, say) and tighten them as you watch the firing rate. If a week passes without a page, don't rush to lower the threshold. An alert that stays quiet is working.

Incident Patterns Worth a Runbook

Having SLOs doesn't tell you what to do when something breaks. Three agent-specific patterns make a good first draft of a runbook.

Pattern 1: a sudden spike in failures (10% to 50% within an hour)

Suspect the upstream LLM provider first: check its status page and your own latency metrics for anomalies, both sudden rises and suspicious drops. Then suspect the last prompt or tool-definition change. Prompts are fragile; one line can change behavior. Keep them in Git like code and write the rollback steps into the runbook.

Pattern 2: quality that decays slowly (the budget leaks over days)

The hardest one. Infra and layer 2 look fine while judge scores slide from 4.2 to 4.0 to 3.8. Usually it is data drift — inputs moving away from what the prompt was designed for. Record token-length distributions and library versions over time so you can pin down when it started.

Pattern 3: failures isolated to one task type

Overall SLIs look healthy, yet one segment suffers. If you label by task type, you can carve a segment-specific SLO out of the same metrics. Don't be stingy with labels; they pay off later.

Evaluation Datasets as First-Class Artifacts

The credibility of a quality SLI rests on its evaluation dataset. Keep input examples paired with a checklist of what a good output satisfies — not an exact expected answer. For code generation: "imports React," "handles the empty array," "follows the naming convention."

Keep the dataset in Git and tag every significant change; let judge prompts reference checklist items. You get three things: you can re-run last month's evaluation against this month's agent and compare, you can diff your way to a cause when quality drops, and non-engineers can read the checklist and ask useful questions.

Refresh it every month or so with samples of real inputs, after consent and a privacy review. Replace only the oldest third; the stable examples are what let you notice a regression. Sampled inputs may contain personal data or secrets, so route them through the same classification and redaction as your logs and keep them out of public repositories.

For keeping flaky evaluations from becoming deploy gates, see my notes on eval-gate flakiness.

Agreeing on SLOs: With a Team, and Alone

On a team, present the SLO as a commitment line for service quality. Agreeing in advance that "below 92% completion we pause new features and work on the agent" keeps quality-versus-revenue from turning into a tug of war. For feature developers, show the budget as a budget: "we've used 60% this month, so let's push risky prompt changes to next month."

If you run apps or sites on your own, the person you have to agree with is yourself. Even so, writing the SLO into a README, with "below this number I stop shipping features," helps. A tired me at night tends to make the lenient call.

Build two dashboards: a one-page overview, and a time-series view for digging in. One screen trying to do both is awkward for everyone.

Label Design

The labels on layer 2 pay off more the longer you run.

Don't label by user_id or anything else high-cardinality; per-user investigation belongs to structured logs and traces. Metrics aggregate, logs and traces individuate. Before adding a label, imagine two or three concrete queries you'd write with it. If you can't, you don't need it yet.

When to Skip SLOs

Not every agent needs this machinery. Under about 100 requests a day, percentage-based SLOs are statistical noise; structured logs and a human eye are enough. A prototype whose purpose is still shifting is the same, since an SLO would lock in assumptions you're about to change.

If you already have a product-level SLO such as "90% of sessions end in a successful outcome" and the agent's behavior is inside it, a separate agent SLO may be unnecessary. But if the agent touches money, handles security-sensitive operations, or runs unsupervised, the cost of an SLO is far smaller than one bad week without it. Instrument from day one.

Pitfalls

These are ones I've hit, or nearly hit.

Pitfall 1: keeping the SLO from the people it affects. If only operations knows the SLO, a request like "let's just ship this speed optimization" quietly eats the budget. Publish the remaining budget somewhere anyone can see it.

Pitfall 2: not versioning the evaluation dataset. When judge scores drop, you can't tell whether the agent regressed or the dataset changed. Tag metrics with the Git commit ID.

Pitfall 3: alerting on the SLI directly. A page at 5% failures wears people out. Alert on burn rate, and make sure the threshold is reachable from your SLO.

Pitfall 4: deferring human review. The judge has tastes of its own. Without an hour of human eyes a week, a broken judge goes unnoticed. Count that hour as an SRE cost.

Pitfall 5: one SLO for many agents. One agent's regression hides inside another's surplus. Give each its own SLO and dashboard.

The First Step

If you'd like to start, please don't build everything at once. SRE comes in at the pace of your maturity.

Instrument layer 2 only — task completion rate — for one week. Layer 1 is usually covered by existing monitoring, and layer 3 costs both money and effort. When the number arrives, notice whether it is higher or lower than you expected; that reaction is where the SLO conversation starts. Then, once you've chosen an SLO, put it into burn-table.mjs and check, just once, that the alert you configured can actually fire.

That is the order in which I rebuilt mine. I still treat the quiet alerts as the ones most worth doubting.