When Your Antigravity Agent Eval Gate Keeps Flickering — Build Notes on Pass/Fail That Survives Non-Determinism

Pin seeds and the environment to shrink the variance

In parallel with measuring variance, kill the variance you can eliminate at the source. Not all non-determinism carries equal value.

For an eval run you generally want to fix: lower temperature and top_p for evaluation; pin tool mock responses (don't mix real-API latency jitter into the eval); inject a seed wherever you depend on time or randomness. And the one most easily overlooked — pin the model version.

// eval-config.ts
// A bundle of settings that narrow non-determinism, only during evaluation.
 
export const EVAL_MODEL_CONFIG = {
  // Pin the model string explicitly.
  // Never use "latest" or a default alias for evaluation (see below).
  model: "antigravity-preview-05-2026",
  temperature: 0,
  topP: 1,
  seed: 42,            // on supporting models this raises determinism substantially
  maxOutputTokens: 4096,
} as const;
 
// Swap tool calls for mocks during evaluation to remove external jitter.
export function buildEvalTools(fixtures: Record<string, unknown>) {
  return {
    async callTool(name: string, args: unknown) {
      const key = `${name}:${JSON.stringify(args)}`;
      if (key in fixtures) return fixtures[key];
      // Make an unregistered call fail immediately, so you can detect
      // that "the evaluation is leaking into the real API."
      throw new Error(`Unregistered tool call: ${key}`);
    },
  };
}

Throwing on an unregistered fixture is deliberate. Without it, an "evaluation" can hit the real API, and latency, cost, and output all start to wobble. Designing it so "you notice the moment it leaks" keeps your eval environment pure.

Regression detection when the default model upgrades on its own

This is the part that bites in 2026. With Antigravity's Managed Agents and the Gemini-family APIs, it's increasingly common to run against a preview string like antigravity-preview-05-2026 or a default alias. Convenient — but it means the model can be swapped underneath you and behavior can change, without you touching a single line of your code.

That's nasty. Since the cause of the regression isn't in your commits, a baseline comparison against the tip of main won't catch all of it. Something that was green yesterday turns red today because of a model-side update, and you'll hunt for the offending commit and never find it.

The countermeasure is to run a canary evaluation that re-takes the baseline on a schedule, independent of code changes. Antigravity has made scheduled launches easier to work with since v2.1.4, so run the agent nightly against the same dataset and record the score over time. When a delta appears, make it possible to tell whether it came from a commit or from the model.

// canary-track.ts
// Re-take the baseline daily and catch model-update regressions over time.
 
import * as fs from "node:fs";
 
interface CanaryEntry {
  date: string;
  modelLabel: string;     // identifier of the model that actually responded, if available
  meanScore: number;
  worstCases: string[];   // top cases whose score dropped sharply
}
 
export function recordCanary(entry: CanaryEntry, path = "canary-history.json") {
  const history: CanaryEntry[] = fs.existsSync(path)
    ? JSON.parse(fs.readFileSync(path, "utf-8"))
    : [];
  history.push(entry);
 
  // Compare the last two points; warn if it dropped without a code change.
  if (history.length >= 2) {
    const [prev, cur] = history.slice(-2);
    const drop = prev.meanScore - cur.meanScore;
    if (drop > 0.05) {
      const modelChanged = prev.modelLabel !== cur.modelLabel;
      console.warn(
        `Score drop ${drop.toFixed(2)} (${prev.date} -> ${cur.date})` +
          (modelChanged
            ? ` / model changed ${prev.modelLabel} -> ${cur.modelLabel}. Likely a model-side regression`
            : ` / model identifier unchanged. Suspect data quality or an external dependency`),
      );
    }
  }
 
  fs.writeFileSync(path, JSON.stringify(history.slice(-90), null, 2));
}

If you can log the identifier of the model that responded, always do. When a change in modelLabel coincides with a score drop, you can quickly conclude it's a model-side regression. If the identifier stays the same and only the score falls, suspect your dataset or an external dependency instead. Automating the root-cause split is your best defense against decay that creeps in quietly overnight.

Balancing trial count against cost

Multiple trials work, but they aren't free. Running one case five times multiplies both API cost and wall-clock time by five. Run all 50 cases five times on every commit and CI gets so heavy nobody waits for it.

The split I settled on is this: at the pull-request stage, run stable cases once and only the cases in the flaky bucket five times. Stable cases don't wobble, so multiple trials add little value there. Decide the trial count dynamically from each case's past spread rather than a fixed number, and you concentrate cost where it's needed.

In the nightly canary, do the opposite: run every case five-plus times with time to spare. Nobody is waiting here, so it's the place to favor precision. Vary the trial count between "evaluation people wait on" and "evaluation nobody waits on." That single move let me get the perceived speed of CI back while keeping the evaluation trustworthy. As an indie developer, CI time is literally my own waiting time, so this allocation pays off especially well.

Where to start

Trying to perfect the whole evaluation at once usually ends with you running out of steam halfway. The realistic order is to first build only "multiple trials + gate on the CI lower bound" over your local 10–20 cases. That alone visibly cuts the time you waste chasing flaky reds.

From there, add baseline comparison and the canary. You don't need to build all the way to production shadow evaluation from the start. It's better to run something small while watching which cases sit in the flaky bucket — that's how the prompts and tool designs worth fixing surface fastest.

Measure the probabilistic as probabilistic. It sounds obvious, yet it's a hard reframe for a mind trained on binary verdicts. For me, only once I could trust the eval gate was I able to focus on actually improving the agent. If you're stuck flickering in the same place, I hope this gives you a thread to pull.