Regression-Testing Antigravity Agent Output in CI

A normalization layer that flattens only the noise

So I insert a normalization layer just before saving and comparing. Its single job is to rewrite environmental and ordering drift into a deterministic form.

import re
import json
 
# Replace environmental drift with stable tokens
NORMALIZERS = [
    (re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z?"), "<TIMESTAMP>"),
    (re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"), "<UUID>"),
    (re.compile(r"/tmp/[^\s\"']+"), "<TMPPATH>"),
    (re.compile(r"run_id=\w+"), "run_id=<RUN_ID>"),
]
 
def normalize_text(text: str) -> str:
    for pattern, token in NORMALIZERS:
        text = pattern.sub(token, text)
    return text.strip()
 
def normalize_json(payload: dict) -> str:
    # Fix key order and sort dicts inside arrays by stable keys
    canonical = json.dumps(payload, ensure_ascii=False, sort_keys=True, indent=2)
    return normalize_text(canonical)

The key is keeping normalization a test-only preprocessor. The agent itself stays untouched. Production output is saved as-is; only the comparison looks at a drift-flattened shadow. Break this separation and you start distorting production output for the sake of testing, which makes regressions easier to miss, not harder.

Golden-file comparison with pytest

Comparing two normalized shadows fits cleanly into pytest. Make it record-or-assert with an update flag: on the first run there is no golden file, so record; afterward, assert.

import os
from pathlib import Path
 
GOLDEN_DIR = Path(__file__).parent / "golden"
UPDATE = os.environ.get("UPDATE_GOLDEN") == "1"
 
def assert_against_golden(name: str, actual_raw: str):
    GOLDEN_DIR.mkdir(exist_ok=True)
    golden_path = GOLDEN_DIR / f"{name}.txt"
    actual = normalize_text(actual_raw)
 
    if UPDATE or not golden_path.exists():
        golden_path.write_text(actual, encoding="utf-8")
        return  # record mode: do not assert
 
    expected = golden_path.read_text(encoding="utf-8")
    assert actual == expected, (
        f"snapshot regression: {name}\n"
        f"--- expected ---\n{expected[:400]}\n"
        f"--- actual ---\n{actual[:400]}"
    )

Commit the golden files to Git. When a diff appears at review time, it is a record that you intentionally updated the expectation. Run UPDATE_GOLDEN=1 pytest to refresh, then review that diff by hand before merging. This round trip guarantees regressions surface as a pull-request diff.

Match structured output by meaning

The third kind of drift that survives normalization, model paraphrasing, cannot be caught by string equality. The shortcut here is Antigravity structured outputs: have the agent return schema-shaped JSON instead of free text.

Instead of comparing free text, freeze only the required fields of the extracted structure as a contract.

def assert_structured(actual: dict, contract: dict):
    # Verify only key presence and type as the contract
    for key, expected_type in contract.items():
        assert key in actual, f"missing required field: {key}"
        assert isinstance(actual[key], expected_type), (
            f"type mismatch: {key} expected {expected_type}"
        )
    # Snapshot the shape, not the prose
    shape = {k: type(v).__name__ for k, v in sorted(actual.items())}
    return shape
 
CONTRACT = {"title": str, "tags": list, "summary": str, "score": (int, float)}

The wording may drift, but if the contract that title is a string, tags is an array, and score is a number breaks, that is a regression. Guard the skeleton of the output, not every character. This was the realistic landing spot for meaning-based matching.

Cutting CI flake: retries and a diff threshold

Some free text still cannot be eliminated, such as a description body. Failing immediately on a single mismatch there turns CI red for nothing.

For this layer only, I add two cushions. One is up to 3 retries to see whether the drift converges. The other is a diff threshold: if the normalized edit distance is under 5% of the whole, I allow it.

from difflib import SequenceMatcher
 
def soft_match(actual_raw: str, expected: str, tolerance: float = 0.05) -> bool:
    actual = normalize_text(actual_raw)
    ratio = SequenceMatcher(None, actual, expected).ratio()
    drift = 1.0 - ratio
    return drift <= tolerance  # tolerate drift within 5%

Numbers strictly, structure by contract, free text by threshold: once I split it into these three tiers, CI false positives dropped sharply. Weekly flake reruns that used to number in the dozens settled into the single digits, in my own operational experience. I log retry counts and thresholds, and treat any test that retries often as a marker for "add more contract here."

Pitfalls I hit in operation

The worst trap was over-normalizing until it absorbed a real regression. After I flattened entire paths to <TMPPATH>, a genuine bug, a wrong output directory, stopped showing up in the diff. Break the rule that normalization targets only content-irrelevant parts, and the test goes silent.

The second was lazy golden updates. UPDATE_GOLDEN=1 is so convenient that if you get into the habit of updating without reading the diff, the snapshot becomes a rubber stamp rather than a record. I switched to keeping update commits separate from content changes, so the diff always lands in review.

A next step

Start by writing down three places in your agent output that change every run despite being content-irrelevant. One of time, ID, or path will always qualify. Register those three in the normalization layer and get a single golden-file test green. From there, the feel for separating drifting red from real red comes quickly.

Thank you for reading. If you run several agents in parallel, I hope this helps keep your CI quietly green.