I Changed One Line in AGENTS.md and Broke the Agent — Regression-Testing the Instructions Themselves

Every time you edit the instructions you hand an agent — AGENTS.md, skill text — some other task quietly breaks. Here is a regression harness whose subject is the instructions, not the code: fixed tasks, deterministic assertions, and a way to wire it into unattended runs.

Antigravity³²⁸ agent design⁷ regression testing³ prompt management unattended ops²

✦ Premium Article

All I did that day was reword one line in AGENTS.md, from "reuse existing functions where possible" to "prefer reusing existing functions." Same intent, surely. Yet the next morning's unattended run had one task writing a fresh utility from scratch every single time. I hadn't touched a line of code. What broke was the instruction.

When you hand Antigravity agents recurring work as an indie developer, this is the failure I fear most. No red test, no exception thrown. Something that used to work just quietly stops working — and because the cause is an instruction you edited in good faith, it takes days to notice.

We write regression tests for code. Why not for instructions? For the past few weeks I have been trying to answer that with an implementation: a regression harness whose subject under test is not the code, but the text you feed the agent.

Treat instructions as implicit code

An agent's behavior is the composition of the model weights and the instructions you provide. You cannot change the weights, but you rewrite the instructions daily. That makes AGENTS.md and your skill files a kind of source code that gets interpreted at runtime. And source code deserves a way to tell whether a change regressed anything.

Ordinary unit tests will not do. Unlike a function's return value, the same input does not yield the same output — the same instruction and task still produce slightly different results each run. So we swap the subject from "the output itself" to "the properties the output must satisfy." Pin the properties, and a little wobble no longer prevents a verdict.

This is continuous with externalizing completion checks. Not trusting an agent's self-report, and verifying from the outside, is the design I covered in the article on graduating agents from self-reported completion. This piece sits one step earlier: a layer that re-verifies many tasks at once, the moment you change an instruction.

Decide fixture granularity first

A regression suite is a set of fixtures. One fixture pairs "a task's instruction" with "the assertions that task must satisfy." Get greedy with granularity and it collapses. My first attempt turned real production tasks into fixtures directly, and it failed: each run took minutes, hit external APIs, and produced environment-dependent results. Far too heavy for a suite you run constantly. Bringing production tasks in wholesale was the first pitfall of this setup; to avoid that weight I now stick to representative, lightweight tasks.

Now I keep fixtures "representative, short, and side-effect closed." My rule of thumb: under 30 seconds per fixture, no external network, and artifacts confined to files inside the sandbox. You are not reproducing the real job — you are reproducing, in minimal form, the property you want the real job to preserve.

Assertion type	What it checks	Noise resistance
File exists	Was the expected artifact produced	High
grep pattern	Did it use a specific function or API	Medium
Exit code	Does the artifact run or build as-is	High
Forbidden	Did it write something you asked it to avoid	High

I do not machine-check semantic correctness — whether the code's intent is right. Lean on a model judge for that and the noise climbs, and the foundation for detecting regressions crumbles. Putting only deterministically measurable properties into assertions is what made the suite trustworthy.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦How to build a regression suite whose subject is the instruction text (AGENTS.md, skills) rather than code, and how to size each fixture

✦Four deterministic assertion types — file existence, grep, exit code, forbidden patterns — plus baseline diffing to catch silent regressions

✦A multi-trial score that keeps stochastic noise from being read as a regression, and how to fold the harness into an unattended schedule

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Write fixtures declaratively

Each fixture is one YAML file per task. Instruction, setup command, and assertions live in one place, and the harness interprets and runs them.

# fixtures/reuse-existing-util.yaml
id: reuse-existing-util
prompt: |
  src/utils.py already has slugify.
  Add logic that builds a slug from an article title.
setup: |
  mkdir -p src
  printf 'def slugify(s):\n    return s.lower().replace(" ", "-")\n' > src/utils.py
assertions:
  - type: file_exists
    path: src/utils.py
  - type: grep
    path: src
    pattern: "from .utils import slugify|utils.slugify|slugify\\("
    must: present
  - type: grep
    path: src
    pattern: "def slugify"
    count_max: 1        # must not redefine slugify
  - type: forbidden
    path: src
    pattern: "re.sub"    # did it ignore the existing helper and rewrite with regex

Declarative fixtures make it painless to add one after editing an instruction. The moment you think "I want to preserve this behavior," you write a single YAML file and it joins the suite. A regression suite survives only if growing it stays cheap.

Implement the harness itself

The harness's job is simple: for each fixture, make a temp directory, run setup, run the agent, and score the assertions. I abstract agent invocation behind run_agent, which is the seam where you actually call the Antigravity CLI headless.

# harness.py
import subprocess, tempfile, shutil, glob, re, os, sys, json
from pathlib import Path
import yaml
 
def run_agent(workdir: str, prompt: str) -> int:
    """Seam for running the Antigravity CLI headless.
    In production, replace the line below with your CLI call."""
    cmd = ["antigravity", "run", "--headless", "--yes", "-p", prompt]
    proc = subprocess.run(cmd, cwd=workdir, capture_output=True, text=True, timeout=120)
    return proc.returncode
 
def _files(base: str, path: str):
    root = os.path.join(base, path)
    if os.path.isfile(root):
        return [root]
    return [p for p in glob.glob(os.path.join(root, "**", "*"), recursive=True)
            if os.path.isfile(p)]
 
def check(base: str, a: dict) -> bool:
    t = a["type"]
    if t == "file_exists":
        return os.path.isfile(os.path.join(base, a["path"]))
    if t in ("grep", "forbidden"):
        pat = re.compile(a["pattern"])
        hits = 0
        for f in _files(base, a["path"]):
            try:
                hits += len(pat.findall(Path(f).read_text(errors="ignore")))
            except OSError:
                pass
        if t == "forbidden":
            return hits == 0
        if a.get("must") == "present" and hits == 0:
            return False
        if "count_max" in a and hits > a["count_max"]:
            return False
        return True
    if t == "exit_zero":
        r = subprocess.run(a["cmd"], cwd=base, shell=True, capture_output=True)
        return r.returncode == 0
    raise ValueError(f"unknown assertion: {t}")
 
def run_fixture(fx: dict, trials: int) -> dict:
    passes = 0
    for _ in range(trials):
        base = tempfile.mkdtemp(prefix="fx-")
        try:
            if fx.get("setup"):
                subprocess.run(fx["setup"], cwd=base, shell=True, check=False)
            run_agent(base, fx["prompt"])
            ok = all(check(base, a) for a in fx["assertions"])
            passes += 1 if ok else 0
        finally:
            shutil.rmtree(base, ignore_errors=True)
    return {"id": fx["id"], "passes": passes, "trials": trials,
            "score": round(passes / trials, 3)}
 
if __name__ == "__main__":
    trials = int(os.environ.get("TRIALS", "3"))
    results = []
    for path in sorted(glob.glob("fixtures/*.yaml")):
        fx = yaml.safe_load(Path(path).read_text())
        results.append(run_fixture(fx, trials))
    json.dump({"results": results}, sys.stdout, ensure_ascii=False, indent=2)

Because check handles only deterministic assertions, it always returns the same verdict for the same sandbox. The only wobble lives inside run_agent. So we run multiple trials and fold the noise into a score.

Catch regressions by diffing against a baseline

A single score cannot tell you "is this worse than yesterday." Save a baseline score and subtract the post-edit score from it. In practice, judging by the size of the drop — not the absolute value — worked best.

# compare.py — diff pre/post scores to detect regressions
import json, sys
 
def load(p):
    return {r["id"]: r["score"] for r in json.load(open(p))["results"]}
 
base = load(sys.argv[1])     # baseline.json (before)
head = load(sys.argv[2])     # current.json (after)
TOL = 0.34                   # with 3 trials, one unlucky miss is tolerable
 
regressed = []
for fid, b in base.items():
    h = head.get(fid, 0.0)
    if b - h > TOL:
        regressed.append((fid, b, h))
 
for fid, b, h in regressed:
    print(f"REGRESSION {fid}: {b:.2f} -> {h:.2f}")
 
sys.exit(1 if regressed else 0)

Pick TOL in conversation with your trial count. With three trials, one occasional miss is within the noise. Set it to zero and the suite stays red while the instructions are fine, and eventually no one looks at it. The philosophy of separating noise from regression is the same as in the article on eval gates you cannot trust because they wobble: a stable verdict is the lifeblood of this kind of gate.

What running it actually taught me

For about six weeks I have run this — 11 fixtures, 3 trials — every time I touch AGENTS.md. A full run takes roughly 4 to 6 minutes. In that window it caught two real regressions. One was the opening "prefer reusing" reword: meant to strengthen the reuse nudge, it actually widened the room to read "writing something new is also fine." The other was tidying the forbidden-word list, where I dropped one existing entry by accident. Both were green on tests, zero exceptions; without the harness I would not have noticed for days.

There were also four false alarms, all early on when I had TOL too strict and misread noise as regression. Once I settled on three trials and a one-miss tolerance, false alarms all but vanished. Going from one trial to three roughly triples the cost of a run, but I personally think the weight is worth paying. A regression suite has to be as good at "not crying wolf over noise" as it is at "never missing a real regression."

Wire it into the unattended schedule

The harness earns its keep not only at edit time but inside recurring runs. I trigger it on any push to the repo that holds AGENTS.md and skill files, and flow straight through to the baseline comparison.

#!/usr/bin/env bash
# ci-regression.sh — stop if an instruction change regresses behavior
set -euo pipefail
TRIALS=3 python harness.py > current.json
if [ ! -f baseline.json ]; then
  cp current.json baseline.json      # first run seeds the baseline
  echo "baseline created"; exit 0
fi
python compare.py baseline.json current.json

For unattended operation, the safe move on a detected regression is to not adopt that instruction change and halt. An OAuth re-auth prompt in the middle would sever the automated flow, so keeping tokens in the keyring is a prerequisite here. I wrote up the auth side of unattended runs in the article on storing OAuth tokens in the OS keyring.

I keep baseline updates manual. Only when I have confirmed a lower score is a "new desirable behavior" rather than a regression do I replace baseline.json. Automate that step and you swallow silent decay into the baseline itself.

Your next step

The first round trip takes three moves:

Pick one behavior to protect and write a single minimal fixture
Deliberately break one line of your instructions and run the harness for three trials
Confirm the score drops and diff it against the baseline Once you have made that "break it and watch it fall" round trip, your instructions are no longer implicit code — they are something whose changes you can measure.

I am still finding my footing with instruction management. Even so, just cutting down the good-intentions one-liner that quietly breaks things has genuinely changed how calm I feel handing work to an agent overnight. Thank you for reading — I would be glad to keep verifying this, a little at a time, together.

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.