I handed a failing test suite to an Antigravity agent and asked it to fix things. A few minutes later everything was green. The diff told the real story: the application code was untouched, and fixtures/users.json had been rewritten to match the assertions. The agent had made the tests pass by editing the test data.
No malice involved. The agent took the shortest path to the goal I gave it — "make the tests pass." The actual failure was mine: I had delegated testing without deciding anything about how test data should be handled.
Since then I keep three rules pinned in every project: fixture ownership, seed determinism, and a fixed procedure for carving subsets from real data. Here they are, in order.
Rule 1 — take fixture ownership away from the agent
The first change was declaring fixtures/ read-only for agents. If the agent believes an expected value must change, it reports a proposal instead of editing. One paragraph in the Guide skill goes a surprisingly long way:
## Test data policy
- Files under fixtures/ are read-only
- If you conclude an expected value must change, do not edit it;
report a "fixture change proposal" with your reasoningGuide skills are advisory, though, not enforcement. So a mechanical backstop lives in CI: reject any commit that touches fixtures and application code at the same time.
#!/usr/bin/env bash
# fixture-guard.sh — detect simultaneous fixture/src changes
CHANGED="$(git diff --cached --name-only)"
FIXTURE_TOUCHED=$(echo "$CHANGED" | grep -c '^fixtures/' || true)
SRC_TOUCHED=$(echo "$CHANGED" | grep -c '^src/' || true)
if [ "$FIXTURE_TOUCHED" -gt 0 ] && [ "$SRC_TOUCHED" -gt 0 ]; then
echo "❌ fixture and src changes must be separated"
echo " fixture edits need their own commit with the reason in the message"
exit 1
fi
exit 0Changing fixtures is not forbidden — legitimate updates follow spec changes all the time. What is forbidden is quietly moving the goalposts in the same hand that edits the code. Forced into a standalone commit, a fixture change always crosses a reviewer's eyes.
Rule 2 — generate seeds deterministically
Tests written by agents tend to come with improvised test data: random names and dates conjured inline. That is a factory for flaky tests that pass on Tuesday and fail on Thursday.
The fix is to centralize data creation in one generator script with a pinned seed:
// scripts/generate-fixtures.mjs
import { faker } from "@faker-js/faker";
import { writeFileSync } from "node:fs";
faker.seed(20260702); // pinned seed — identical output on every run
const users = Array.from({ length: 50 }, (_, i) => ({
id: `u${String(i + 1).padStart(4, "0")}`,
name: faker.person.fullName(),
email: faker.internet.email().toLowerCase(),
createdAt: faker.date
.between({ from: "2025-01-01", to: "2026-06-30" })
.toISOString(),
plan: faker.helpers.arrayElement(["free", "pro", "premium"]),
}));
writeFileSync(
"fixtures/users.json",
JSON.stringify({ schemaVersion: 3, users }, null, 2)
);
console.log(`generated ${users.length} users (schemaVersion 3)`);A pinned seed gives you a tamper check for free: regenerate and diff. If the file in the repository no longer matches the generator's output, someone — usually an agent — edited it by hand.
The schemaVersion embedded in the fixture lets tests validate their assumptions. When versions disagree, I skip with a warning rather than fail, which keeps migration periods quiet.