When Your Antigravity-Written Tests Only Look Green: Measuring Effectiveness with Mutation Testing

A minimal setup

For JavaScript / TypeScript, Stryker is easy to reach for. As a subject, here is a small module that decides whether an ad may be shown.

// adReady.js
export function canShowInterstitial(state) {
  // After one display since launch, wait at least 60 seconds
  if (state.lastShownAt !== null) {
    const elapsed = state.now - state.lastShownAt;
    if (elapsed < 60_000) return false;
  }
  // Never show to paying users
  if (state.isPremium) return false;
  return true;
}

Here is what the agent wrote first.

// adReady.weak.test.js
import { describe, it, expect } from "vitest";
import { canShowInterstitial } from "./adReady.js";
 
describe("canShowInterstitial", () => {
  it("returns a value", () => {
    const result = canShowInterstitial({
      now: 100_000, lastShownAt: null, isPremium: false,
    });
    expect(result).toBeDefined();
  });
 
  it("does not throw for paying users", () => {
    expect(() =>
      canShowInterstitial({ now: 0, lastShownAt: null, isPremium: true })
    ).not.toThrow();
  });
});

These tests are green, and coverage doesn't look bad at a glance. Let's add Stryker and measure effectiveness.

npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner vitest

// stryker.conf.js
export default {
  testRunner: "vitest",
  coverageAnalysis: "perTest",
  mutate: ["adReady.js"],
  reporters: ["clear-text", "html"],
  thresholds: { high: 90, low: 70, break: 70 },
};

npx stryker run

The result is sobering.

Mutation score: 27.27%
Survived mutants:
  - adReady.js:4  `elapsed < 60_000` → `elapsed <= 60_000`
  - adReady.js:4  `60_000` → `60_001`
  - adReady.js:8  `if (state.isPremium) return false` → `return true`
  - adReady.js:9  `return true` → `return false`

The number makes it plain: the green tests verified almost none of the logic. The run can't clear break: 70, so this commit stops in CI.

Reading the survivors

Every surviving mutant is a blueprint for a test you should write.

The fact that if (state.isPremium) return false survived being changed to return true means nothing asserts the single most important rule — "don't show ads to paying users." Structurally, it was the exact mistake I made in the opening.

Rewrite the weak tests so they kill the mutants.

// adReady.strong.test.js
import { describe, it, expect } from "vitest";
import { canShowInterstitial } from "./adReady.js";
 
const base = { now: 1_000_000, lastShownAt: null, isPremium: false };
 
describe("canShowInterstitial", () => {
  it("shows when conditions are met", () => {
    expect(canShowInterstitial(base)).toBe(true);
  });
 
  it("never shows to paying users", () => {
    expect(canShowInterstitial({ ...base, isPremium: true })).toBe(false);
  });
 
  it("does not show within 60s of the last display", () => {
    expect(
      canShowInterstitial({ ...base, lastShownAt: base.now - 59_999 })
    ).toBe(false);
  });
 
  it("shows at exactly 60s elapsed (boundary)", () => {
    expect(
      canShowInterstitial({ ...base, lastShownAt: base.now - 60_000 })
    ).toBe(true);
  });
});

That last "exactly 60s" case kills the mutant that swaps < for <=. One value placed right on the boundary leaves the off-by-one edit nowhere to survive. Re-run, and the score reaches 100% and clears the gate.

Wiring it into the agent loop

This is where Antigravity earns its place. Reading survivors by hand doesn't scale once there are many. The list of surviving mutants is exactly the kind of input an agent handles well.

Stryker can emit a machine-readable JSON report.

npx stryker run --reporters json
# → reports/mutation/mutation.json

Pull out only the entries with status "Survived", reduced to file, line, and the edit, and hand that to the agent.

node -e '
const r = require("./reports/mutation/mutation.json");
const out = [];
for (const [file, f] of Object.entries(r.files)) {
  for (const m of f.mutants) {
    if (m.status === "Survived") {
      out.push(`${file}:${m.location.start.line} ${m.mutatorName} → ${m.replacement}`);
    }
  }
}
console.log(out.join("\n"));
' > survived.txt

Then give the Antigravity chat agent an instruction like the following. The crucial part is forbidding it to touch the implementation.

survived.txt lists the mutants the current tests miss. Add only test cases that kill each mutant. Do not change the implementation in adReady.js. After adding tests, run npx stryker run and repeat until the survivor count is 0.

By freezing the implementation and pointing the agent at concrete targets — the surviving mutants — you keep it from spraying out pass-only tests. With the edits it must kill given as a list, the assertions naturally gravitate toward the core of the behavior.

In my own use, the quality of the generated tests changed visibly depending on whether I handed over this list. Without it, the agent drifts toward weak assertions; with it, it reaches for boundary values and negative paths on its own.

Why chasing 100% is a mistake

One warning up front. Pinning the mutation score to 100% actually breaks your workflow.

The cause is equivalent mutants. A mutant that doesn't change behavior at all — say a <=/< difference on a loop bound that never affects the final result, or an edit inside dead code — is impossible for any test to kill, by definition. These are not blind spots; they are mutants that are correct to leave alive.

A tool can't fully decide this equivalence. So if you force 100%, the agent starts trying to kill the unkillable, writing strange assertions that don't engage the implementation's meaning, or tests so tightly coupled to internal details that they break on every refactor. That is a different kind of decay from weak tests, but just as troublesome.

In my own use, I review surviving mutants one by one and sort each into "blind spot" or "equivalent," excluding the equivalent ones explicitly with Stryker's // Stryker disable comment. I keep the threshold at 80–90% and own the rest by hand. With that line drawn, the targets I hand to the agent narrow down to only what genuinely should be killed.

How far to take it — thresholds and diff mutation

Mutation testing is expensive. Running it across the whole repo every time takes too long to sustain. Two practical landing spots.

First, scope it to the diff. Stryker can evaluate only changed files.

npx stryker run --since main

Running mutation only on the code a pull request touched finishes in tens of seconds. You protect the effectiveness of newly written code and accept that as the boundary.

Second, don't use a single uniform threshold. For modules where a mistake costs real money — billing, auth, ad-display conditions — demand a high score, and relax it for places like log formatting where breakage is minor. Splitting the mutate patterns or keeping per-directory config is the manageable way to do this.

You don't need everything at 100%. The goal is to hold one operational brake: don't trust an agent's green tests unconditionally.

As a next step, pick the single module you least want to break, and add stryker run --since main as a CI check there. The moment the surviving mutants print as a list, what your tests were missing becomes clear — not in words, but in numbers.

Thank you for reading. I hope it serves as a first filter for anyone who, like me, has started handing tests over to AI.