Before a Major Update Silently Breaks Your Overnight Automation — Designing a Staged-Adoption Canary Gate

The Core Idea Is "Freeze, Then Update"

The design philosophy here is plain. Freeze the working setup first, make that frozen point restorable at any moment, and only then try the new version somewhere else. Adopt it into production's unattended runs only once you have confirmed it behaves as it did before.

Put differently: turn an update from an irreversible one-shot into a reversible procedure that has a pass/fail result. As an indie developer I cannot pour staff hours into a verification environment. So having two things ready first — reversibility, and a pass/fail a machine can decide — is what lets me trust the nightly runs.

The Procedure — A Staged-Adoption Canary Gate

The actual flow has four stages. Walk them in order and there is nothing to agonize over at each decision.

1. Freeze the working setup into a manifest, keeping one prior generation as a fallback.
2. Install the new version into a profile isolated from production. Do not touch the production settings.
3. Run a representative task once in the isolated profile and compare it against a golden output.
4. If the similarity score clears the threshold, adopt; if not, roll back automatically to the frozen version.

The heart of these four stages is the third, the golden comparison. We do not ask whether the new version got faster or smarter, only whether it behaves the way it used to. For unattended runs, getting the same result today as yesterday matters more than sharp brilliance.

Implementation 1: Freeze the Working Setup

First, lock the currently working setup into a single file. The point is to record not just the version number but the hash of the extension list and the default model alongside it. Capturing all three variables together lets you later explain what moved as a diff.

#!/usr/bin/env bash
# capture-env.sh — freeze a verified, working setup
set -euo pipefail
OUT="env.lock.json"
ag_version="$(antigravity --version 2>/dev/null | head -1)"
node_version="$(node -v)"
default_model="$(antigravity config get model.default 2>/dev/null || echo unknown)"
ext_hash="$(antigravity ext list --json | sha256sum | cut -c1-16)"
cat > "$OUT" <<JSON
{
  "captured_at": "$(date -u +%FT%TZ)",
  "antigravity": "${ag_version}",
  "node": "${node_version}",
  "default_model": "${default_model}",
  "extensions_sha": "${ext_hash}"
}
JSON
echo "frozen -> $OUT"

Keep this file in your repository and the history of updates lives on as a git diff. Once I started committing env.lock.json, I could trace "when did what change," and the diagnosis mornings got much shorter.

Implementation 2: The Canary Verification Runner

Next, run the new version once in the isolated profile and compare it against the previous output. Demanding an exact match would fail you on every tiny fluctuation, so judge "did the behavior change" with a similarity threshold.

# canary.py — try the new version in an isolated profile, return pass/fail via exit code
import json, subprocess, sys, difflib
 
GOLDEN = "canary/golden_output.txt"
TASK   = "canary/task_prompt.txt"
 
def run_on(profile: str) -> str:
    out = subprocess.run(
        ["antigravity", "run", "--profile", profile, "--prompt-file", TASK],
        capture_output=True, text=True, timeout=900,
    )
    if out.returncode != 0:
        raise RuntimeError(f"run failed: {out.stderr[:200]}")
    return out.stdout.strip()
 
def similarity(a: str, b: str) -> float:
    return difflib.SequenceMatcher(None, a, b).ratio()
 
def main() -> int:
    golden = open(GOLDEN, encoding="utf-8").read().strip()
    actual = run_on("canary-next")
    score = similarity(golden, actual)
    print(json.dumps({"similarity": round(score, 3)}))
    # below 0.92, treat behavior as changed and block adoption
    return 0 if score >= 0.92 else 1
 
if __name__ == "__main__":
    sys.exit(main())

The 0.92 threshold is my own value; tune it to how noisy your tasks are. If the output mixes in dates or random values, mask those parts before comparing to cut false positives. Returning pass/fail through the exit code alone means a downstream shell can branch on it directly.

Implementation 3: Watch Health and Pin Back Automatically

Finally, split adoption and rollback by the canary result. Self-update only on a pass, then re-freeze the new setup. On a fail, simply re-pin to the frozen version and you are back where you started.

#!/usr/bin/env bash
# adopt-or-rollback.sh — adopt only on a canary pass, otherwise return to the frozen version
set -euo pipefail
if python3 canary.py; then
  cp env.lock.json env.lock.prev.json     # keep one prior generation
  antigravity self-update --channel stable
  ./capture-env.sh                         # re-freeze the new setup
  echo "adopted"
else
  pinned="$(jq -r .antigravity env.lock.json)"
  antigravity self-update --pin "$pinned"
  echo "rolled back to ${pinned}"
fi

I run these three scripts once, in the window before the unattended runs begin. Whether or not there is an update, every cycle goes through "freeze, canary, adopt or roll back," so the foundation can be swapped safely even while no one is awake. I recommend placing this about two hours before the nightly tasks start, so that even if verification surfaces a problem, there is slack before production begins.

The Numbers and Pitfalls I Saw in Practice

Here is the result after running this for a while. The numbers are measured in my own environment and will vary with the number and kind of tasks you run.

More than the success rate itself, what mattered to me was that the time to notice trouble went from "next morning" to zero. It cut off the worst path: spending a whole night with a broken setup.

A few pitfalls worth recording. First, if you leave extension auto-update on, freezing the CLI still lets the extensions move behind your back. I turned extension auto-update off and now update them explicitly, together with updates to env.lock.json. Second, if you create a golden output once and forget it, it will keep failing even legitimate spec changes. Put a step on the adopt side that always refreshes the golden from the newly adopted output, so verification does not go stale. Third, an isolated profile can accidentally read production's settings file. Separate the profile's config directory fully via environment variables and avoid handing it production credentials.

The more a task feeds directly into the next day's numbers — like AdMob revenue aggregation or release prep for Google Play — the more these two things, reversibility and a clear pass/fail, pay off.

Where to Start

You do not need all three scripts from day one. The first step I recommend is to run only capture-env.sh starting today and commit env.lock.json to git. That alone lets you pull out "what was it before" as a diff when the next major update arrives. The canary and the automatic rollback can be added once that foundation exists.

A stable foundation is not a flashy result, but at the scale of running things unattended, it matters more than anything. If it brings a little peace of mind to your nights as someone running several things alone, I would be glad.