Deciding When to Stop a Staged Rollout, Before You Have To — Agents Watch, I Halt

Handing the watch loop to an Antigravity agent

The loop queries the Play Developer Reporting API, normalizes the metrics, then passes them to the decision function. The agent holds a read-only service account credential — never a token that could publish or halt.

# rollout_watch.py — collect rollout metrics, hand them to a three-state verdict
import time
from dataclasses import dataclass
 
MIN_SESSIONS = 800          # below this, no verdict
WARN_RATIO = 1.5            # relative to the previous stable build
HALT_RATIO = 2.0
ANR_DELAY_HOURS = 12        # ANRs surface late
 
@dataclass
class Snapshot:
    sessions: int
    crash_rate: float       # 0.0042 == 0.42%
    anr_rate: float
    hours_since_start: float
 
def fetch_snapshot(client, package: str, version_code: int) -> Snapshot:
    """Pull the last hour of metrics. Aggregation lags, so poll no faster
    than every 15 minutes."""
    m = client.query_metrics(
        package=package,
        version_code=version_code,
        metrics=["crashRate", "anrRate", "distinctUsers"],
        window="PT1H",
    )
    return Snapshot(
        sessions=int(m["distinctUsers"]),
        crash_rate=float(m["crashRate"]),
        anr_rate=float(m["anrRate"]),
        hours_since_start=m["elapsedHours"],
    )

The one-hour window is deliberate. Play's aggregation lag measured 40–70 minutes in practice. A five-minute window returned empty results, and the agent came close to reading emptiness as "zero crashes." Absent data and healthy data are not the same thing — obvious in prose, easy to lose in code.

Three states: continue, hold, halt candidate

def decide(now: Snapshot, baseline_crash: float, baseline_anr: float) -> tuple[str, str]:
    """Return (decision, reason). continue / hold / halt_candidate / insufficient_data."""
    if now.sessions < MIN_SESSIONS:
        return "insufficient_data", f"sessions={now.sessions} < {MIN_SESSIONS}"
 
    crash_ratio = now.crash_rate / max(baseline_crash, 1e-6)
    if crash_ratio >= HALT_RATIO:
        return "halt_candidate", f"crash x{crash_ratio:.2f} (>= {HALT_RATIO})"
 
    if now.hours_since_start >= ANR_DELAY_HOURS:
        anr_ratio = now.anr_rate / max(baseline_anr, 1e-6)
        if anr_ratio >= HALT_RATIO:
            return "halt_candidate", f"anr x{anr_ratio:.2f} after {now.hours_since_start:.0f}h"
        if anr_ratio >= WARN_RATIO:
            return "hold", f"anr x{anr_ratio:.2f}"
 
    if crash_ratio >= WARN_RATIO:
        return "hold", f"crash x{crash_ratio:.2f}"
 
    return "continue", f"crash x{crash_ratio:.2f}, sessions={now.sessions}"

The name halt_candidate matters. The agent proposes. It does not act.

What reaches me is one Slack message carrying that reason string. At three in the morning, crash x2.31 tells me exactly which number to open first.

Keeping the halt action outside the agent follows the same reasoning as Putting approval inside the loop — a production HITL pipeline. Delegate what can be undone. Insert a human finger where it cannot.

Gate on exposure, not on the clock

For a long time I waited 24 hours per stage. That treats a Saturday and a Tuesday as equivalent, which they are not — my Saturday 5% produced about 1.4x the sessions of a Tuesday 5%.

So the waiting unit moved from hours to estimated sessions.

Stage 3 requires an ANR verdict because by then the sample problem has dissolved and ANR finally carries signal.

Stage 4 demands two consecutive verdicts so that a single lucky poll cannot walk the release to full exposure.

What four releases actually produced

Between April and July 2026 I ran four releases across two wallpaper apps through this loop.

halt_candidate fired once. On Android 15 devices only, switching themes drove the crash rate to 2.4x baseline. Six minutes passed between the notification and the rollout being stopped.

The old me would not have found that until morning.

insufficient_data came back 31 times. Every one of those windows was a moment I would previously have spent staring at a number and worrying. A verdict of "not yet" absorbs a remarkable amount of noise.

hold fired three times. Two resolved back to continue on their own. The third turned out to be a known ANR carried over from the previous build.

What still bites

Choosing the baseline remains unsatisfying. The ratio assumes the previous build serves 100% of users, which breaks down when hotfixes stack up quickly.

For now, if the previous build has been live under 72 hours, I fall back to the build before it. That is a patch, not a design. Stratifying by OS version would be the honest answer.

The other trap: when the agent hit a Play Developer API rate limit, the exception was swallowed and the loop nearly returned continue. Exceptions now fall through to insufficient_data. Not to halt — to "no verdict." Get that wrong and a bad build sails to 100% unopposed.

Steering failure modes toward the safe side led me to the same conclusion in An unattended canary gate for major updates.

On the act of writing down when you stop

The implementation was not the hard part. Putting words to "what makes me stop" was.

Fixing the numbers meant walking back through old releases and measuring how much they normally moved. Doing that, I learned what a normal day looks like for my own apps — for the first time.

Preparing to delegate the watch is what finally made my own operation visible to me. That inversion still sits with me.

Stratification by OS version comes next. Splitting the population before taking the ratio should make those late-night decisions shorter still.

If you are about to define halt criteria for your own rollout, I hope these notes give you somewhere to start.