The night my agent shipped nothing: giving generation agents an abstain outcome

Decouple run success from artifact count

Even with a three-valued gate, if you judge the whole run by "was there at least one ACCEPT," you have slid right back to a throughput metric. Define the run outcome on a different axis from production count.

from collections import Counter
 
 
class RunOutcome(str, Enum):
    PRODUCED = "produced"                 # >= 1 ACCEPT. net positive happened
    HEALTHY_ABSTAIN = "healthy_abstain"   # all ABSTAIN. a correct, successful skip
    DEGRADED = "degraded"                 # REJECT-dominated. input or model anomaly
    BLOCKED = "blocked"                   # never entered generation (see below)
 
 
def classify_run(results: list[GateResult]) -> RunOutcome:
    if not results:
        return RunOutcome.BLOCKED
 
    tally = Counter(r.verdict for r in results)
    accepts = tally[Verdict.ACCEPT]
    rejects = tally[Verdict.REJECT]
 
    if accepts >= 1:
        return RunOutcome.PRODUCED
 
    # zero ACCEPT still means different things by composition
    if rejects > len(results) // 2:
        # more than half were violations = not the gate's win, an input anomaly
        return RunOutcome.DEGRADED
 
    # mostly ABSTAIN = a healthy conclusion that nothing was worth shipping
    return RunOutcome.HEALTHY_ABSTAIN

The crux is treating HEALTHY_ABSTAIN as a success state. The only outcomes that page you are DEGRADED and BLOCKED. A night that ends with every candidate abstained just leaves a quiet log line and goes back to sleep. That 2 a.m. notification stopped firing because of this single classification.

There is a reason DEGRADED is kept separate too. The surface looks the same — zero ACCEPT — but "skipped" and "everything came back broken" call for entirely different moves. The former means change the inputs. The latter means suspect the model, the prompt, or corrupted reference data. If you do not decompose the surface number, you will confuse the two.

Read abstention rate as a leading indicator, not a failure

Once run outcomes are separated, the next axis is time. The single number I lean on most is not production count but the abstention rate — ABSTAIN over total candidates — and its trend.

@dataclass
class AbstentionWindow:
    """Track abstention rate over the last N runs and warn on trend."""
    window: int = 14
    history: list[float] = field(default_factory=list)
 
    def record(self, results: list[GateResult]) -> None:
        if not results:
            return
        abstains = sum(1 for r in results if r.verdict is Verdict.ABSTAIN)
        self.history.append(abstains / len(results))
        self.history = self.history[-self.window:]
 
    def rate(self) -> float:
        return sum(self.history) / len(self.history) if self.history else 0.0
 
    def trend_signal(self) -> str:
        if len(self.history) < self.window:
            return "collecting"
        recent = sum(self.history[-3:]) / 3
        base = sum(self.history[:-3]) / (len(self.history) - 3)
        if recent > base + 0.20:
            # a rising abstention rate is usually topic saturation, not quality loss
            return "input_saturation"   # -> change inputs, leave the gate alone
        if recent < base - 0.20 and self.rate() < 0.10:
            return "gate_too_loose"     # -> too much is passing, revisit the gate
        return "stable"

When trend_signal returns input_saturation, intuition whispers "the gate is too strict, loosen it." In practice the opposite held. A rising abstention rate is almost always a sign that you have written the topic area dry, and the right response is to swap the inputs — the topic source, the grounding data. Loosen the gate here and you merely stack thin artifacts onto a saturated field.

Guaranteeing grounding freshness before a run pairs well with the idea in measuring generation opportunity with a freshness SLO. Abstention rate reports input saturation; the freshness SLO reports input decay — two different angles on the same input.

A threshold only means something paired with its metric

The gate above carries a default of novelty_floor: float = 0.35. When I wrote it, I picked 0.35 because it felt about right. Looking back, that single line was the most dangerous one in the file.

If you implement novelty_score() as the inverse of similarity against your existing corpus, there are at least two obvious ways to do it, and both take only a few lines:

• Compare against every document with Python's difflib.SequenceMatcher and take the highest quick_ratio() as similarity
• Split the body into 3-gram shingles, narrow candidates with an inverted index, then take the highest Jaccard coefficient as similarity

I measured both on my own Linux environment (Python 3.10) against the 1,059 Japanese articles on Antigravity Lab — body text only, code blocks and HTML tags stripped, median length 3,372 characters. Comparing one document against the whole corpus, with a fixed random seed, gave me this:

That is a 36x difference. Read on its own, it looks like a simple swap. But the same run surfaced something that matters far more than speed. The two implementations produce nearest-neighbour scores that do not overlap at all.

Define novelty as 1 - similarity and abstain when novelty < 0.35, and here is what you get. With the Jaccard version, novelty sits around 0.9, so the condition never fires once — the novelty gate exists while judging nothing at all. With the quick_ratio version, novelty lands around 0.27, so every candidate abstains unconditionally.

The same 0.35 flipped from "can never fire" to "always fires" on nothing but an implementation swap. The first case is the nastier one——the gate appears to be running while one of its conditions is quietly dead. That failure shape is structurally the same as the one in my notes on the allowlist line that matched nothing and approved everything.

While I was there, I checked whether the fast implementation can stand in for the slow one. The difflib nearest neighbour appeared inside the inverted index's top 50 candidates in 6 of 15 documents. Treat one as a faster version of the other and the meaning of the verdict changes underneath you. Either choice is defensible — but if you change the measure, you have to re-derive the threshold in the same commit.

So I bound the threshold to a metric ID and a measured sample, and made anything outside the observed range fail at startup.

from __future__ import annotations
from dataclasses import dataclass
import statistics
 
 
class VacuousThreshold(RuntimeError):
    """A threshold outside the observed distribution. Fail at startup."""
 
 
@dataclass(frozen=True)
class CalibratedFloor:
    """Keeps a threshold bound to the metric it was measured with.
 
    If metric_id changes, the floor must be re-derived. `samples` is the
    measured nearest-neighbour series from your own corpus, already
    converted to novelty = 1 - similarity.
    """
    metric_id: str
    floor: float
    samples: tuple[float, ...]
 
    def __post_init__(self) -> None:
        if len(self.samples) < 30:
            raise VacuousThreshold(
                f"{self.metric_id}: only {len(self.samples)} samples; calibrate on 30+"
            )
        lo, hi = min(self.samples), max(self.samples)
        if not (lo <= self.floor <= hi):
            side = "never abstains" if self.floor < lo else "always abstains"
            raise VacuousThreshold(
                f"{self.metric_id}: floor={self.floor} is outside the observed range "
                f"[{lo:.3f}, {hi:.3f}] ({side})"
            )
 
    def fires_on(self) -> float:
        """Share of the measured samples this threshold would abstain on."""
        return sum(1 for v in self.samples if v < self.floor) / len(self.samples)
 
    def describe(self) -> str:
        return (
            f"{self.metric_id}: floor={self.floor:.3f} "
            f"median={statistics.median(self.samples):.3f} "
            f"expected abstention={self.fires_on():.0%}"
        )
 
 
def load_novelty_samples(path: str) -> tuple[float, ...]:
    """Read the calibration series, converting similarity into novelty."""
    with open(path, encoding="utf-8") as fp:
        sims = [float(line) for line in fp if line.strip()]
    return tuple(1.0 - s for s in sims)

Feeding it the 120 measured values from above prints this:

samples: 120
OK   : jaccard3gram-novelty-v1: floor=0.900 median=0.909 expected abstention=25%
BLOCK: jaccard3gram-novelty-v1: floor=0.35 is outside the observed range [0.850, 0.943] (never abstains)
OK   : jaccard3gram-novelty-v1: floor=0.860 median=0.909 expected abstention=1%

fires_on() is there because I wanted to be able to say out loud, at the moment of choosing a threshold, what share of candidates it would turn away. On paper 0.900 and 0.860 differ by 0.04; in expected abstention they split into 25% and 1%. Those three lines may be the ones I would most want to show the version of myself who committed a threshold without ever looking at the spread.

When grounding is missing, abstain fail-closed

The scariest thing in unattended execution is generation running while the reference data is missing. Handed an empty file or a truncated context, an agent does not fall silent and stop — it fills in something plausible and ships it. At night, that piles up until morning.

Make this fail-closed. If the inputs are not satisfied, do not even enter generation; end the run as BLOCKED. Not even ABSTAIN — stop short of the gate.

@dataclass
class GroundingCheck:
    required_paths: list[str]
    min_bytes: int = 512
    max_age_hours: float = 48.0
 
    def ready(self, fs, now_ts: float) -> tuple[bool, list[str]]:
        problems: list[str] = []
        for path in self.required_paths:
            meta = fs.stat(path)
            if meta is None:
                problems.append(f"missing: {path}")              # never fail silently
            elif meta.size < self.min_bytes:
                problems.append(f"no substance ({meta.size}B): {path}")
            elif (now_ts - meta.mtime) / 3600 > self.max_age_hours:
                age = (now_ts - meta.mtime) / 3600
                problems.append(f"stale ({age:.0f}h): {path}")
        return (len(problems) == 0, problems)
 
 
def run_generation(candidates_fn, gate, grounding, fs, now_ts, log):
    ok, problems = grounding.ready(fs, now_ts)
    if not ok:
        # do not enter generation on inputs that cannot guarantee net positive
        log.warn("BLOCKED: grounding not ready", extra={"problems": problems})
        return RunOutcome.BLOCKED, []
 
    results = [gate.evaluate(c) for c in candidates_fn()]
    outcome = classify_run(results)
    log.info("run complete", extra={"outcome": outcome, "n": len(results)})
    return outcome, results

max_age_hours is there because it is not only absence that is dangerous — a reference file whose updates have stalled is dangerous too. "Present, but three days old" is worse than missing precisely because it is harder to notice. Folding freshness into the fail-closed decision keeps quiet staleness from eroding quality.

Three things that ran counter to my expectations

After a few weeks on the three-valued gate and the net-positive SLO, three things came out the opposite of what I had predicted.

One. Raising production count made outcomes worse. During the period I pushed toward shipping at least one artifact every night, I was mixing should-abstain candidates into ACCEPT, and the average quality drifted down. From the day I allowed zero, the density of what passed went up and the downstream metrics turned toward recovery. Give yourself the freedom not to produce, and what you do produce gets better. I knew that in my head, and my hand still reached for the button every time the dashboard showed a zero — it is the kind of paradox I may never fully stop feeling.

Two. A rising abstention rate was usually good news. I braced for it as a sign of decline, but in practice it almost always meant "you've written that topic enough." The move to make is swapping inputs, not relaxing the gate. Had I read abstention as a failure rate, I would have trampled that signal every single time.

Three. Auto-repairing near-misses bred silent corruption. I once added a step that mechanically edited ABSTAIN candidates up to ACCEPT, and pulled it out fast. Repair only erased the surface violation marker; the underlying "not net positive" state remained, and the traces of it were hard to find later. Surfacing the violation at the gate and regenerating — rewritten in its own specific context — turned out faster and cleaner. That call is of a piece with treating candidates in a three-pattern validation loop.

The order to adopt this, and the first line to hold

You do not need to swap everything in at once. The lightest start is to drop in classify_run alone and take HEALTHY_ABSTAIN off the alert list. That by itself stops the nightly false alarms.

Next, add abstention-rate recording, watch two weeks of it, and only then enable trend detection. Set thresholds before you have a feel for the numbers and you will usually miss. Last, add the grounding fail-closed. One silent-failure incident and its value sinks in for good.

There is only one line to hold at the start: do not put "how much it produced" at the top of your success metrics. Hold that and the gate is freed from the pressure to loosen. You become able to count a zero-artifact night, quietly, as a success. Running an agent solo, that quiet on its own may be what lowers the cost of operations more than anything else I changed.

I am still tuning this myself, but I hope it helps the design of anyone facing an unattended generation pipeline of their own. Thank you for reading.