When Successful Automation Quietly Stops Earning Its Keep: Designing for Value Decay and Retirement
A dashboard full of green success logs often hides the most dangerous kind of failure. Here is a design — with working code — for surfacing automations that keep succeeding while producing no value, and deciding whether to retire, repair, or keep them.
When I open a scheduled-run dashboard, a satisfying column of green stares back at me. Everything succeeded. Run times are steady. Yet last week I noticed that one automation buried in that green column had produced essentially nothing for about three weeks.
The task ran every day, returned exit code 0 every day, and dutifully logged "completed" every day. The only problem: at some point its input set had dropped to zero items. It kept faithfully processing the empty set and faithfully succeeding. The apparent success rate stayed at 100% while the value quietly drained away.
As an indie developer who increasingly hands real work to agents, I've come to see this — automation that stays green while thinning out — as the most awkward failure mode of all. If something crashes, I find out. But a success that has stopped mattering will never come to me. I have to go looking for it.
The "success" log tells the quietest lie
We usually anchor monitoring on whether something broke: exit codes, exceptions, timeouts. These tell us the moment an automation is damaged.
But an automation's useful life can end before it breaks. Assumptions shift. The input source dries up. An upstream format changes so that parsing still succeeds but the payload comes back empty. In every case the code is intact, the tests are green, and yet the output trends toward zero.
I started calling this state value decay to keep it separate from breakage. A broken automation is a repair problem. A decayed automation is a judgment problem: is it worth fixing, or has it simply finished its job? You cannot answer that from a green log alone.
Why value decay is hard to observe
Decay hides for three structural reasons.
First, success and value are collapsed into the same signal. Most tasks run on "exit code 0 = success = fine." That carries no information about how many items were processed, or whether processing them mattered.
Second, decay is continuous. Twelve items yesterday, nine today, two next week — a smooth decline never produces an "abnormal" instant. Threshold-based alerts cannot catch this gentle kind of death.
Third, success is psychologically reassuring. A green log reads as "you don't need to look here." When your attention budget is small, that reassurance turns straight into a blind spot. I personally let one AdMob aggregation task spin on empty for two weeks without noticing.
✦
Thank you for reading this far.
Continue Reading
What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.
WHAT YOU'LL LEARN
✦An outcome ledger design that separates success signals from value signals to expose automations that stay green while quietly thinning out
✦A Python evaluator (with real code) that extracts retirement candidates from the last N runs of each task's valueMetric
✦A 4-quadrant decision matrix for retire vs. repair vs. keep, plus an operational log of actually retiring one task
Secure payment via Stripe · Cancel anytime
✦
Unlock This Article
Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.
The fix starts by recording success and value as separate signals. At the end of each run, the automation declares, in one line, how much value it produced. I call this an outcome ledger.
I narrowed the declaration to five fields:
taskId — which automation
ranAt — when it ran (ISO 8601, UTC)
status — the usual pass/fail (ok / fail)
valueMetric — how much value this run produced (items processed, lines changed, artifacts generated — a task-specific primary quantity)
evidence — a short trace backing the value (an output path, a summary string)
The crucial move is to make valueMetric independent of success. When status is ok but valueMetric is 0, that run is recorded as a "successful miss."
In TypeScript this becomes a tiny helper called at the end of each task, appending one line to a JSONL file.
import { appendFile } from "node:fs/promises";type Outcome = { taskId: string; ranAt: string; // ISO 8601 (UTC) status: "ok" | "fail"; valueMetric: number; // primary amount of value for this run evidence: string; // what backs the value (output path or summary) notes?: string;};export async function recordOutcome(o: Omit<Outcome, "ranAt">): Promise<void> { const line = JSON.stringify({ ...o, ranAt: new Date().toISOString() }); // Append-only. The evaluator reads nothing but this ledger. await appendFile(process.env.OUTCOME_LEDGER ?? "./ledger.jsonl", line + "\n");}// Example: the tail of a task that detects screenshot diffsconst changed = await diffScreenshots();await recordOutcome({ taskId: "visual-regression", status: "ok", valueMetric: changed.length, // number of diffs found is this task's value evidence: changed.length ? changed.slice(0, 3).join(", ") : "no diff",});
After a few days the ledger looks like the sample below. With status pinned at ok, the thinning of valueMetric finally becomes something you can follow at a glance.
Once the ledger fills up, detecting decay is not hard. For each task, look at the last N values of valueMetric and pull out the ones that keep succeeding while their value has gone to zero or keeps thinning.
Rather than swinging a single threshold, I look at two things: the trailing zero streak, and a decay ratio comparing the average of the first half against the second half. The latter is what catches automations dying smoothly.
import json, sysfrom collections import defaultdictWINDOW = 9 # how many recent runs to inspectZERO_STREAK = 5 # this many trailing zeros -> retirement candidateDECAY_RATIO = 0.4 # second-half avg / first-half avg below this -> decayingdef load(path): runs = defaultdict(list) with open(path) as f: for line in f: line = line.strip() if not line: continue r = json.loads(line) runs[r["taskId"]].append(r) return runsdef classify(recs): recent = recs[-WINDOW:] values = [r["valueMetric"] for r in recent] ok_rate = sum(1 for r in recent if r["status"] == "ok") / len(recent) trailing_zero = 0 for v in reversed(values): if v == 0: trailing_zero += 1 else: break half = max(1, len(values) // 2) first = sum(values[:half]) / half last = sum(values[half:]) / max(1, len(values) - half) decay = (last / first) if first > 0 else (0.0 if last == 0 else 1.0) if ok_rate < 1.0 and trailing_zero < ZERO_STREAK: return "repair" # it is failing -> repair if trailing_zero >= ZERO_STREAK: return "retire?" # green but a run of zeros -> consider retiring if first > 0 and decay < DECAY_RATIO: return "decaying" # thinning smoothly -> watch more closely return "healthy"def main(path): for task_id, recs in load(path).items(): if len(recs) < 3: continue verdict = classify(recs) if verdict != "healthy": tail = [r["valueMetric"] for r in recs[-WINDOW:]] print(f"[{verdict:9}] {task_id:22} values={tail}")if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "ledger.jsonl")
Running the earlier asset-audit through this prints something like [retire? ] asset-audit values=[11, 4, 1, 0, 0, 0] — a run of zeros clustered at the recent end. Nothing crashed, so existing monitoring stays silent. This evaluator still catches it.
There is one operational pitfall here. Some tasks are legitimately zero most days — anomaly detectors, for instance, where a quiet day is a healthy day. Run them through the same threshold and you'll get false positives. I gave the task definition a zeroIsHealthy: true flag and excluded such tasks from the zero-streak rule. It's an unglamorous workaround, but without it the evaluator itself loses trust.
Retire, repair, or keep
A candidate surfacing does not mean deletion. I sort each one into four quadrants across two axes: is it succeeding, and is it producing value?
Producing value
Value thinned / zero
Succeeding (green)
Healthy. Keep it running
The focus of this article. Retire, or rebuild the input
Failing (red)
Close call. Repair to protect the value
Retire first. It is broken and unwanted
For tasks landing in the top-right quadrant I ask two more questions. First, did the value hit zero because the input dried up, or because something upstream broke and the task is grabbing an empty set? The former means retirement; the latter is a repair, guided by the evidence field. Second, if I were writing this task from scratch today, would I write it? If not, I take that as a signal to fold it up.
When I decide to retire, I don't just disable it — I leave a retirement line in the ledger: when, and why. Months later, when I go looking for "that aggregation, where did it go," that one line pays off.
What I learned from actually retiring one
That opening task — the aggregation spinning on a dried-up input — I ended up retiring. The surprise wasn't the removal itself, but the effect of being able to decide it could go.
Remove one green log and the meaning of the remaining green thickens. "All green" turns from noise back into signal. For an indie developer, attention is the scarcest resource in monitoring, and a decayed automation had been eating that attention thinly and broadly. Honestly, the attention it stole cost more than its few dozen wasted runs.
There was a second dividend. Deciding to retire forces you to articulate what the task's value actually was. Choosing what valueMetric should be — item count, lines changed, minutes of manual work saved — is a question that bounces straight back onto the design of your living tasks. I ended up revising the valueMetric definition for my App Store and Google Play release-helper tasks too.
Wiring it into operations
Finally, the part that puts this into everyday use. The evaluator does not need to run daily. I run it weekly and only get notified when a candidate appears — the same philosophy as failure-only alerting.
#!/usr/bin/env bash# weekly-decay-review.sh — evaluate the ledger weekly, notify only on candidatesset -euo pipefailLEDGER="${OUTCOME_LEDGER:-$HOME/ops/ledger.jsonl}"REPORT="$(python3 evaluate_decay.py "$LEDGER")"if [ -n "$REPORT" ]; then # Send a body only when retirement/decay candidates exist printf 'Automations whose value has thinned:\n%s\n' "$REPORT" | notify-ops --title "weekly decay review"else echo "no decay candidates" # let a quiet week end quietlyfi
The rollout order I recommend is to add only the valueMetric recording to your existing tasks first, one line at a time. Leave evaluation and notification for later. It takes a few weeks of accumulated ledger before decay becomes visible at all. You don't need perfect thresholds from the start; tuning ZERO_STREAK and DECAY_RATIO to your own run frequency in production is the realistic path.
We've grown very comfortable adding automations. But the practice of seeing off an automation that has finished its job is, I think, still rarely discussed. Once a week, quietly check whether anything green is thinning out among the rest. That small habit is what keeps the remaining green worth believing. I hope it gives your own setup a useful starting point, and thank you for reading.
Share
Thank You for Reading
Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.