Letting a Background Agent Work Overnight Without Regretting It by Morning — Guardrails for Unattended Runs

Don't make "tests are green" the completion gate

As the opening failure shows, test results alone won't catch a quiet regression. The thinner the tests in an area, the more freely the agent can behave there. So before deciding whether a branch is mergeable, I run it through a light inspection layer. The aim is to confirm, from an angle other than tests, that behavior really hasn't changed.

What I actually use is a gate that inspects the shape of the change.

#!/usr/bin/env python3
# scripts/diff_guard.py — mechanically inspect an overnight branch's diff
import subprocess, sys, re
 
BASE = "origin/main"
MAX_FILE_LINES = 120          # keep this in sync with the task definition
PUBLIC_SIG = re.compile(r'^[+-]\s*export\s+(async\s+)?function\s+\w+')
 
def changed_files(branch):
    out = subprocess.run(
        ["git", "diff", "--numstat", f"{BASE}...{branch}"],
        capture_output=True, text=True, check=True).stdout
    return [l.split("\t") for l in out.splitlines() if l.strip()]
 
def main(branch):
    problems = []
    for added, removed, path in changed_files(branch):
        if added == "-" or removed == "-":          # binary
            problems.append(f"binary touched: {path}")
            continue
        if int(added) > MAX_FILE_LINES:
            problems.append(f"{path}: +{added} lines exceeds cap {MAX_FILE_LINES}")
        d = subprocess.run(["git", "diff", f"{BASE}...{branch}", "--", path],
                           capture_output=True, text=True).stdout
        sig_changes = [l for l in d.splitlines() if PUBLIC_SIG.match(l)]
        # paired +/- likely just a rename; an odd count is suspicious
        if len(sig_changes) % 2 != 0:
            problems.append(f"{path}: possible one-sided public signature change")
    if problems:
        print("BLOCK:")
        for p in problems:
            print("  -", p)
        sys.exit(1)
    print("diff_guard: ok")
 
if __name__ == "__main__":
    main(sys.argv[1])

This gate does nothing clever. It looks at two things: the size of the diff, and whether changes to public functions come in matched pairs. Even so, the kind of change that bit me in the opening — a function that disappeared from only one side — stops right here. The moment you try to reason about meaning, the check itself becomes brittle, so for unattended operation I settled on catching only anomalies that can be judged mechanically. That trade kept things stable.

I also watch the coverage delta. If coverage dropped after the change, the agent probably touched a poorly-tested area — or loosened a test to "make it pass." I compare vitest run --coverage against the prior night's number, and if it clearly fell, that branch goes on hold.

Idempotent task definitions that survive a re-run

Nightly jobs fail: sandbox timeouts, transient network errors, a CLI update. When they fail, I want to re-run. But a naive setup will apply the same formatting twice, or split an already-split function again — double application.

The way around it is to write the task as a declaration that "brings the current state closer to the target state." Not "split this function," but "this function should have a single responsibility; if it already does, do nothing." Because the agent observes the current state before acting, you can run the same task any number of times and it converges. On the launch side, one extra line that skips a new run when today's same-named branch already exists made cron's duplicate triggers stop causing accidents.

# add near the top of nightly-agent.sh
if git ls-remote --exit-code --heads origin "$BRANCH" >/dev/null 2>&1; then
  echo "branch ${BRANCH} already exists — skip"
  exit 0
fi

Finish the morning triage in a few minutes

The payoff of unattended runs only materializes when the morning check is light. If three or four branches arrive every night, reading each one carefully won't last. I look at "the machine's first verdict" first, and reserve human attention for the ones that actually need judgment.

#!/usr/bin/env bash
# scripts/morning-review.sh — clear overnight branches at a glance
git fetch origin --quiet
for b in $(git branch -r | grep "origin/agent/nightly-$(date +%Y%m%d)"); do
  name="${b#origin/}"
  files=$(git diff --name-only origin/main..."$b" | wc -l | tr -d ' ')
  if python3 scripts/diff_guard.py "$b" >/dev/null 2>&1; then
    verdict="REVIEW"          # passed the gate; on to human judgment
  else
    verdict="REJECT"          # blocked by the gate; discard by default
  fi
  printf "%-8s %-45s files=%s\n" "$verdict" "$name" "$files"
done

I don't open REJECT branches unless there's a strong reason — going to read them pulls you toward "but it's so close" and melts your time. I look only at REVIEW ones, leaning on the per-file summaries the task definition required, and merge when convinced, discard when unsure. Making discard-when-unsure the default keeps overnight automation an asset rather than a source of decision fatigue. If a discarded change is truly needed, it gets proposed again the next night.

What thirty nights actually produced

There were no dramatic wins. Of the branches generated each night, I merge a little over half. The rest get blocked by the diff gate or discarded at triage because I can't be sure behavior is unchanged. Even so, the steady drip of "maintenance I knew I should do but kept postponing" — filled-in comments, small function splits, added type annotations — making progress on its own was a real benefit.

The biggest shift might be where I now place my trust in the codebase. I used to hope the agent would fix things cleverly. Now I design so that even when it errs cleverly, the damage stays inside the fence. What you should demand of something that runs unattended isn't a ceiling on its capability but a shallow floor under its failures. It took a few cold-sweat mornings before that finally sank in.

If you want to try overnight automation, start the first night with an absurdly narrow fence: one file allowed, fifty lines of diff at most. It should feel too cramped to be useful — that's about right — and as you widen the fence gradually, you'll naturally come to see which parts of your codebase you can hand to an agent. Thanks for reading.