Measuring the Rework Rate of What You Delegate to Agents: Drawing Delegation Boundaries with Numbers, Not Instinct

Prerequisite: mark the agent's work

To measure a rework rate, you first need to tell, after the fact, which commits were the agent's output. When I have the Antigravity agent implement something, I leave a one-line marker in the commit trailer.

feat(wallpaper): add adaptive brightness sampler

Antigravity-Task: brightness-sampler
Antigravity-Agent: implement
Antigravity-Category: feature-impl

Antigravity-Category is the key. It carries a task type such as feature-impl, bugfix, refactor, config, or boilerplate. This is what lets me aggregate the rework rate per type later.

You can add trailers mechanically with git interpret-trailers, or bake them into the agent's commit-message convention so they appear automatically. Human commits get no marker. A commit without a marker is simply treated as human.

Implementation: compute the rework rate from git history

Here is the core. We read the git history, decide for each agent commit whether a human later touched the same files, and produce a rate per type. We also attach a confidence interval, because staring at a rate alone on a small sample is an easy way to fool yourself.

#!/usr/bin/env python3
"""rework_rate.py — compute the rework rate of agent output from git history."""
import subprocess, json, math
from collections import defaultdict
 
WINDOW_HOURS = 72        # human fixes within this window count as rework
WINDOW_COMMITS = 5       # and within the next N commits touching the same files
 
def git(*args):
    out = subprocess.run(["git", *args], capture_output=True, text=True, check=True)
    return out.stdout
 
def load_commits():
    """Read commit metadata and file lists, newest first."""
    fmt = "%H%x1f%ct%x1f%B%x1e"
    raw = git("log", "--no-merges", f"--pretty=format:{fmt}", "--name-only")
    commits = []
    for block in raw.split("\x1e"):
        block = block.strip("\n")
        if not block:
            continue
        head, *file_lines = block.split("\n")
        sha, ts, body = head.split("\x1f")
        files = {f for f in file_lines if f.strip()}
        cat, agent = None, False
        for line in body.splitlines():
            if line.startswith("Antigravity-Category:"):
                cat = line.split(":", 1)[1].strip()
            if line.startswith("Antigravity-Agent:"):
                agent = True
        commits.append({"sha": sha, "ts": int(ts), "files": files,
                        "category": cat, "agent": agent, "body": body})
    return commits
 
def is_rework(body):
    low = body.lower()
    return any(k in low for k in ("fix", "revert", "rework", "hotfix"))
 
def compute(commits):
    ordered = list(reversed(commits))   # walk oldest first
    stats = defaultdict(lambda: {"total": 0, "reworked": 0})
    for i, c in enumerate(ordered):
        if not c["agent"] or not c["category"]:
            continue
        stats[c["category"]]["total"] += 1
        deadline = c["ts"] + WINDOW_HOURS * 3600
        touched_after, reworked = 0, False
        for later in ordered[i + 1:]:
            if later["ts"] > deadline:
                break
            if not (later["files"] & c["files"]):
                continue
            touched_after += 1
            if not later["agent"] and is_rework(later["body"]):
                reworked = True
                break
            if touched_after >= WINDOW_COMMITS:
                break
        if reworked:
            stats[c["category"]]["reworked"] += 1
    return stats
 
def wilson(reworked, total, z=1.96):
    """Wilson interval lower/upper bound. Guards against overconfidence on small samples."""
    if total == 0:
        return (0.0, 0.0, 0.0)
    p = reworked / total
    denom = 1 + z * z / total
    center = (p + z * z / (2 * total)) / denom
    half = (z * math.sqrt(p * (1 - p) / total + z * z / (4 * total * total))) / denom
    return (p, max(0.0, center - half), min(1.0, center + half))
 
if __name__ == "__main__":
    stats = compute(load_commits())
    rows = []
    for cat, s in sorted(stats.items()):
        p, lo, hi = wilson(s["reworked"], s["total"])
        rows.append({"category": cat, "n": s["total"],
                     "rework_rate": round(p, 3),
                     "ci_low": round(lo, 3), "ci_high": round(hi, 3)})
    print(json.dumps(rows, ensure_ascii=False, indent=2))

The output looks like this. For each category you get the sample size n, the rework rate, and the lower and upper bounds of the confidence interval.

[
  { "category": "boilerplate",  "n": 41, "rework_rate": 0.049, "ci_low": 0.013, "ci_high": 0.163 },
  { "category": "feature-impl", "n": 33, "rework_rate": 0.394, "ci_low": 0.245, "ci_high": 0.564 },
  { "category": "bugfix",       "n": 18, "rework_rate": 0.278, "ci_low": 0.126, "ci_high": 0.509 },
  { "category": "refactor",     "n": 12, "rework_rate": 0.500, "ci_low": 0.251, "ci_high": 0.749 }
]

Why the confidence interval is never optional

The refactor row above shows a rate of 0.50, but n is only 12. Its interval spans 0.25 to 0.75 — almost as wide as "we know nothing." If you looked at the rate alone and concluded "refactoring can't be delegated," you would be swayed by a handful of coincidences.

By contrast, boilerplate has n of 41, and even its upper bound is 0.16. You can read that as "safe to delegate with confidence." feature-impl has both a large n and a high lower bound; that is clearly a caution zone.

Since I started looking at the interval, I have grown a habit of not over-narrating the numbers. For thin-sample areas, I can now choose the decision of "not deciding yet."

The decision rule: delegate, review, or do it yourself

Once you have rates and intervals, the rest is turning them into action. I use the lower bound of the interval as my basis, so I do not lean optimistic.

The thresholds themselves shift with your business and risk tolerance. For code tied directly to revenue, even 0.15 may be too high. For a throwaway prototype, 0.5 may be fine. What matters is naming the threshold once and measuring every category against the same ruler.

A record of actually pulling one category back

In my own repo, the lower bound for feature-impl stayed high, hovering around 0.24 for a long time. The decision rule said "require review," but when I dug into the breakdown, most of the rework clustered around "misread specifications." The cause was not the agent's implementation ability; it was the vagueness of my instructions.

So instead of stopping delegation, I changed what came before it. Before handing off a feature, I now write the acceptance criteria as a three-line list first, and leave that in the commit trailer too. Over two weeks of adding this one step, the rework rate for feature-impl dropped from 0.39 to 0.19. The lower bound came down from 0.24 into the low 0.10s.

The numbers were not saying "don't delegate" — they were saying "change how you hand it over." Had I gone on instinct alone, I would probably have pulled feature implementation back wholesale as "too soon," and lost a delegation I had every reason to keep.

Where this measurement fits, and where it does not

This approach suits work whose output lands in files, commit by commit. Implementation, fixes, refactoring — areas where git can trace who touched what afterward.

It does not suit work whose output rarely shows up as a file diff, such as investigation, design, or interactive back-and-forth. Those deserve a different metric — the adoption rate of proposals, say, or how often a direction changed after a conversation. Measurement is not a universal ruler; you choose the shape that fits each domain. I learned that while running it.

One more caution is the discipline of marking. If the practice of leaving a trailer on agent commits breaks down, this measurement quietly starts to lie. I embed the trailer in my Antigravity commit-convention template from the start, and I have made it a promise to myself never to strip it by hand.

Closing thoughts

You can draw the delegation line by instinct. With experience, instinct is reasonably accurate. But for me, being able to check my hits and misses against numbers afterward made the work far less stressful — because I can redraw a line I got wrong the following week.

The rework rate is not a flashy metric. Yet these days it feels like a quiet companion that teaches me how to work with an agent. If you are unsure about a delegation decision, try starting with one line of marker in your own commit history. I hope it helps with your own work. Thank you for reading.