One weekend evening I was scrolling back through the history of a small admin tool I use for my wallpaper app's delivery pipeline. A few days earlier, I had approved an agent's PR that added retry handling, and it merged without fuss.
A little further down the log, though, there was a commit of my own titled "fix retry edge cases." The lines it touched were exactly the ones that PR had added.
When I approved it, I had read the diff and watched the tests pass. Being wrong about it bothered me less than the fact that it took me almost a week to notice.
Whatever review misses only shows up after the merge. So I decided to keep a small tool around that surfaces post-merge rework as a number. This is how I built it, and the moment the numbers came out wrong along the way.
"Approved" is not a quality signal
An approval rate looks like it measures the agent's output, but it also measures how easy the PR was to approve. A tidy diff with a polite description makes it easy for a human reviewer to move on.
How many of those lines get rewritten after merge is a different kind of number. It checks, after time has passed, whether the lines actually held up.
The rule I settled on as an indie developer is simple: I approve on the day; git checks my answer two weeks later. Before adding more review steps, I wanted one tool that grades the reviews I already do.
What gets counted
I call it the survival rate: of the lines a commit added, the share still attributed to that commit N days later.
- Lines added come from
git show --numstat - The revision at N days comes from
git rev-list -1 --before=<date> - Who owns each line at that point comes from
git blame --porcelain
To limit this to agent commits, each one carries a provenance trailer, Agent: antigravity. I wrote about the trailer design itself in commit granularity and provenance trailers.
# Git 2.32 and later can add the trailer directly
git commit -m "Add retry for asset upload" --trailer "Agent: antigravity"The script
Run it at the repository root. For each agent commit it prints lines added, lines kept, and the survival rate. Commits younger than N days are skipped.
#!/usr/bin/env python3
# Count how many lines a commit added are still attributed to it N days after merge
import subprocess, sys
def git(*args):
return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout
def added_lines(sha):
# Lines added per file; binary files show "-" and are skipped
out = git("show", "--numstat", "--format=", sha)
counts = {}
for row in out.splitlines():
a, _, path = row.split("\t", 2)
if a != "-" and int(a) > 0:
counts[path] = int(a)
return counts
def surviving(sha, at_rev, path):
# Count lines still blamed on sha at at_rev
try:
out = git("blame", "--porcelain", "-w", "-M", at_rev, "--", path)
except subprocess.CalledProcessError:
return 0 # file deleted, or a rename blame could not follow
n = 0
for line in out.splitlines():
parts = line.split(" ")
# Per-line header is "sha orig_line final_line [count]"; content lines start with a tab
if parts[0] == sha and len(parts) in (3, 4):
n += 1
return n
def main(days, grep):
rows = git("log", "--no-merges", f"--grep={grep}", "--format=%H %ct").split("\n")
head_time = int(git("log", "-1", "--format=%ct").strip())
for row in filter(None, rows):
sha, ct = row.split()
limit = int(ct) + days * 86400
if limit > head_time:
continue # not old enough yet
at_rev = git("rev-list", "-1", f"--before={limit}", "HEAD").strip()
add = added_lines(sha)
total = sum(add.values())
if total == 0:
continue
kept = sum(surviving(sha, at_rev, p) for p in add)
print(f"{sha[:8]} added={total:4d} kept={kept:4d} survival={kept/total:5.0%}")
if __name__ == "__main__":
main(int(sys.argv[1]) if len(sys.argv) > 1 else 14, sys.argv[2] if len(sys.argv) > 2 else "Agent:")-w keeps whitespace-only edits from counting as rewrites, and -M keeps lines moved within the same file attributed to their original commit. If you want to follow moves across files, add -C, at the cost of speed.
Checking it on a tiny repository
I built a throwaway repository in a Linux environment (Git 2.34.1, Python 3.10) with two agent commits. Four days after the first one, a separate commit rewrote 12 of its 40 lines.
python3 survival.py 14 "Agent:"
# 8dc1f602 added= 30 kept= 30 survival= 100%
# ba896ce2 added= 40 kept= 28 survival= 70%
python3 survival.py 3 "Agent:"
# 8dc1f602 added= 30 kept= 30 survival= 100%
# ba896ce2 added= 40 kept= 40 survival= 100%The 3-day window lands before the fix, so it reads 100%; the 14-day window includes it, so it reads 70%. On this five-commit history the run took about 0.04 seconds. Real repositories cost one blame per file per commit, so I'd start with the last month of commits.
When the numbers went wrong: date order
The first time I built the test repository, the 3-day window also reported 70%. The fix happened on day four, so it should have been 100%.
The cause was my own: I had given the test commits dates that didn't match their order in history. rev-list --before walks back from the newest commit and stops at the first one older than the cutoff. If a commit with an earlier date sits after the fix, it gets picked as "day three," and the snapshot already contains the fix.
This isn't only a test-repo problem. Rebases and cherry-picks rewrite committer dates, and history order and date order can drift apart. Before counting, I now check once that the main line's dates run in order.
# Walk the first-parent chain and flag any place where dates go backwards (no output means fine)
git log --first-parent --format='%ct %h' | awk 'NR>1 && $1>prev {print "date reversal:", $2} {prev=$1}'Where to look once you have numbers
I treat the survival rate as a pointer to what to reread, not as a score. Once a number becomes a target, PRs start getting shaped to raise it.
- Pick the three lowest-surviving commits and read the follow-up fixes. Note for each whether it was a misread spec, a gap in tests, or something I missed in review
- If the same reason shows up twice, change how the work is handed over. As in cutting agent tasks into reviewable units, the first move is a smaller request
- Large commits with high survival get a second, slower read. Sometimes nobody has touched them simply because nobody has looked
The survival rate doesn't replace review; it grades it. I wrote about keeping diffs small in a month of changing how I review.
One thing to start with
The easiest first step is to add an Agent: trailer to agent commits starting today. Run the script once two weeks from now, and you'll see which PRs quietly took your time afterward.
My judgment on approval day, checked by me two weeks later. I plan to keep that loop going for a while.