The evening I started sketching a setup where an agent opens pull requests on my behalf, I had a number in my head, and it was a bad one: agent PRs would sit longer than human PRs. A machine can open them faster than I can read them, so the queue would only grow.
I decided to check that guess against public data instead of my own feelings. The repository I picked was dotnet/runtime, where GitHub's Copilot coding agent opens PRs as a matter of routine. For everything created in July and August 2026, I pulled 123 PRs opened by the agent and 123 opened by people, and counted.
The short version first: the guess was backwards. The first human reaction on an agent PR arrived at a median of 1.79 hours; on a human PR it took 5.76 hours. But if you read that as "agent PRs get reviewed with care," you will get burned on a solo project. The reason sits in how I counted, so I'll start there.
Counting only the first human reaction
My first question was what to call "waiting." Time to merge depends on diff size and CI as much as on review. So I counted one thing: the time from a PR being opened until someone other than its author leaves anything on it.
"Anything" means a review, an inline comment on the diff, or a plain comment. Bots are excluded; only accounts whose type is User count. An agent PR's author is not a person, which means the comment from the human who handed the task over lands in this count too. That detail matters later.
| Item | Agent side | Human side |
|---|---|---|
| Population | PRs matching author:app/copilot-swe-agent | All other PRs whose author is a User |
| Period | Created 2026-07-01 to 2026-08-31 | Same (collected on 2026-10-14) |
| Count from the search API | 233 | 1,697 (bots included) |
| Sample | 123 | 123 |
| Sampling | 14 at random per week, from the newest 100 search hits | Same |
| States kept | Merged, closed, and still open | Same (dropping open PRs would remove exactly the ones that waited) |
I split the sample by week because my first version searched the whole period once and took the first 100 hits, and that leaned hard toward the start of the period. Drawing about 14 per week fixed it.
The measurement script
It uses only the standard library. It runs without a token, but each PR costs around four API calls, so passing a token to get the 5,000-requests-per-hour limit is the safer way.
# pr_review_lag.py
#!/usr/bin/env python3
"""Count, for a public repo, how long PRs opened by an agent and PRs opened by people
wait for the first human reaction, how long they take to resolve, and how big they are.
Standard library only. With GITHUB_TOKEN set the rate limit is 5,000 requests/hour.
Usage: python3 pr_review_lag.py OWNER/REPO 2026-07-01 2026-08-31 123
"""
import json, os, random, statistics, sys, time
import urllib.error, urllib.parse, urllib.request
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
API = "https://api.github.com"
TOKEN = os.environ.get("GITHUB_TOKEN", "")
def get(path, params=None):
url = API + path + ("?" + urllib.parse.urlencode(params) if params else "")
headers = {"Accept": "application/vnd.github+json", "User-Agent": "pr-review-lag"}
if TOKEN:
headers["Authorization"] = "Bearer " + TOKEN
for attempt in range(5):
try:
with urllib.request.urlopen(urllib.request.Request(url, headers=headers), timeout=30) as r:
return json.load(r)
except urllib.error.HTTPError as e:
if e.code in (403, 429, 502, 503):
time.sleep(15 * (attempt + 1)) # secondary rate limits clear if you wait
continue
raise
raise RuntimeError("giving up: " + url)
def ts(s):
return datetime.fromisoformat(s.replace("Z", "+00:00")) if s else None
def hours(a, b):
return (b - a).total_seconds() / 3600 if a and b else None
def week_windows(start, end):
"""Split the period into 7-day windows so the sample does not lean toward the start."""
from datetime import date, timedelta
d, e, out = date.fromisoformat(start), date.fromisoformat(end), []
while d <= e:
out.append((d.isoformat(), min(d + timedelta(days=6), e).isoformat()))
d += timedelta(days=7)
return out
def search_prs(repo, extra, start, end, limit):
wins = week_windows(start, end)
per = max(1, -(-limit // len(wins))) # per-window quota, rounded up
items = []
tag = "agent" if "copilot" in extra and not extra.startswith("-") else "human"
for a, b in wins:
cache = f"win_{tag}_{a}.json"
if os.path.exists(cache): # search is capped at 30 calls/min, so keep each window and make the run resumable
items += json.load(open(cache))
continue
q = f"repo:{repo} is:pr created:{a}..{b} {extra}"
got = get("/search/issues", {"q": q, "per_page": 100, "sort": "created", "order": "desc"})["items"]
got = [i for i in got if i["user"]["type"] == "User" or "copilot" in i["user"]["login"].lower()]
random.Random(14).shuffle(got)
got = got[:per]
json.dump(got, open(cache, "w"))
items += got
time.sleep(6)
return items
def first_human_reaction(repo, pr):
author = pr["user"]["login"]
n = pr["number"]
opened = ts(pr["created_at"])
times = []
for r in get(f"/repos/{repo}/pulls/{n}/reviews", {"per_page": 100}):
u = r.get("user") or {}
if u.get("type") == "User" and u["login"] != author and r.get("submitted_at"):
times.append(ts(r["submitted_at"]))
for c in get(f"/repos/{repo}/pulls/{n}/comments", {"per_page": 100}):
u = c.get("user") or {}
if u.get("type") == "User" and u["login"] != author:
times.append(ts(c["created_at"]))
for c in get(f"/repos/{repo}/issues/{n}/comments", {"per_page": 100}):
u = c.get("user") or {}
if u.get("type") == "User" and u["login"] != author:
times.append(ts(c["created_at"]))
return hours(opened, min(times)) if times else None, len(times)
def collect(repo, label, extra, start, end, want, budget_s=None):
"""Freeze the sampled PR numbers in pool_<label>.json and append finished rows to
rows_<label>.jsonl. If the run stops midway, run it again and it resumes."""
pool_file, rows_file = f"pool_{label}.json", f"rows_{label}.jsonl"
if os.path.exists(pool_file):
pool = json.load(open(pool_file))
else:
pool = [i["number"] for i in search_prs(repo, extra, start, end, want)][:want]
json.dump(pool, open(pool_file, "w"))
done = {}
if os.path.exists(rows_file):
for line in open(rows_file):
r = json.loads(line)
done[r["n"]] = r
t0 = time.time()
def one(n):
if budget_s and time.time() - t0 > budget_s:
return None
pr = get(f"/repos/{repo}/pulls/{n}")
lag, reactions = first_human_reaction(repo, pr)
end_at = ts(pr["merged_at"] or pr["closed_at"])
return {
"n": n, "kind": label, "state": "merged" if pr["merged_at"] else pr["state"],
"lag_h": lag, "reactions": reactions,
"resolve_h": hours(ts(pr["created_at"]), end_at),
"lines": pr["additions"] + pr["deletions"], "files": pr["changed_files"],
}
todo = [n for n in pool if n not in done]
with ThreadPoolExecutor(max_workers=6) as ex, open(rows_file, "a") as out:
for row in ex.map(one, todo):
if row:
out.write(json.dumps(row) + "\n")
done[row["n"]] = row
return [done[n] for n in pool if n in done], len(pool)
def pct(xs, p):
xs = sorted(xs)
return xs[min(len(xs) - 1, int(len(xs) * p))] if xs else None
def summarize(rows):
lags = [r["lag_h"] for r in rows if r["lag_h"] is not None]
res = [r["resolve_h"] for r in rows if r["resolve_h"] is not None]
n = len(rows)
return {
"n": n,
"merged_pct": round(100 * sum(r["state"] == "merged" for r in rows) / n, 1),
"open_pct": round(100 * sum(r["state"] == "open" for r in rows) / n, 1),
"no_human_reaction_pct": round(100 * (n - len(lags)) / n, 1),
"lag_median_h": round(statistics.median(lags), 2) if lags else None,
"lag_p90_h": round(pct(lags, 0.9), 2) if lags else None,
"resolve_median_h": round(statistics.median(res), 2) if res else None,
"lines_median": statistics.median(r["lines"] for r in rows),
"reactions_median": statistics.median(r["reactions"] for r in rows),
}
if __name__ == "__main__":
repo, start, end, want = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
budget = int(os.environ.get("BUDGET_S", "0")) or None
agent, na = collect(repo, "agent", "author:app/copilot-swe-agent", start, end, want, budget)
human, nh = collect(repo, "human", "-author:app/copilot-swe-agent", start, end, want, budget)
if len(agent) < na or len(human) < nh:
print(f"partial: agent {len(agent)}/{na}, human {len(human)}/{nh} - run again to resume")
sys.exit(0)
out = {"repo": repo, "window": [start, end], "agent": summarize(agent), "human": summarize(human)}
json.dump({"summary": out, "rows": agent + human}, open("pr_review_lag.json", "w"), ensure_ascii=False)
print(json.dumps(out, ensure_ascii=False, indent=2))Three choices are worth explaining. Search results are saved per weekly window, because the search API allows 30 calls a minute and a secondary rate limit (a 403) can stall you for tens of seconds; if the run dies, running it again picks up where it stopped. The PR numbers are frozen before any details are fetched, so every run compares the same sample. And a PR that never got a reaction keeps its lag as None. Folding it in as zero would drag the median down and make the numbers look better than they are.
Running it:
export GITHUB_TOKEN=YOUR_API_KEY # read-only access to public repos is enough
python3 pr_review_lag.py dotnet/runtime 2026-07-01 2026-08-31 123On my machine, freezing the sample took a few minutes and fetching the PR details took a few more—most of it was waiting on search. The summary (excerpt):
{
"agent": { "n": 123, "merged_pct": 72.4, "open_pct": 4.1, "no_human_reaction_pct": 13.0,
"lag_median_h": 1.79, "lag_p90_h": 49.57, "resolve_median_h": 31.15,
"lines_median": 46, "reactions_median": 4 },
"human": { "n": 123, "merged_pct": 72.4, "open_pct": 9.8, "no_human_reaction_pct": 13.8,
"lag_median_h": 5.76, "lag_p90_h": 71.41, "resolve_median_h": 45.09,
"lines_median": 84, "reactions_median": 2 }
}