Reading the Same History in 38 Seconds — or 0.4: Handing a Read-Only .git to Your Agent

The Obvious Loop Took 38 Seconds

I measured against synthetic repositories rather than a real one, because I wanted to vary exactly one thing: commit count. A generator streams commits into git fast-import — 60 modules, implementation and test files co-changing 72% of the time, a shared registry dragged in 18% of the time, all under a fixed random seed. Generating 8,000 commits takes 2.2 seconds, which makes redoing an experiment painless.

# gen_fixture.py (excerpt): build synthetic history via a fast-import stream
files = [f"src/api/mod_{m}.py"]
if rng.random() < 0.72:
    files.append(f"tests/test_mod_{m}.py")   # impl/test co-change
if rng.random() < 0.18:
    files.append("src/core/registry.py")     # shared-file entanglement
# ... assemble blobs and commits as text, then feed the stream once
subprocess.run(["git", "-C", path, "fast-import", "--quiet"],
               input=stream, check=True)

The first implementation is the one anybody writes first. Get the commit list from git rev-list, then ask each commit for its changed files with git diff-tree and its author with git show -s.

# The obvious implementation: one git invocation per commit
shas = run(["rev-list", "HEAD"], repo).split()
for sha in shas:
    out = subprocess.run(
        ["git", "-C", repo, "diff-tree",
         "--no-commit-id", "--name-status", "-r", sha],
        capture_output=True, text=True, check=True).stdout
    files = [l.split("\t", 1)[1] for l in out.splitlines() if "\t" in l]
    churn.update(files)
    # author fetched the same way, via git show -s --format=%an

It works. The output is correct. It also takes 38.4 seconds at 8,000 commits.

Adding 38 seconds to every session start was never going to survive. That option died here.

The Same Output From a Single git Call

The second implementation folds everything into one invocation. git log --name-status emits every commit's changed files and author in a single run, and a single pass over that output rebuilds the same aggregates.

# One-pass implementation: git starts exactly once
def build_onepass(repo):
    out = run(["log", "--name-status", "--pretty=format:@%H|%an"], repo)
    churn, pairs, authors = Counter(), Counter(), Counter()
    files, n = [], 0
    def flush():
        churn.update(files)
        for a, b in itertools.combinations(sorted(files)[:10], 2):
            pairs[(a, b)] += 1
    for line in out.splitlines():
        if line.startswith("@"):          # commit boundary
            if n:
                flush()
            files, n = [], n + 1
            authors[line.split("|", 1)[1]] += 1
        elif "\t" in line:                # a name-status line
            files.append(line.split("\t")[-1])  # rename lines: take the new path
    if files or n:
        flush()
    return pack(churn, pairs, authors, n)

Co-change pairs are capped at the first 10 files per commit. The occasional bulk commit — a repository-wide reformat, say — would otherwise explode the pair count quadratically and let a single commit dominate the statistics.

The results, after confirming both implementations agree on the top hotspots and top co-change pairs:

Each value is the median of three runs (the 8,000-commit per-commit loop ran once, for time reasons). Environment: Linux 6.8, 4 vCPUs, 3.8 GB RAM, git 2.34.1, Python 3.10.12. These numbers are environment-dependent — rerun the scripts on your own machine before relying on them.

The Cost Was Never the Data. It Was Process Startup

My prediction going in was the opposite. I expected the single git log to be the risky one: asking one process to dump name-status output for 8,000 commits sounds expensive, and I worried about the memory footprint of catching it all.

What was actually expensive was starting processes. Divide the per-commit loop's time by the number of commits and a pattern appears:

Roughly 4.7 ms per commit, nearly flat across repository sizes. The loop launches two subprocesses per commit — diff-tree and show — so each git process costs about 2.3 to 2.4 ms in startup and teardown alone. The time was disappearing at the doorway, not inside the work.

On its own, 2.3 ms is invisible. Multiplied by two spawns across 8,000 commits, it becomes 38 seconds. The one-pass version receives the same volume of information and finishes in 0.38 seconds; a plain Python parsing loop was more than fast enough.

The lesson transfers directly to agent tool design. Build a tool that returns information about one commit, and an agent that diligently loops over it pays full spawn cost on every iteration. History aggregation belongs inside the tool, folded into a single git invocation, returning only the finished summary.

What a Read-Only .git Quietly Costs You

Every measurement so far ran against a writable clone. Since the whole point is a read-only .git, leaving that untested felt like skipping the part that mattered. I ran chmod -R a-w across .git and measured again.

The aggregation side came through untouched. git log --name-status, git rev-list --count, and git blame all returned without a single warning, in the same time as before. The pack-building path is read-only end to end, so that much was expected.

What surprised me was the neighbor: git status and git diff.

When those run, git re-reads and re-hashes every file whose stat data disagrees with the index, then writes the refreshed result back. If the write succeeds, later runs only compare stat data. When the write cannot happen, every run starts over.

Here are three consecutive git status --short runs on 1,200 files of 400KB each (a 474MB working tree), after touching every file to change only its mtime.

With a writable .git, the second run is effectively free. Read-only pays the same 1.3 seconds indefinitely. Every time the agent checks what changed, the time goes into recomputing hashes rather than finding differences.

No error surfaces here either. Once git discovers it cannot take the lock, it silently gives up on writing the index back. The only loud failure is on the write path: git add stops immediately with fatal: Unable to create '/repo/.git/index.lock': Permission denied. The read-only constraint showing up as "reads get expensive" rather than "writes fail" was the part I had not anticipated.

--no-optional-locks does not help. It tells git not to attempt the lock; it does not stop the re-hashing that made the call expensive.

What did help was keeping a writable copy of the index elsewhere and pointing GIT_INDEX_FILE at it.

# .git stays read-only; only a working copy of the index is writable
export GIT_INDEX_FILE=/tmp/agent-idx/index
mkdir -p /tmp/agent-idx
cp /repo/.git/index "$GIT_INDEX_FILE"
 
git status --short   # 1.32 s the first time, 0.00 s after that

I verified with md5 that git status --short produces byte-identical output across all three variants (plain, --no-optional-locks, and GIT_INDEX_FILE). The speedup does not come from detecting less; it comes from having somewhere to write the result.

That copy does go stale, though. Switch branches and its contents no longer match reality, so treat it as something you rebuild at session start. I regenerate mine inside the pack script, at the same point as the history check.

One more boundary: this gap only appears when the files are large. Repeating the same procedure on 4,000 source files of 3KB each, read-only git status finished in 0.06 seconds and the difference disappeared into measurement noise. Re-hashing costs scale with bytes, so a text-heavy repository will never notice. If you are carrying images or model weights, measure before you assume.

Shallow Clones Corrupt the Pack Silently

Back to this morning. What happens when this pack is generated against a --depth 1 copy?

No error. The pack builds successfully. Only the contents are meaningless.

In my measurement, cloning the 2,000-commit repository with --depth 1 left git rev-list --count HEAD returning 1, and the pack's top hotspot became "docs/changelog.md, changed once" — whichever files happened to ride the final commit, crowned as the most-churned code in the project.

The silence is the dangerous part. The aggregation succeeds, the JSON is well-formed, and the agent reads it in good faith: "the most frequently changed file in this repository is docs/changelog.md." A false context is worse than no context.

The fix is blunt: verify the history actually exists before building anything, and fail closed if it does not.

def assert_history(repo, min_commits=50):
    count = int(run(["rev-list", "--count", "HEAD"], repo))
    shallow = os.path.exists(os.path.join(repo, ".git", "shallow"))
    if shallow or count < min_commits:
        raise SystemExit(
            f"history too short: commits={count} shallow={shallow} — "
            "fetch full history before building the pack")

The .git/shallow check exists because a commit-count threshold alone cannot distinguish a genuinely young repository from a truncated one. Provisioning for automated environments defaults to depth-1 clones more often than not, so if you build anything on read-only .git, I would put this guard in on day one.

Renamed Files Were Counted as Paths That Do Not Exist

When I pointed the pack at one of my own repositories, an unfamiliar row showed up among the hotspots: a path with a tab character in the middle, as if src/a.py and src/b.py had been glued together.

The cause was one line in the one-pass parser. With rename detection on (the default in git 2.34.1), git log --name-status emits a rename commit as three columns — R100, the old path, the new path.

@fa54302…|t
R100	src/a.py	src/b.py

My code split on the tab once and took everything after the first column as the path. On a rename line that "path" becomes a single string, src/a.py<TAB>src/b.py. I checked it on a tiny repository: edit a.py three times, rename it to b.py, edit once more.

One rename, one phantom path. The same string also leaks into the co-change pairs, so this is not only a hotspot problem. My 8,000-commit fixture contained no renames, which is how "both implementations produce identical output" slipped past it. That equality check only means something if the input contains renames.

There are two fixes.

Either way a file's history is split across its old and new names, so in a repository with many renames the hotspot numbers will undercount. Tuning the -M similarity threshold can push the tracking further, but if you want to stay on a single pass, I would rather accept the split. The pack is a rough pointer to where to be careful, not a guarantee of per-file provenance.

The code earlier in this article now takes the last column.

Verifying means checking that the input has the same shape as reality, not just that the outputs agree. Matching output says nothing about shapes the input never contained — renames, merges, binaries.

Deciding What Goes In the Pack — Where to Cut the Top N

Until now the step that turns counters into JSON has only appeared as pack(). That function is where the actual design decisions live, so here it is.

def pack(churn, pairs, authors, n, top_files=14, top_pairs=10):
    total = sum(churn.values()) or 1
    return {
        "schema": "history-pack/1",
        "commits": n,
        "hotspots": [
            {"path": p, "changes": c, "share": round(c / total, 3)}
            for p, c in churn.most_common(top_files)
        ],
        "cochange": [
            # conf: of the commits touching a, the share that also touched b
            {"pair": [a, b], "together": c, "conf": round(c / churn[a], 2)}
            for (a, b), c in pairs.most_common(top_pairs)
        ],
        "authors": [{"name": a, "commits": c} for a, c in authors.most_common(5)],
    }

Where you cut the top N is what sets the pack size. Against the same 2,000-commit repository, 12 files and 8 pairs produced 1,455 bytes, 14 and 10 produced 1,724 bytes, and 16 and 10 produced 1,834 bytes. That works out to roughly 55 bytes per hotspot and 80 bytes per co-change pair, varying with how long your paths are.

I settled on 14 and 10. Cutting hotspots at 10 sometimes splits a module's implementation from its test, while stretching to 20 starts admitting files that changed exactly once. On the co-change side, past 10 pairs the confidence values drop below 0.3 and stop being useful to whoever reads them.

I include conf because the raw together count cannot stand on its own. A file that changes constantly changes alongside everything. Here is an excerpt from real output.

{
  "schema": "history-pack/1",
  "commits": 2000,
  "hotspots": [
    { "path": "src/core/registry.py", "changes": 367, "share": 0.095 },
    { "path": "src/api/mod_34.py",    "changes": 45,  "share": 0.012 }
  ],
  "cochange": [
    { "pair": ["src/api/mod_1.py", "tests/test_mod_1.py"],
      "together": 35, "conf": 0.83 }
  ],
  "authors": [
    { "name": "Masaki", "commits": 1545 },
    { "name": "ci-bot", "commits": 455 }
  ]
}

A conf of 0.83 says that 83% of the commits touching mod_1.py also touched its test. When you want the agent to flag a change that edited an implementation and left its test alone, that number is the threshold. registry.py, by contrast, appears in 9.5% of all commits — that is not a co-change partner but a shared file, and it should be read as "editing this reaches widely."

share exists for the same reason. The absolute changes count tracks how old the repository is, which makes packs from different projects impossible to compare. A ratio stays readable on its own.

The output is not embedded in the prompt. It lands at .agent/history-pack.json and joins the set of files the agent reads first. Three lines in AGENTS.md were enough.

## Repository history
 
Read `.agent/history-pack.json` before you start.
`hotspots` shows where changes concentrate; `conf` under `cochange`
is how often the second file changed whenever the first one did.

A file rather than an embed, because the agent should read it only when it needs it. 1.7KB is small, but pinning it to every prompt makes it accumulate as the session grows. I leave it on disk and let the agent decide when to open it.

Settling Into a Routine

With the numbers in hand, the setup I landed on is simple.

The pack is rebuilt at every session start. At 0.38 seconds and 1.7 KB for 8,000 commits, it vanishes into the startup noise. The output goes into the working directory as a file — not inlined into the prompt — and sits among the first things the agent reads.

The exclusions are worth recording too. Full diff bodies never get passed; they are large, and their presence defeats the point of a summary. Whole-repository blame sweeps are also out of this pack. Hotspots and co-change already answer "where should I be careful," and line-level provenance can be fetched on demand for the specific file that needs it.

Read-only .git is not a flashy change. But being able to implement "pass the information, withhold the authority" plainly, without workarounds, feels like solid ground gained.

Start by running git rev-list --count HEAD on the repository your agent actually sees. Whether that number matches your expectation is the first real step toward handing history to an agent — and if the structure of these scripts saves you a detour, I am glad for it.