The File Is Right There in ls, and Your Agent Still Can't Open It

Three Axes You Must Not Conflate

Worth pausing to sort this out. The symptom is singular — "the name doesn't match" — but the causes run along three independent axes. Try to fix them as one thing and you will miss some.

Axis 1: Unicode normalization (NFC / NFD)

What we just walked through. It bites kana with voiced marks and accented Latin characters. カ + ゛ versus ガ. e + ´ versus é. Same glyph, different bytes.

If you touch Japanese filenames at all, this is your most frequent offender.

Axis 2: Case folding

APFS on macOS is case-insensitive by default. ext4 on Linux is not.

An agent looks for utils.ts, doesn't find it, creates Utils.ts. On macOS that silently overwrites the same file. In Linux CI, you now have two. Works locally, fails in CI — the classic shape. It is also harder to spot than the Japanese case, which makes "fixed it" and "actually fixed it" easy to confuse.

Axis 3: Invisible and confusable characters

Ideographic space (U+3000), non-breaking space, a trailing ASCII space, zero-width space. Fullwidth alphanumerics.

Normalization does not touch these. Convert to NFC and an ideographic space is still an ideographic space. You need separate detection.

The third one is the nastiest, precisely because nothing breaks. It looks like a non-problem right up until an agent quotes that path.

Stop Looking. Start Scanning Bytes.

The approach follows from the diagnosis: stop trusting eyes, make a machine count.

I wrote a small script that walks every tracked path and reports anomalies on all three axes. This is the version actually running in my repos.

#!/usr/bin/env python3
"""filename_gate.py — check every tracked path across three axes.
 
  python3 filename_gate.py <repo_root>
  exit 0 = clean / 1 = needs attention
"""
import subprocess
import sys
import unicodedata
from collections import defaultdict
from pathlib import Path
 
# Axis 3: characters normalization will never remove
SUSPICIOUS_CHARS = {
    "　": "ideographic space",
    " ": "non-breaking space",
    "​": "zero-width space",
    "‎": "left-to-right mark",
    "": "BOM",
}
 
 
def tracked_paths(repo: Path):
    """Ask Git for raw bytes rather than its escaped, quoted rendering."""
    out = subprocess.run(
        ["git", "-C", str(repo), "ls-files", "-z"],
        capture_output=True, check=True,
    ).stdout
    for chunk in out.split(b"\x00"):
        if chunk:
            yield chunk.decode("utf-8", errors="surrogateescape")
 
 
def check(repo: Path):
    findings = []
    casefold_groups = defaultdict(list)
 
    for path in tracked_paths(repo):
        name = Path(path).name
 
        # --- Axis 1: normalization ---
        nfc = unicodedata.normalize("NFC", path)
        if path != nfc:
            findings.append(("NFD", path, f"normalizes to {nfc!r}"))
 
        # --- Axis 2: case collision candidates ---
        casefold_groups[nfc.casefold()].append(path)
 
        # --- Axis 3: invisible / confusable ---
        for ch, label in SUSPICIOUS_CHARS.items():
            if ch in name:
                findings.append(("CHAR", path, f"contains {label} (U+{ord(ch):04X})"))
        if name != name.strip():
            findings.append(("CHAR", path, "leading or trailing whitespace"))
        if any(unicodedata.east_asian_width(c) == "F" for c in name):
            findings.append(("CHAR", path, "contains fullwidth alphanumerics"))
 
    for _, group in casefold_groups.items():
        if len(group) > 1:
            findings.append(("CASE", " / ".join(group), "paths differing only by case"))
 
    return findings
 
 
def main():
    repo = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
    findings = check(repo)
    if not findings:
        print("✅ filename_gate: clean")
        return 0
    by_kind = defaultdict(list)
    for kind, path, note in findings:
        by_kind[kind].append((path, note))
    for kind in ("NFD", "CASE", "CHAR"):
        for path, note in by_kind.get(kind, []):
            print(f"[{kind}] {path}\n        └ {note}")
    print(f"\n❌ filename_gate: {len(findings)} finding(s)")
    return 1
 
 
if __name__ == "__main__":
    sys.exit(main())

One implementation detail is not optional: the -z on git ls-files.

Without it, Git returns non-ASCII paths as octal-escaped strings inside quotes, like "\346\226\207...". Read those naively and you never see the bytes you came to inspect. My first draft did exactly that and cheerfully reported everything clean — the scanner was scanning the wrong thing.

errors="surrogateescape" earns its place for the same reason. Malformed bytes get carried through and reported instead of raising. A checker that dies on the input it exists to check is not much of a checker.

First real run:

[NFD] content/reference/壁紙_配色ガイド.md
        └ normalizes to 'content/reference/壁紙_配色ガイド.md'
[NFD] content/reference/申請_チェックリスト.md
        └ normalizes to 'content/reference/申請_チェックリスト.md'
[CASE] src/lib/Utils.ts / src/lib/utils.ts
        └ paths differing only by case
[CHAR] docs/リリース　メモ.md
        └ contains ideographic space (U+3000)
 
❌ filename_gate: 14 finding(s)

Lines one and two of that output look like the same string. That is the whole problem in one screenshot. Rendering is identical; content is not. No human review will ever catch this.

Close It in Three Layers

Detection is step one. Preventing recurrence is the rest. Rather than chase perfection in one place, I split the job across three layers with different characters, and the gaps came out smaller.

Layer 1: narrow the entrance (this did the heavy lifting)

I added one paragraph to AGENTS.md:

## Filename rules
 
- Every path added to this repo uses ASCII lowercase, digits,
  and hyphens only, for both directories and filenames.
- Japanese never goes in a filename. It goes inside the file —
  the frontmatter title, for instance.
- To reference an existing Japanese filename, never reconstruct
  the name from context. Read it from `git ls-files` output.

Underwhelming, I know. But across six weeks of measurement, roughly 90% of newly introduced name drift stopped right here, leaving the other 10% for layers two and three. Removing the place where the mess originates beats handling the mess cleverly. I'd recommend taking them in that order.

The third bullet is the agent-specific one, and it mattered. "Don't reconstruct it — read it" directly addresses the root cause. If the problem is that model output is NFC, the surest fix is not asking the model to produce the string at all.

Layer 2: fail it mechanically in CI

Rules in a document get broken. Not maliciously — just forgotten. So a machine watches.

#!/usr/bin/env bash
# ci-filename-gate.sh — check only paths this branch touched
set -euo pipefail
 
BASE="${1:-origin/main}"
 
# -z again: full scans are wasteful on large repos
CHANGED=$(git diff --name-only -z "${BASE}"...HEAD | tr '\0' '\n' | grep -c . || true)
if [ "${CHANGED}" -eq 0 ]; then
  echo "No paths changed"
  exit 0
fi
 
python3 scripts/filename_gate.py . || {
  cat <<'MSG'
 
────────────────────────────────────────────
Blocked by the filename gate.
 
  NFD  → rename the path to an ASCII name
  CASE → resolve the case-only duplicate
  CHAR → strip fullwidth / invisible characters
 
Rename in two steps:
  git mv old temp && git mv temp new
(A single-step case-only rename is ignored on macOS.)
────────────────────────────────────────────
MSG
  exit 1
}

That last note is in there because I walked into it. Run git mv Utils.ts utils.ts on macOS and the filesystem treats both as the same file, so the command succeeds and changes nothing at all. You need the temp-name round trip. That cost me half an hour of confused staring.

Layer 3: a tolerant resolver on the agent side

Existing NFD files still live in the repo. Renaming years of assets in one sweep isn't worth the broken-link risk.

So the tooling the agent calls gets a forgiving resolver:

import unicodedata
from pathlib import Path
 
 
def resolve_tolerant(root: Path, wanted: str) -> Path | None:
    """Resolve a requested path, absorbing normalization and case.
 
    Exact match wins. A loose match is accepted only when it is unique.
    Multiple candidates mean a ghost duplicate, so we refuse to guess.
    """
    exact = root / wanted
    if exact.exists():
        return exact
 
    def key(s: str) -> str:
        return unicodedata.normalize("NFC", s).casefold()
 
    target = key(wanted)
    matches = [p for p in root.rglob("*") if key(str(p.relative_to(root))) == target]
 
    if len(matches) == 1:
        return matches[0]
    if len(matches) > 1:
        raise FileExistsError(
            f"{len(matches)} real files resolve to the same name: {matches}"
        )
    return None

The design decision that matters here: never quietly pick when it's ambiguous.

Multiple candidates raise. Returning one of them would let the run continue, and continuing might mean overwriting years of notes with an empty stub. Stopping to fetch a human is orders of magnitude cheaper. The more unattended a process is, the more it should fail toward stopping.

Six Weeks of Numbers

Six weeks, four repos, gate running on everything.

The three ghost duplicate pairs were the surprise. One had coexisted for four months. One file held years of notes on color palettes for my wallpaper apps; the other was a 12-line fragment an agent produced overnight.

I had read that fragment. More than once. I remember thinking it seemed thinner than I recalled, and reading on anyway, assuming it was the real thing. The real thing was sitting next to it, under the same name, quietly ignored.

The chilling part isn't the mistake. It's that four months passed and nothing surfaced it. No error. Green CI. Just an asset drifting out of reach.

That 0.4 second runtime also made the decision easy. Nobody objects to a gate that fast. A ten-second gate gets suspended on a deadline day; a 0.4-second gate gives nobody an excuse. Trimming to changed paths only makes it roughly 3x faster still. If you want a gate to stick, speed is a feature.

Where I Stop Leaning on Normalization

One last thing, about limits.

After writing layer three, I felt the pull to make it clever. Fold katakana against hiragana too? Collapse fullwidth and halfwidth?

I didn't, for two reasons.

First, tolerant resolution hides the problem. The smarter the resolver, the less anyone suffers from a dirty repo — so it gets dirtier, until it bites somewhere outside the resolver: another person's tool, Git itself, a shell script in CI. I can rescue my agent. I cannot rescue grep.

Second, folding rules depend on language and context. Turkish folds I differently than English does. Working as an indie developer against Japanese only, you won't notice — but the moment you hand-roll the rules, you own a private standard forever.

So my conclusion: put your weight on layer one, and keep layer three as a minimal bridge for legacy assets. Don't widen the bridge. Take it down once you've crossed.

When I do rename, I work in this order:

1. Run filename_gate.py and list every finding
2. Clear CASE and CHAR first (normalization won't help them, and their blast radius is harder to read)
3. For NFD, grep out every referrer before renaming (git mv in two steps)
4. Check whether each renamed path needs a redirect (mandatory if it was ever a public URL)
5. Once fully clean, make the CI gate required

Invert that order and CI goes red mid-migration, and people get comfortable with red. Making the gate required after reaching clean worked much better.

What to Do Next

Run filename_gate.py against whatever repo is open right now. It takes seconds.

If it prints ✅ clean, that isn't luck — it's the accumulated judgment of everyone who avoided Japanese filenames before you. If something does come out, it's worth running ls | xxd on the results to see whether a ghost is hiding among them.

What you can see is not necessarily what is there. The more you delegate to agents, the more quietly that assumption erodes.

For adjacent ground: fixing garbled Japanese in the Antigravity terminal covers the display side, empty placeholder files in cloud-synced workspaces covers the storage side, and arbitrating shared file writes covers what happens when several agents reach for the same path at once.