Three Byte-Level Checks I Run Before an Agent Edits Files That Contain Japanese

Check 1: invalid UTF-8 sequences

Start with the failure mode that is easiest to pin down: files that do not hold up as byte sequences at all.

# Scan the repository in parallel and print only the invalid files
find content -type f \( -name '*.mdx' -o -name '*.json' -o -name '*.strings' \) -print0 \
  | xargs -0 -P4 -n50 sh -c '
      for f do
        iconv -f UTF-8 -t UTF-8 "$f" >/dev/null 2>&1 || echo "INVALID: $f"
      done' _

Throw away the converted output and read only the exit code. You never wanted the conversion.

Against my own repository (2,064 MDX files) this finished in 667 milliseconds with zero findings. At that scale you can run it on every commit without anyone perceiving a wait.

But this check has one decisive hole.

A file "repaired" by replacement characters passes this check

Agents and tool chains are often written not to crash when they meet a byte they cannot decode. The most common shape is a lenient read that substitutes a replacement character.

# The usual "don't crash" read
raw = open("broken.txt", "rb").read()
s = raw.decode("utf-8", errors="replace")   # invalid bytes -> U+FFFD
open("healed.txt", "w", encoding="utf-8").write(s)

Inspect healed.txt afterwards and you get this:

healed.txt  decode=OK  (the check passes)
content     : const title = "�奈川沖浪裏";
U+FFFD count: 1
iconv exit=0

iconv returns 0. Python's strict decode succeeds. At the byte level the file is once again perfectly valid UTF-8. The only thing that has not come back is the content.

Line up the bytes and the substitution is obvious:

after corruption: 2074 6974 6c65 203d 2022 e7a5 e5a5 88e5   title = "......
after replacement: 2074 6974 6c65 203d 2022 efbf bde5 a588   title = "......

The remnant e7a5 has been replaced by efbfbd, the three bytes of U+FFFD. There is no way to recover the original character from that.

Learning this changed my mind about one thing specifically: where the check belongs. Put a validity check behind your tool chain and the evidence of corruption can disappear before the check runs. If it has to live behind, it needs a different signal.

Check 2: count the replacement characters

Practically speaking, this is the only way to recover traces of what a lenient read destroyed. Search for the byte sequence EF BF BD.

# Find files where a replacement character has crept into the content
grep -rlP '\xef\xbf\xbd' content/ 2>/dev/null

One caveat. U+FFFD is a character almost nobody writes deliberately, but it legitimately appears in articles about mojibake itself and in test fixtures used as expected values. I exclude only the directory holding test data.

Keep that exclusion list short. When it starts growing, that is a signal the check itself is aimed at the wrong thing.

Check 3: normalization drift and BOM

The third category covers byte sequences that are entirely valid yet still break how the file is handled.

On macOS, filesystem behavior can leave you with NFD strings where dakuten and handakuten are decomposed. They look identical. The bytes are not.

NFC: e381 b7e3 8289 e381 990a          (ぷ as one character)
NFD: e381 b5e3 829a e382 89e3 8199 0a  (ふ plus a combining mark)

That difference lands directly on search.

$ grep -c "ぷ" nfc.txt
1
$ grep -c "ぷ" nfd.txt
0

You believe you are searching for the same character and get nothing. Ask an agent to "fix every occurrence of this term" and matches will quietly go missing. The equivalent problem on the filename side is covered in when a Japanese filename becomes a different string on macOS and Linux. This one is about file contents.

BOM only requires looking at the first three bytes. JSON parsers and shebang detection both break on a leading BOM, so it is worth checking whenever you handle resource files.

Folding all three into a single pass

Running three separate commands means reading every file three times. At a few thousand files that difference stops being free. I folded all three judgments into one read.

#!/usr/bin/env python3
"""Encoding check for resources containing Japanese, before and after edits.
Exit code: 1 if there is a problem, otherwise 0.
"""
import glob
import sys
import unicodedata
 
PATTERNS = ["content/**/*.mdx", "resources/**/*.json", "ios/**/*.strings"]
SKIP = ("/fixtures/", "/testdata/")   # only where mojibake is intentional
 
def scan():
    invalid, fffd, non_nfc, bom, total = [], [], [], [], 0
    for pattern in PATTERNS:
        for path in glob.glob(pattern, recursive=True):
            if any(s in path for s in SKIP):
                continue
            total += 1
            raw = open(path, "rb").read()
 
            # Check 3b: BOM (valid bytes, so look before decoding)
            if raw[:3] == b"\xef\xbb\xbf":
                bom.append(path)
 
            # Check 1: byte-sequence validity
            try:
                text = raw.decode("utf-8")
            except UnicodeDecodeError as e:
                invalid.append(f"{path} (pos={e.start}, {e.reason})")
                continue   # nothing further is meaningful once decode fails
 
            # Check 2: traces left by a lenient read
            if "�" in text:
                fffd.append(f"{path} (x{text.count(chr(0xfffd))})")
 
            # Check 3a: normalization drift
            if text != unicodedata.normalize("NFC", text):
                non_nfc.append(path)
 
    return total, invalid, fffd, non_nfc, bom
 
total, invalid, fffd, non_nfc, bom = scan()
print(f"scanned={total} invalid_utf8={len(invalid)} "
      f"replacement_char={len(fffd)} non_nfc={len(non_nfc)} bom={len(bom)}")
 
for label, items in (("INVALID UTF-8", invalid), ("U+FFFD", fffd),
                     ("NON-NFC", non_nfc), ("BOM", bom)):
    for item in items:
        print(f"  [{label}] {item}")
 
# Invalid bytes and U+FFFD stop the build; drift and BOM are warnings
sys.exit(1 if (invalid or fffd) else 0)

Running it against 2,064 MDX files:

scanned=2064 invalid_utf8=0 replacement_char=0 non_nfc=0 bom=0

That took 149 milliseconds, faster than the 667 milliseconds of the parallel iconv version. Process startup cost outweighed the cost of one extra read per file.

A note on why only two of the four conditions are fatal. Normalization drift and BOM show up for legitimate reasons, such as third-party data stored verbatim. Failing on those makes the exclusion list grow without end, and an exclusion list that long stops being read by anyone.

Choose the placement by how far you can roll back

There are three plausible places for this, and I have run all of them.

I keep the pre-commit hook as my primary gate, and the reason is rollback distance. Before a commit, git checkout -- <file> restores everything. By the time CI catches it, other edits have often landed on top, and peeling off one broken character becomes real work.

# .git/hooks/pre-commit
#!/bin/sh
python3 tools/check_encoding.py || {
  echo "Encoding check failed. Review the files listed above." >&2
  exit 1
}

Running the same script before a bulk edit is worth the two seconds as well. It settles the question of whether the agent broke the file or the file arrived broken, instantly. Without that distinction you end up with a wide field of suspects, the same way a failed patch application can consume an afternoon.

What to do next

Run one search for U+FFFD across your own repository.

grep -rlP '\xef\xbf\xbd' . --exclude-dir=.git 2>/dev/null

Zero results means nothing has slipped through as of today. A single hit is a record of a character that was lost somewhere in your history, and git log -S will walk you back to the commit that introduced it.

Until I put this check in place, I assumed I would simply see it when Japanese text broke. In a diff, one character is invisible. Things that are invisible are better counted by a machine.