I Edited a Sentence and Only the New Characters Turned into Tofu: Adding a Missing-Glyph Check to a Subsetted Japanese Web Font

After editing the Japanese copy on my own site, I went to look at the page and found that only the characters I had just added were rendering as empty tofu boxes. Everything else was in the usual typeface.

The cause was simple. The site serves a Japanese web font subsetted to the characters it uses, and I had changed the text without rebuilding the subset.

Mistakes like this slip through whenever the only safeguard is a human eye. So I wrote a small check that finds characters missing from the subset and stops if even one is absent. This post walks through it, with output from actually running it.

What was going on

A Japanese font carries thousands of kanji, so serving it whole means several megabytes. The usual remedy is to keep only the characters your pages use. That is subsetting.

The price is that anything not in the file cannot be drawn. The browser looks for a fallback face, and if it finds none, it shows tofu. Every copy change therefore raises one question: are the new characters in the subset?

I first assumed I could simply remember to rebuild. That did not hold up: even when the person editing the copy and the person rebuilding the font are both me, I forget on busy days. I would rather build it so that forgetting is expected and the check stops me.

What you need

I ran everything below in this environment.

There are three files: a shared character scanner, the subset builder, and the checker. The scanner is shared so that the builder and the checker cannot disagree about what counts as a character.

Step 1: collect the characters on the site

The scanner walks the files and yields each character with where it came from (file and line).

#!/usr/bin/env python3
"""サイトの文字を集める共通部品。build と check の両方がこれを読む。"""
import pathlib, re, unicodedata
 
TEXT_EXT = {".html", ".mdx", ".md", ".json", ".txt", ".tsx", ".ts"}
# 描画に字形が要らない文字（改行・タブ・ゼロ幅など）
SKIP_CATEGORIES = {"Cc", "Cf", "Zl", "Zp"}
 
def iter_chars(root: pathlib.Path):
    """(文字, ファイル, 行番号) を順に返す。"""
    for path in sorted(root.rglob("*")):
        if path.suffix not in TEXT_EXT or not path.is_file():
            continue
        text = path.read_text(encoding="utf-8")
        # JSON は \uXXXX のエスケープを元に戻してから走査する
        if path.suffix == ".json":
            text = re.sub(r"\\u([0-9a-fA-F]{4})", lambda m: chr(int(m.group(1), 16)), text)
        for lineno, line in enumerate(text.splitlines(), start=1):
            for ch in line:
                if unicodedata.category(ch) in SKIP_CATEGORIES:
                    continue
                if ch == " ":
                    continue
                yield ch, path, lineno

The line that decodes JSON escapes matters most. A character like 龍 may appear in JSON as \u9f8d, and searching for the literal character will never find it.

Step 2: build a font with only those characters

#!/usr/bin/env python3
"""サイト内の全文字だけを残した woff2 を作る。"""
import pathlib, sys
from fontTools import subset
from fontTools.ttLib import TTCollection
from scan_chars import iter_chars
 
src_ttc, site_dir, out = sys.argv[1], pathlib.Path(sys.argv[2]), sys.argv[3]
BASE = "".join(chr(c) for c in range(0x20, 0x7F))     # ASCII は常に残す
chars = sorted({c for c, _, _ in iter_chars(site_dir)} | set(BASE))
 
# .ttc から JP の1書体だけ取り出す（先頭が JP とは限らないので名前で探す）
font = next(f for f in TTCollection(src_ttc).fonts if f["name"].getDebugName(1).endswith(" JP"))
opts = subset.Options(flavor="woff2", notdef_outline=True)
sub = subset.Subsetter(opts)
sub.populate(text="".join(chars))
sub.subset(font)
subset.save_font(font, out, opts)
pathlib.Path(out).with_suffix(".chars.txt").write_text("".join(chars), encoding="utf-8")
print(f"chars={len(chars)} -> {out} ({pathlib.Path(out).stat().st_size} bytes)")

At first I passed layout_features=["*"] to keep every OpenType feature. With the same 142 characters, that version came to about 80 KB and the default came to about 41 KB. If you use neither vertical writing nor ligatures, the default is enough.

Step 3: list what is missing

#!/usr/bin/env python3
"""サブセット済みフォントに無い文字を、ファイルと行番号つきで出す。1文字でも欠けたら exit 1。"""
import pathlib, sys, unicodedata
from fontTools.ttLib import TTFont
from scan_chars import iter_chars
 
font_path, site_dir = sys.argv[1], pathlib.Path(sys.argv[2])
cmap = TTFont(font_path).getBestCmap()
missing = {}
for ch, path, lineno in iter_chars(site_dir):
    if ord(ch) not in cmap:
        missing.setdefault(ch, []).append(f"{path.relative_to(site_dir)}:{lineno}")
 
for ch, where in sorted(missing.items()):
    name = unicodedata.name(ch, "?")
    print(f"MISSING U+{ord(ch):04X} {ch} {name}  {where[0]}" + (f" (+{len(where)-1})" if len(where) > 1 else ""))
print(f"missing={len(missing)}")
sys.exit(1 if missing else 0)

The checker reads the font file you actually ship, not the list the builder produced. Only the real file catches drift such as a stale file being served or a font placed in the wrong directory.

What happened when I ran it

First I built from a two-line page and ran the check.

$ python3 build_subset.py /usr/share/fonts/opentype/noto/NotoSerifCJK-Bold.ttc site fonts/body.woff2
chars=117 -> fonts/body.woff2 (26016 bytes)
$ python3 check_glyphs.py fonts/body.woff2 site; echo exit=$?
missing=0
exit=0

Then I mimicked the original mistake: one new sentence and a JSON data file, with no rebuild.

$ python3 check_glyphs.py fonts/body.woff2 site; echo exit=$?
MISSING U+96C6 集 CJK UNIFIED IDEOGRAPH-96C6  page.html:3
MISSING U+9F8D 龍 CJK UNIFIED IDEOGRAPH-9F8D  data/cards.json:1
MISSING U+FF5E ～ FULLWIDTH TILDE  page.html:3
missing=26
exit=1

One added line left 26 characters missing, including plain hiragana such as い, う, and く. A subset holds only what is in use right now, so even familiar characters may be absent from a new sentence.

Rebuilding fixes it.

$ python3 build_subset.py /usr/share/fonts/opentype/noto/NotoSerifCJK-Bold.ttc site fonts/body.woff2
chars=143 -> fonts/body.woff2 (41724 bytes)
$ python3 check_glyphs.py fonts/body.woff2 site; echo exit=$?
missing=0
exit=0

Two things that surprised me

The first is the wave dash. The "～" on screen was the fullwidth tilde (U+FF5E), which is a different character from the wave dash "〜" (U+301C) that Japanese text usually uses. They look nearly identical but are subsetted separately. The source font had both (I checked), so the glyph was not missing from the font; it was simply not in my list of characters in use.

The second is that tofu is not limited to new kanji. Of the 26 missing characters in that run, 11 were hiragana. The feeling that hiragana has always been there is no guarantee when you subset.

Run it before every release

The checker exits with 1 when anything is missing, so running it right before publishing keeps a gap from shipping.

python3 build_subset.py "$SRC_TTC" site fonts/body.woff2 && python3 check_glyphs.py fonts/body.woff2 site

Building first and checking second also catches characters the source font does not contain at all, such as variant forms. A rebuild cannot fix those. You either change the character or assign another typeface to that run of text.

You can also split a font into several files with unicode-range. This check assumes a single file, and I did not run it against a split setup.

If you hand copy edits to an agent

An agent asked to revise copy may touch dozens of lines at once. I think it helps to state the finish condition before handing the work over. For example:

At the end of the task, run check_glyphs.py and continue until it exits 0. If characters are reported missing, show me the list before replacing any of them.

I word it that way because I do not want the agent quietly swapping a missing character for another one just to pass. Once passing becomes the goal, the writing itself changes. I also wrote about what to check before delegating work in general in a post on sync-folder exclusion.

A copy change is also a font change. Before you publish, make this one-line check non-negotiable.

As a first step, run check_glyphs.py once against your current copy and the font you serve right now.