Before handing a link clean-up to an agent, I wanted a count of the body links in my repository. As an indie developer, I keep a website of 2,238 MDX files across Japanese and English.
The first 30-line script I wrote said that 594 of the 1,119 Japanese articles were never linked from another article's body. That is more than half.
I was about to plan an all-night session of adding internal links. What stopped me was remembering that the pages show a related-articles box.
Counting differently turned 594 into 6
Counting only the article links in the body means looking at half of what a reader sees. Each article page lists three related articles under the text.
The logic lives in RelatedArticles.tsx. It adds 3 points for the same category and 2 points per shared tag, then shows the top three.
I copied that scoring into Python and added it to the body-link count.
| How I counted | Japanese | English |
|---|---|---|
| Zero body-link entrances | 594 | 868 |
| Still zero with the related box (searchable pages only) | 6 | 11 |
Most of the 594 "orphans" were not orphans on screen. Only 6 need attention.
The full script
It runs on the standard library alone. From the repository root, run it like python3 link_audit.py . ja.
Two details matter. Link examples inside code fences are not counted, because an article that explains link syntax would otherwise count its own examples as real links.
The related-articles replica also pushes pages kept out of search to the back of the candidates, because the component does the same. Leaving that out changed the numbers a lot.
#!/usr/bin/env python3
"""Internal-link audit for an MDX site: body links, the related-articles widget, and links the build strips.
usage: link_audit.py <repo> <ja|en> [--suggest N]"""
import re, sys, collections, pathlib
repo, lang = pathlib.Path(sys.argv[1]), sys.argv[2]
CAP = int(sys.argv[sys.argv.index("--suggest") + 1]) if "--suggest" in sys.argv else 0
PREFIX = "/articles/" if lang == "ja" else "/en/articles/"
LINK = re.compile(r'\]\((/(?:en/)?articles/[^)#\s?]+)|href="(/(?:en/)?articles/[^"#\s?]+)')
FENCE = re.compile(r"```.*?```", re.S)
def tag_block(front): # tags: appears both as [..] and as a dashed list on following lines
t = re.search(r"^tags:(.*?)(?=^\S|\Z)", front, re.S | re.M)
return t.group(1) if t else ""
def load(lang):
arts = {}
for p in (repo / "content/articles" / lang).rglob("*.mdx"):
m = re.match(r"^---\n(.*?)\n---\n", p.read_text(encoding="utf-8"), re.S)
if not m:
continue
fm = dict(l.split(":", 1) for l in m.group(1).splitlines() if ":" in l and l[0] != " ")
body = p.read_text(encoding="utf-8")[m.end():]
arts[f"{p.parent.name}/{p.stem}"] = {
"cat": p.parent.name,
"date": fm.get("date", "").strip().strip('"'),
"hidden": fm.get("noindex", "").strip() == "true", # flag for pages kept out of search
"tags": set(re.findall(r'"([^"]+)"', tag_block(m.group(1)))),
"body": FENCE.sub("", body), # do not count link examples inside code fences
}
return arts
arts = load(lang)
newest_first = sorted(arts, key=lambda k: arts[k]["date"], reverse=True)
body_in, dead = collections.Counter(), []
out_n = collections.Counter()
for k in arts:
for m in LINK.finditer(arts[k]["body"]):
u = m.group(1) or m.group(2)
if not u.startswith(PREFIX):
continue
t = u[len(PREFIX):].strip("/")
if t == k:
continue
if t in arts:
body_in[t] += 1
out_n[k] += 1
else:
dead.append((k, t)) # links the build silently removes
# Replica of the related-articles widget: +3 for the same category, +2 per shared tag, top 3.
# Ties keep the newest_first order (sorted() is stable).
rel_in = collections.Counter()
for k in arts:
a = arts[k]
scored = [(j, (3 if arts[j]["cat"] == a["cat"] else 0) + 2 * len(arts[j]["tags"] & a["tags"]))
for j in newest_first if j != k]
# Hidden pages always go last, not just lower on ties. This mirrors the component.
top = sorted((s for s in scored if s[1] > 0), key=lambda s: (arts[s[0]]["hidden"], -s[1]))[:3]
for j, _ in top:
rel_in[j] += 1
n = len(arts)
orph_body = [k for k in arts if body_in[k] == 0]
orph_all = [k for k in orph_body if rel_in[k] == 0 and not arts[k]["hidden"]] # only pages that are meant to be found
edges = sum(rel_in.values())
recent = [k for k in arts if arts[k]["date"] >= sorted(a["date"] for a in arts.values())[-79]]
print(f"{lang}: articles {n} / zero body-link entrances {len(orph_body)} / still zero with the widget (searchable only) {len(orph_all)}")
print(f"widget links {edges}; share going to the newest {len(recent)} articles: {sum(rel_in[k] for k in recent) / edges:.1%}"
f" (articles receiving any: {len(rel_in)})")
print(f"links stripped at build: {len(dead)} ({len({d[0] for d in dead})} files)")
if CAP: # For each orphan, propose a same-category article with overlapping tags as the link source (at most CAP per source)
load_ = collections.Counter(); plan = {}
for o in sorted(orph_all, key=lambda k: arts[k]["date"], reverse=True):
cands = sorted((s for s in arts if s != o and arts[s]["cat"] == arts[o]["cat"] and load_[s] < CAP),
key=lambda s: (-len(arts[s]["tags"] & arts[o]["tags"]), load_[s], s))
if cands and arts[o]["tags"] & arts[cands[0]]["tags"]:
plan[o] = cands[0]; load_[cands[0]] += 1
print(f"proposal: entrances for {len(plan)}/{len(orph_all)} orphans (sources: {len(load_)}, at most {CAP} each)")
for o, s in list(plan.items())[:3]:
print(" ", s, "->", o)
if "--fail-over" in sys.argv: # for CI: exit 1 when orphans exceed the baseline
limit = int(sys.argv[sys.argv.index("--fail-over") + 1])
sys.exit(1 if len(orph_all) > limit else 0)sorted is stable, so ties keep the newest_first order, which means newer articles survive on a tie. That property drives the result in the next section.