When the JSDoc an AI Wrote Quietly Stopped Matching the Code: Field Notes on Measuring Documentation Drift

The detector: pulling the shape out with ts-morph

To read TypeScript syntax accurately I use ts-morph (a wrapper over the TypeScript Compiler API), not regular expressions. Any regex approach to extracting a signature will break on generics, union types, or multi-line parameter lists.

// scripts/doc-drift.ts
// Hash each public function's signature and compare it against a stored manifest to detect drift.
import { Project, Node, FunctionDeclaration } from "ts-morph";
import { createHash } from "node:crypto";
import { readFileSync, writeFileSync, existsSync } from "node:fs";
 
const MANIFEST = "doc-manifest.json";
 
// Reduce the external shape (param names/types, return, thrown types) to a stable string.
function signatureOf(fn: FunctionDeclaration): string {
  const params = fn.getParameters().map((p) => {
    const name = p.getName();
    const type = p.getType().getText(p); // resolved type text
    const optional = p.isOptional() ? "?" : "";
    return `${name}${optional}:${type}`;
  });
  const ret = fn.getReturnType().getText(fn);
 
  // Collect the types explicitly thrown in the body.
  const throwsSet = new Set<string>();
  fn.forEachDescendant((node) => {
    if (Node.isThrowStatement(node)) {
      const expr = node.getExpression();
      throwsSet.add(expr ? expr.getType().getText(node) : "unknown");
    }
  });
  const throws = [...throwsSet].sort().join(",");
 
  // Parameter order is meaningful, so don't sort it. Throws are unordered, so sort them.
  return `(${params.join(";")})=>${ret}|throws:${throws}`;
}
 
// The raw JSDoc text. If this changes, we treat the documentation as "updated."
function docTextOf(fn: FunctionDeclaration): string {
  return fn.getJsDocs().map((d) => d.getInnerText()).join("\n").trim();
}
 
function hash(s: string): string {
  return createHash("sha1").update(s).digest("hex").slice(0, 12);
}
 
const project = new Project({ tsConfigFilePath: "tsconfig.json" });
const manifest: Record<string, { sig: string; doc: string }> =
  existsSync(MANIFEST) ? JSON.parse(readFileSync(MANIFEST, "utf8")) : {};
 
type Finding = { id: string; kind: "drift" | "undocumented" | "new" };
const findings: Finding[] = [];
const nextManifest: typeof manifest = {};
 
for (const sf of project.getSourceFiles("src/**/*.ts")) {
  for (const fn of sf.getFunctions()) {
    if (!fn.isExported() || !fn.getName()) continue; // public functions only
    const id = `${sf.getFilePath().replace(process.cwd(), "")}#${fn.getName()}`;
    const sig = hash(signatureOf(fn));
    const doc = hash(docTextOf(fn));
    const prev = manifest[id];
 
    if (docTextOf(fn) === "") {
      findings.push({ id, kind: "undocumented" });
    } else if (!prev) {
      findings.push({ id, kind: "new" });
    } else if (prev.sig !== sig && prev.doc === doc) {
      // Shape changed but documentation stayed put = drift.
      findings.push({ id, kind: "drift" });
    }
    nextManifest[id] = { sig, doc };
  }
}
 
// Only write the manifest back on --update (run this after you've fixed the docs).
if (process.argv.includes("--update")) {
  writeFileSync(MANIFEST, JSON.stringify(nextManifest, null, 2));
}
 
const drift = findings.filter((f) => f.kind === "drift");
const documented = Object.keys(nextManifest).length;
const rate = documented ? ((drift.length / documented) * 100).toFixed(1) : "0.0";
console.log(`documented public fns: ${documented}`);
console.log(`drift: ${drift.length}  (drift rate: ${rate}%)`);
for (const f of drift) console.log(`  DRIFT  ${f.id}`);
process.exit(drift.length > 0 ? 1 : 0);

The crux is that the signature and the documentation text are hashed separately. By calling something drift only when "the shape changed" and "the documentation stayed put" hold at the same time, you avoid false positives from ordinary implementation changes or intentional doc rewrites.

Using getType().getText() for the resolved type text matters because it catches cases where a type alias changed underneath. Add 'c' to type Status = 'a' | 'b' and the signature hash of every function taking that type changes too, surfacing the drift.

Watch the drift rate over time

The drift rate from a single run is only a snapshot. What pays off operationally is tracking that number over time.

A spike right after adoption is normal. Divergences that had been sitting there for months all become visible at once — this is a debt inventory. What matters is that the rate falls and settles after the gate goes in. If it keeps climbing week over week, that's a signal that what's taking root isn't a culture of fixing docs but a culture of looking past drift.

The CI gate: stop only the new drift

Once you can measure the drift rate, the next step is stopping new drift in CI. But turning every build red over pre-existing debt gets nothing fixed. The only thing worth stopping is "drift newly introduced by this PR."

#!/usr/bin/env bash
# ci/check-doc-drift.sh
set -euo pipefail
 
npx tsx scripts/doc-drift.ts > /tmp/drift.txt || true
DRIFT_COUNT=$(grep -c '^  DRIFT' /tmp/drift.txt || true)
 
echo "current drift findings: ${DRIFT_COUNT}"
 
# Baseline drift count (accepted debt)
BASE=$(cat .doc-drift-baseline 2>/dev/null || echo 0)
 
if [ "${DRIFT_COUNT}" -gt "${BASE}" ]; then
  echo "New documentation drift: ${DRIFT_COUNT} (allowed: ${BASE})"
  grep '^  DRIFT' /tmp/drift.txt
  echo "Update the drifted functions' JSDoc, then run doc-drift.ts --update"
  exit 1
fi
echo "No new drift"

The baseline approach makes debt repayment incremental. Lower .doc-drift-baseline gradually and you put a deadline on the existing divergences while stopping new ones immediately. You never have to fix everything at once.

Let Antigravity fix only the drifted blocks

Here, finally, AI generation comes back into the picture — but, as I said at the top, without regenerating whole files. The list of functions the detector emitted feeds straight into the instruction to Antigravity.

The following functions changed signature but their JSDoc is stale.
For each, update only @param / @returns / @throws to match the current implementation.

Constraints:
- Preserve the existing prose (the summary description); only fix the shape descriptions
- Check whether each @example works with the current argument shape; fix only the ones that don't
- Do not touch any function not in the list

Targets:
  src/lib/orders.ts#fetchUserOrders
  src/lib/billing.ts#reconcileInvoice

The constraint "preserve the prose, fix only the shape descriptions" is what keeps the diff small. Told this way, the AI doesn't rewrite the whole block; it pinpoints only the lines that actually diverged — the @param type, the @throws entry. The diff a reviewer sees stays a few lines, and "what actually changed" is obvious at a glance.

When the fixes are in, run doc-drift.ts --update to pin the manifest to the current state and reset the baseline for next time.

READMEs and CHANGELOGs break differently

Function JSDoc can be measured for staleness by its signature, but READMEs and CHANGELOGs break in another way. Here you measure whether "the thing the prose references still exists."

The parts of a README that most often go stale are the setup commands, the environment-variable keys, and the API endpoint paths. Those can be reconciled against reality.

// The gist of scripts/readme-drift.ts
// Check that ENV keys mentioned in the README exist in .env.example.
import { readFileSync } from "node:fs";
 
const readme = readFileSync("README.md", "utf8");
const envExample = readFileSync(".env.example", "utf8");
const declaredKeys = new Set(
  envExample.split("\n").map((l) => l.split("=")[0].trim()).filter(Boolean)
);
 
// Pull upper-snake identifiers the README wraps in backticks as env-var candidates.
const mentioned = [...readme.matchAll(/`([A-Z][A-Z0-9_]{3,})`/g)].map((m) => m[1]);
const stale = mentioned.filter((k) => k.endsWith("_KEY") || k.endsWith("_URL") || k.endsWith("_ID"))
  .filter((k) => !declaredKeys.has(k));
 
if (stale.length) {
  console.log("Env vars the README references but .env.example lacks:", stale.join(", "));
  process.exit(1);
}

This reconciliation catches, for instance, renaming an env var from STRIPE_SECRET to STRIPE_SECRET_KEY while the README steps keep the old name. It was consistent when the AI generated the README; subsequent code changes drifted it silently. For CHANGELOGs, the same idea applies: check that referenced version tags exist, and that mentioned commands are present in package.json scripts.

Looking back

Getting an AI to write documentation is no longer the hard part. The hard part is guaranteeing that documentation, once written, stays correct. And because generation got easy, the volume of documentation grows — and so does the total amount of staleness.

That's why I moved the center of gravity from "generate it cleanly" toward "measure the divergence." Catch shape changes with a signature hash, watch overall health through the drift rate, stop new divergence with a gate, and let the AI fix only the blocks that drifted. When those four fit together, documentation goes back to being something you can trust.

If you want a place to start, run doc-drift.ts once against your own project and look at the first drift rate. That number is the total amount of confidently-stale documentation your codebase is carrying. I hope it helps in your own work.