I found a console.log in the production bundle by opening the browser console after a release.
My AGENTS.md says not to leave console.log in production code. I wrote that line. One file, no monorepo, 47 lines sitting at the repository root. There was no precedence conflict to blame.
The agent had read the file and ignored it anyway.
My first instinct was to blame the model's mood. But once you blame the mood, there is nothing left to do about it. So I stopped judging compliance from memory and started counting it.
What the counting showed is that the rules being ignored had a pattern.
Running on "it feels like it's mostly following"
My workflow up to that point looked like this. Hand a task to the agent. Read the diff that comes back. Point out what bothers me. On days with a lot of pointing, conclude the model is having an off day.
The problem with that loop is that it has no denominator.
Of ten rules, how many held? Is this week better or worse than last? Which of the 47 lines is actually doing work? I had none of those answers, and I kept adding lines anyway. Every time I caught a violation, I rewrote the rule in stronger language with more detail. The file went from 12 lines in March to 47 by July.
And adherence, as far as I could tell, was getting worse. "As far as I could tell" was the whole problem.
There are two kinds of rules
The first wall I hit was that a machine cannot answer "was this rule followed?" for most of what I had written.
"Functions must be at most 30 lines, split them if longer" is countable. "Use naming that conveys intent" is not. The first collapses into a predicate. The second does not.
That distinction turned out to matter more than I expected. I started with only the ones that collapse. Of 47 lines, exactly 19 became checkable predicates. The rest were paraphrases of "write good code."
More than half of my file could not be verified by anything. At that point I started to wonder what I had actually been writing.
Turning rules into predicates
I rewrote each of the 19 rules as a function that takes a file and returns violations. Nothing clever — read content, return where it breaks.
# tools/adherence/rules.py
# One check function per rule in AGENTS.md.
# If a rule can't be expressed here, it doesn't go in AGENTS.md either.
import re
from dataclasses import dataclass
from typing import Callable
@dataclass
class Violation:
rule_id: str
file: str
line: int
excerpt: str
@dataclass
class Rule:
id: str # the same ID appears in AGENTS.md, so the mapping is explicit
description: str
applies_to: str # regex over the file path
check: Callable[[str, str], list]
def no_console_log(path: str, content: str) -> list:
out = []
for i, line in enumerate(content.splitlines(), start=1):
# Restrict to real calls so prose inside comments doesn't get flagged.
if re.search(r'(?<!//\s)\bconsole\.(log|debug)\s*\(', line):
out.append(Violation("R-001", path, i, line.strip()))
return out
def function_length_limit(path: str, content: str, limit: int = 30) -> list:
out = []
lines = content.splitlines()
start, name, depth = None, None, 0
for i, line in enumerate(lines, start=1):
m = re.match(r'\s*(?:export\s+)?(?:async\s+)?function\s+(\w+)', line)
if m and start is None:
start, name, depth = i, m.group(1), 0
if start is not None:
depth += line.count("{") - line.count("}")
if depth <= 0 and i > start:
if (i - start) > limit:
out.append(Violation("R-004", path, start,
f"{name}: {i - start} lines (limit {limit})"))
start, name = None, None
return out
def no_any_type(path: str, content: str) -> list:
out = []
for i, line in enumerate(content.splitlines(), start=1):
# Deliberate `any` must carry an `// allow-any:` reason comment.
if re.search(r':\s*any\b', line) and "allow-any:" not in line:
out.append(Violation("R-002", path, i, line.strip()))
return out
RULES = [
Rule("R-001", "no console.log in production code",
r"^src/(?!.*\.test\.).*\.(ts|tsx)$", no_console_log),
Rule("R-002", "no any type",
r"^src/.*\.(ts|tsx)$", no_any_type),
Rule("R-004", "functions at most 30 lines",
r"^src/.*\.(ts|tsx)$", function_length_limit),
# ... 19 of these
]Writing no_console_log forced me to add a caveat about comment lines.
That looks like an implementation detail, but it was the real lesson. I had never once defined my own rules precisely. Does "don't leave console.log" cover test files? Commented-out calls? I had never decided. A rule I never pinned down cannot be interpreted the way I meant it.
Decomposing rules into predicates was, by itself, an audit of the rules.
Scoring against the diff
With the checks in place, I run them after the agent finishes. Pull changed files from git, apply only the rules whose path pattern matches.
# tools/adherence/measure.py
import json
import re
import subprocess
from datetime import datetime, timezone
from rules import RULES
def changed_files(base: str = "HEAD~1") -> list:
out = subprocess.run(
["git", "diff", "--name-only", "--diff-filter=d", base, "HEAD"],
capture_output=True, text=True, check=True,
)
return [p for p in out.stdout.splitlines() if p]
def read_at_head(path: str) -> str:
out = subprocess.run(["git", "show", f"HEAD:{path}"],
capture_output=True, text=True)
return out.stdout if out.returncode == 0 else ""
def measure(base: str = "HEAD~1") -> dict:
files = changed_files(base)
per_rule = {r.id: {"applicable": 0, "violated": 0,
"description": r.description} for r in RULES}
violations = []
for path in files:
content = read_at_head(path)
if not content:
continue
for rule in RULES:
if not re.match(rule.applies_to, path):
continue
# Denominator is "times the rule was in scope".
# Counting out-of-scope files flatters the score.
per_rule[rule.id]["applicable"] += 1
found = rule.check(path, content)
if found:
per_rule[rule.id]["violated"] += 1
violations.extend(found)
for rid, s in per_rule.items():
s["adherence"] = (
None if s["applicable"] == 0
else round(1 - s["violated"] / s["applicable"], 3)
)
scored = [s for s in per_rule.values() if s["adherence"] is not None]
overall = (round(sum(s["adherence"] for s in scored) / len(scored), 3)
if scored else None)
return {
"measured_at": datetime.now(timezone.utc).isoformat(),
"commit": subprocess.run(["git", "rev-parse", "HEAD"],
capture_output=True, text=True).stdout.strip(),
"files_changed": len(files),
"overall_adherence": overall,
"per_rule": per_rule,
"violations": [v.__dict__ for v in violations],
}
if __name__ == "__main__":
result = measure()
with open("adherence.jsonl", "a") as f:
f.write(json.dumps(result, ensure_ascii=False) + "\n")
print(f"overall={result['overall_adherence']} "
f"violations={len(result['violations'])}")The denominator took a couple of tries to get right.
My first version divided by total changed files, which made any day spent editing .md files look excellent. Counting only the times a rule was actually in scope produced numbers that survive day-to-day comparison.
Each commit appends one JSON line, so the trend is there when I want it.