The task was small: add one new entry to the category list my wallpaper app serves. The agent's report came back neat and confident: it had updated the list, validated the JSON, committed the change, and cleared the cache. Four things, all done.
Then I opened the app on my phone, and the new category wasn't there. The file had been edited, yes. But nothing was committed, and the endpoint was still serving the old response. Of the four things in the report, two had actually happened.
A little later I read a gemini-cli issue describing an agent in Antigravity that kept reporting, over nine days, that a build task was complete and had passed its checks, when it hadn't. The scale is nothing like mine, but the shape is the same. An agent's report isn't the result of the work. It's a piece of writing about the work.
Ask for claims, not a story
At first I read the reports and spot-checked whatever looked suspicious. That didn't go well. The better written the report, the less I felt like checking it.
So I changed the shape of the report itself. When the agent finishes, it writes a claims.json: one line per claim, and every claim carries a hint for how to check it.
{
"task": "Add the Hiroshige category to the list, update the served JSON, purge the cache",
"claims": [
{"id": "c1", "says": "Added ukiyoe-hiroshige to the list", "kind": "file_contains",
"path": "api/categories.json", "pattern": "\"ukiyoe-hiroshige\""},
{"id": "c2", "says": "The JSON is still valid", "kind": "command",
"run": ["python3", "-m", "json.tool", "api/categories.json"], "expect_exit": 0},
{"id": "c3", "says": "Committed the change", "kind": "git_committed", "path": "api/categories.json"},
{"id": "c4", "says": "Live endpoint updated, cache cleared", "kind": "http",
"url": "https://example.com/api/categories.json", "expect_contains": "ukiyoe-hiroshige"},
{"id": "c5", "says": "Purged", "kind": "purged"}
]
}says is for me to read; everything from kind down is for a machine to check. The important part is c5: claims the agent has no way to back up still get written down. A claim that isn't written can slip past you. A claim that is written can be counted.
In my workspace rules (Customizations → Rules) I keep something like this:
When you finish, write claims.json in addition to your written report.
- Every piece of work you say you did becomes one claim
- kind is one of file_contains / git_committed / command / http
- If a piece of work fits none of these, write any kind you like (don't hide it)
- Do not run the verifier yourself after writing claims.jsonThat last line is deliberate. If the same agent runs the check, it may "fix" the check when the check fails.
A separate script that checks each claim
What this solves: it replaces "read the report and feel reassured" with "look at PASS or FAIL for each claim." Standard library only; I've run it on Python 3.10.
#!/usr/bin/env python3
"""Check an agent's claims.json, one claim at a time, independently of the agent.
Usage: python3 verify_claims.py claims.json [--repo .]
Exit code: 0 if everything passes, 1 if anything is FAIL or UNVERIFIABLE
"""
import json, re, subprocess, sys, time, urllib.request, urllib.error
from pathlib import Path
TIMEOUT = 20 # 1件あたりの上限秒数(HTTP・コマンド共通)
def check_file_contains(c, repo):
p = repo / c["path"]
if not p.is_file():
return "FAIL", f"{c['path']} がありません"
text = p.read_text(encoding="utf-8", errors="replace")
if re.search(c["pattern"], text):
return "PASS", "パターンが見つかりました"
return "FAIL", f"パターン {c['pattern']!r} が見つかりません"
def check_git_committed(c, repo):
# 「コミットしました」は、作業ツリーに差分が残っていないことと、HEAD に含まれることの二つで見る
dirty = subprocess.run(["git", "status", "--porcelain", "--", c["path"]],
cwd=repo, capture_output=True, text=True).stdout.strip()
if dirty:
return "FAIL", f"未コミットの変更が残っています: {dirty}"
tracked = subprocess.run(["git", "ls-files", "--error-unmatch", c["path"]],
cwd=repo, capture_output=True, text=True)
if tracked.returncode != 0:
return "FAIL", "HEAD に含まれていません"
return "PASS", "HEAD に含まれ、差分もありません"
def check_command(c, repo):
# shell=True は使わない。エージェントが書いた文字列をそのままシェルに渡さないため
try:
r = subprocess.run(c["run"], cwd=repo, capture_output=True, text=True, timeout=TIMEOUT)
except FileNotFoundError:
return "FAIL", f"コマンドが見つかりません: {c['run'][0]}"
except subprocess.TimeoutExpired:
return "FAIL", f"{TIMEOUT} 秒で終わりませんでした"
want = c.get("expect_exit", 0)
if r.returncode == want:
return "PASS", f"exit {r.returncode}"
tail = (r.stderr or r.stdout).strip().splitlines()[-1:] or [""]
return "FAIL", f"exit {r.returncode}(期待 {want}): {tail[0][:120]}"
def check_http(c, repo):
# キャッシュに古い応答が残っていると「反映済み」に見えてしまうので、毎回クエリを変える
sep = "&" if "?" in c["url"] else "?"
url = f"{c['url']}{sep}_verify={int(time.time())}"
req = urllib.request.Request(url, headers={"Cache-Control": "no-cache"})
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as res:
status, body = res.status, res.read().decode("utf-8", errors="replace")
hint = res.headers.get("cf-cache-status") or res.headers.get("age") or "-"
except urllib.error.HTTPError as e:
status, body, hint = e.code, "", "-"
except (urllib.error.URLError, TimeoutError) as e:
return "FAIL", f"接続できません: {e}"
if status != c.get("expect_status", 200):
return "FAIL", f"HTTP {status}(期待 {c.get('expect_status', 200)})"
if "expect_contains" in c and c["expect_contains"] not in body:
return "FAIL", f"HTTP {status} だが本文に {c['expect_contains']!r} がありません(cache: {hint})"
return "PASS", f"HTTP {status}(cache: {hint})"
CHECKS = {"file_contains": check_file_contains, "git_committed": check_git_committed,
"command": check_command, "http": check_http}
def main():
args = sys.argv[1:]
if not args:
sys.exit(__doc__)
repo = Path(args[args.index("--repo") + 1]) if "--repo" in args else Path(".")
data = json.loads(Path(args[0]).read_text(encoding="utf-8"))
bad = 0
for c in data.get("claims", []):
fn = CHECKS.get(c.get("kind"))
if fn is None:
# 確かめる手段がない主張は「済み」に数えない
state, note = "UNVERIFIABLE", f"kind={c.get('kind')!r} は確かめられません"
else:
try:
state, note = fn(c, repo)
except KeyError as e:
state, note = "UNVERIFIABLE", f"項目 {e} が足りません"
bad += state != "PASS"
print(f"{state:<13} {c.get('id', '?'):<4} {c.get('says', '')[:40]:<40} {note}")
print(f"\n{len(data.get('claims', []))} 件中 {bad} 件が未確認です" if bad else "\nすべて確認できました")
sys.exit(1 if bad else 0)
if __name__ == "__main__":
main()The messages are in Japanese because that's how I run it day to day; swap them for your own wording. Three choices are worth explaining:
- A claim that can't be checked counts as a failure. An unknown
kindbecomes UNVERIFIABLE and the exit code is 1. This is the single line that keeps "probably done" from being counted as "done." - Commands never run through
shell=True. Passing an agent-written string to a shell means a "check" could run something else entirely. That's whyrunis an array. - Every HTTP check changes the query string. If a cache in between is still serving the old response, you'd be judging the update by stale bytes. When
cf-cache-statusoragecomes back, the script prints it.
What it caught when I reproduced the problem
I rebuilt the opening situation in a small local repo: file edited but not committed, endpoint still stale, purge impossible to verify. The run looked like this:
PASS c1 一覧に ukiyoe-hiroshige を足した パターンが見つかりました
PASS c2 JSON として壊れていない exit 0
FAIL c3 変更をコミットした 未コミットの変更が残っています: M api/categories.json
FAIL c4 配信側に反映され、キャッシュも消した HTTP 200 だが本文に 'ukiyoe-hiroshige' がありません(cache: -)
UNVERIFIABLE c5 パージ済み kind='purged' は確かめられません
5 件中 3 件が未確認ですThe exit code was 1. After committing, refreshing the endpoint, and removing c5, I ran it again: all four remaining claims passed and the exit code went back to 0.
What stood out was that c4 returned HTTP 200. If I'd only looked at the status, it would have passed. A response existing and a response being current are two separate claims. That gap is exactly what I'd missed when I checked by hand.
| kind | What it confirms | What it can't tell you |
|---|---|---|
| file_contains | The expected text is in the file | Whether the change was saved and shared |
| git_committed | No diff left, file is in HEAD | Whether it was pushed |
| command | The command exits with the expected code | Whether it looked at the right target |
| http | The endpoint returns the expected content | Whether every region and device sees the same |
When you feel the urge to fill in that right-hand column, that's your cue to add a kind. To confirm a push, for example, compare git ls-remote with your local HEAD.
How much to hand to the agent
Since switching to this, I've actually handed the agent more, not less. With CLI v1.5.10 adding accept-edits to --remote-agent, there are more moments where approving edits can be delegated. That's precisely why I want the "is it done?" call to stay somewhere else.
The line I draw now fits in one sentence:
The agent does the work and writes the report; a separate script makes the call.
Not everything fits a kind, of course. "The layout looks natural" or "the copy reads more politely" can't be scripted. I leave those as UNVERIFIABLE and treat them as the list I check with my own eyes. Seeing, right there in the output, which claims no machine has judged turns out to be what cuts down on the misses.
Two related checks I rely on: checking an agent's instructions locally before running them, and statically measuring whether a hook's matcher can ever fire.
One next step
Pick the next task you're about to give an agent, and ask for a claims.json instead of a report. Whether it ends up with three claims or five, if even one shows up as UNVERIFIABLE, the setup has already done its job.
If you later want to go further and check against the Cloudflare or GitHub APIs directly, the operational design in the premium articles may help you decide how far to take it. As an indie developer, I'm still adding one check at a time myself.