◉ANTIGRAVITY LABJP
Articles/Integrations
⬡ Integrations/2026-11-19Intermediate

When an Agent Says the Integration Is Done: Checking a Claims List With a Small Verifier

An agent can report that an integration task is finished while the real files, commits, and served responses lag behind. Here is how I have the agent write its report as a claims.json, then check each claim with a separate script: files, commits, commands, and live endpoints, with working code.

Antigravity419Agents27VerificationIntegrationsPython27

The task was small: add one new entry to the category list my wallpaper app serves. The agent's report came back neat and confident: it had updated the list, validated the JSON, committed the change, and cleared the cache. Four things, all done.

Then I opened the app on my phone, and the new category wasn't there. The file had been edited, yes. But nothing was committed, and the endpoint was still serving the old response. Of the four things in the report, two had actually happened.

A little later I read a gemini-cli issue describing an agent in Antigravity that kept reporting, over nine days, that a build task was complete and had passed its checks, when it hadn't. The scale is nothing like mine, but the shape is the same. An agent's report isn't the result of the work. It's a piece of writing about the work.

Ask for claims, not a story

At first I read the reports and spot-checked whatever looked suspicious. That didn't go well. The better written the report, the less I felt like checking it.

So I changed the shape of the report itself. When the agent finishes, it writes a claims.json: one line per claim, and every claim carries a hint for how to check it.

{
  "task": "Add the Hiroshige category to the list, update the served JSON, purge the cache",
  "claims": [
    {"id": "c1", "says": "Added ukiyoe-hiroshige to the list", "kind": "file_contains",
     "path": "api/categories.json", "pattern": "\"ukiyoe-hiroshige\""},
    {"id": "c2", "says": "The JSON is still valid", "kind": "command",
     "run": ["python3", "-m", "json.tool", "api/categories.json"], "expect_exit": 0},
    {"id": "c3", "says": "Committed the change", "kind": "git_committed", "path": "api/categories.json"},
    {"id": "c4", "says": "Live endpoint updated, cache cleared", "kind": "http",
     "url": "https://example.com/api/categories.json", "expect_contains": "ukiyoe-hiroshige"},
    {"id": "c5", "says": "Purged", "kind": "purged"}
  ]
}

says is for me to read; everything from kind down is for a machine to check. The important part is c5: claims the agent has no way to back up still get written down. A claim that isn't written can slip past you. A claim that is written can be counted.

In my workspace rules (Customizations → Rules) I keep something like this:

When you finish, write claims.json in addition to your written report.
- Every piece of work you say you did becomes one claim
- kind is one of file_contains / git_committed / command / http
- If a piece of work fits none of these, write any kind you like (don't hide it)
- Do not run the verifier yourself after writing claims.json

That last line is deliberate. If the same agent runs the check, it may "fix" the check when the check fails.

A separate script that checks each claim

What this solves: it replaces "read the report and feel reassured" with "look at PASS or FAIL for each claim." Standard library only; I've run it on Python 3.10.

#!/usr/bin/env python3
"""Check an agent's claims.json, one claim at a time, independently of the agent.
Usage: python3 verify_claims.py claims.json [--repo .]
Exit code: 0 if everything passes, 1 if anything is FAIL or UNVERIFIABLE
"""
import json, re, subprocess, sys, time, urllib.request, urllib.error
from pathlib import Path
 
TIMEOUT = 20  # 1件あたりの上限秒数(HTTP・コマンド共通)
 
 
def check_file_contains(c, repo):
    p = repo / c["path"]
    if not p.is_file():
        return "FAIL", f"{c['path']} がありません"
    text = p.read_text(encoding="utf-8", errors="replace")
    if re.search(c["pattern"], text):
        return "PASS", "パターンが見つかりました"
    return "FAIL", f"パターン {c['pattern']!r} が見つかりません"
 
 
def check_git_committed(c, repo):
    # 「コミットしました」は、作業ツリーに差分が残っていないことと、HEAD に含まれることの二つで見る
    dirty = subprocess.run(["git", "status", "--porcelain", "--", c["path"]],
                           cwd=repo, capture_output=True, text=True).stdout.strip()
    if dirty:
        return "FAIL", f"未コミットの変更が残っています: {dirty}"
    tracked = subprocess.run(["git", "ls-files", "--error-unmatch", c["path"]],
                             cwd=repo, capture_output=True, text=True)
    if tracked.returncode != 0:
        return "FAIL", "HEAD に含まれていません"
    return "PASS", "HEAD に含まれ、差分もありません"
 
 
def check_command(c, repo):
    # shell=True は使わない。エージェントが書いた文字列をそのままシェルに渡さないため
    try:
        r = subprocess.run(c["run"], cwd=repo, capture_output=True, text=True, timeout=TIMEOUT)
    except FileNotFoundError:
        return "FAIL", f"コマンドが見つかりません: {c['run'][0]}"
    except subprocess.TimeoutExpired:
        return "FAIL", f"{TIMEOUT} 秒で終わりませんでした"
    want = c.get("expect_exit", 0)
    if r.returncode == want:
        return "PASS", f"exit {r.returncode}"
    tail = (r.stderr or r.stdout).strip().splitlines()[-1:] or [""]
    return "FAIL", f"exit {r.returncode}(期待 {want}): {tail[0][:120]}"
 
 
def check_http(c, repo):
    # キャッシュに古い応答が残っていると「反映済み」に見えてしまうので、毎回クエリを変える
    sep = "&" if "?" in c["url"] else "?"
    url = f"{c['url']}{sep}_verify={int(time.time())}"
    req = urllib.request.Request(url, headers={"Cache-Control": "no-cache"})
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT) as res:
            status, body = res.status, res.read().decode("utf-8", errors="replace")
            hint = res.headers.get("cf-cache-status") or res.headers.get("age") or "-"
    except urllib.error.HTTPError as e:
        status, body, hint = e.code, "", "-"
    except (urllib.error.URLError, TimeoutError) as e:
        return "FAIL", f"接続できません: {e}"
    if status != c.get("expect_status", 200):
        return "FAIL", f"HTTP {status}(期待 {c.get('expect_status', 200)})"
    if "expect_contains" in c and c["expect_contains"] not in body:
        return "FAIL", f"HTTP {status} だが本文に {c['expect_contains']!r} がありません(cache: {hint})"
    return "PASS", f"HTTP {status}(cache: {hint})"
 
 
CHECKS = {"file_contains": check_file_contains, "git_committed": check_git_committed,
          "command": check_command, "http": check_http}
 
 
def main():
    args = sys.argv[1:]
    if not args:
        sys.exit(__doc__)
    repo = Path(args[args.index("--repo") + 1]) if "--repo" in args else Path(".")
    data = json.loads(Path(args[0]).read_text(encoding="utf-8"))
    bad = 0
    for c in data.get("claims", []):
        fn = CHECKS.get(c.get("kind"))
        if fn is None:
            # 確かめる手段がない主張は「済み」に数えない
            state, note = "UNVERIFIABLE", f"kind={c.get('kind')!r} は確かめられません"
        else:
            try:
                state, note = fn(c, repo)
            except KeyError as e:
                state, note = "UNVERIFIABLE", f"項目 {e} が足りません"
        bad += state != "PASS"
        print(f"{state:<13} {c.get('id', '?'):<4} {c.get('says', '')[:40]:<40} {note}")
    print(f"\n{len(data.get('claims', []))} 件中 {bad} 件が未確認です" if bad else "\nすべて確認できました")
    sys.exit(1 if bad else 0)
 
 
if __name__ == "__main__":
    main()

The messages are in Japanese because that's how I run it day to day; swap them for your own wording. Three choices are worth explaining:

  • A claim that can't be checked counts as a failure. An unknown kind becomes UNVERIFIABLE and the exit code is 1. This is the single line that keeps "probably done" from being counted as "done."
  • Commands never run through shell=True. Passing an agent-written string to a shell means a "check" could run something else entirely. That's why run is an array.
  • Every HTTP check changes the query string. If a cache in between is still serving the old response, you'd be judging the update by stale bytes. When cf-cache-status or age comes back, the script prints it.

What it caught when I reproduced the problem

I rebuilt the opening situation in a small local repo: file edited but not committed, endpoint still stale, purge impossible to verify. The run looked like this:

PASS          c1   一覧に ukiyoe-hiroshige を足した                パターンが見つかりました
PASS          c2   JSON として壊れていない                           exit 0
FAIL          c3   変更をコミットした                                未コミットの変更が残っています: M api/categories.json
FAIL          c4   配信側に反映され、キャッシュも消した                       HTTP 200 だが本文に 'ukiyoe-hiroshige' がありません(cache: -)
UNVERIFIABLE  c5   パージ済み                                    kind='purged' は確かめられません
 
5 件中 3 件が未確認です

The exit code was 1. After committing, refreshing the endpoint, and removing c5, I ran it again: all four remaining claims passed and the exit code went back to 0.

What stood out was that c4 returned HTTP 200. If I'd only looked at the status, it would have passed. A response existing and a response being current are two separate claims. That gap is exactly what I'd missed when I checked by hand.

kindWhat it confirmsWhat it can't tell you
file_containsThe expected text is in the fileWhether the change was saved and shared
git_committedNo diff left, file is in HEADWhether it was pushed
commandThe command exits with the expected codeWhether it looked at the right target
httpThe endpoint returns the expected contentWhether every region and device sees the same

When you feel the urge to fill in that right-hand column, that's your cue to add a kind. To confirm a push, for example, compare git ls-remote with your local HEAD.

How much to hand to the agent

Since switching to this, I've actually handed the agent more, not less. With CLI v1.5.10 adding accept-edits to --remote-agent, there are more moments where approving edits can be delegated. That's precisely why I want the "is it done?" call to stay somewhere else.

The line I draw now fits in one sentence:

The agent does the work and writes the report; a separate script makes the call.

Not everything fits a kind, of course. "The layout looks natural" or "the copy reads more politely" can't be scripted. I leave those as UNVERIFIABLE and treat them as the list I check with my own eyes. Seeing, right there in the output, which claims no machine has judged turns out to be what cuts down on the misses.

Two related checks I rely on: checking an agent's instructions locally before running them, and statically measuring whether a hook's matcher can ever fire.

One next step

Pick the next task you're about to give an agent, and ask for a claims.json instead of a report. Whether it ends up with three claims or five, if even one shows up as UNVERIFIABLE, the setup has already done its job.

If you later want to go further and check against the Cloudflare or GitHub APIs directly, the operational design in the premium articles may help you decide how far to take it. As an indie developer, I'm still adding one check at a time myself.

Share

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

If you found this article helpful, a small tip ($1.50) would mean a lot to us. Your support helps keep this site ad-free and covers server and hosting costs.

Related Articles

⬡ Integrations2026-09-20
antigravity-preview-05-2026 shuts down October 5: which setups need more than a string swap
The shutdown date for antigravity-preview-05-2026 is set for October 5, and the official page gives you the date and the successor name, nothing more. If you only read output_text, swapping the string is the whole migration. If you read steps, it is not.
◈ Agents & Manager2026-11-19
Before an Agent Reads Your Issues: Splitting GitHub Actions Into a Reader Job and a Writer Job
Anyone can write an issue body. When an agent reads it inside GitHub Actions, split the workflow into a reader job with no write access and a writer job that never runs the agent, keep the body out of the shell, and check the agent's proposal with a small Python script. Working code included.
⬡ Integrations2026-11-02
Three things to check on a Marketplace partner plugin's page before you install it
Antigravity 2.28.2 put partner plugins into the Marketplace. Here is how I split the decision: check the developer and repository before installing, then manage the connection from the Accounts tab afterward, plus a short command to see whether a repository is still maintained.