How Far to Narrow an Agent's Choices in a 30-Category Wallpaper Classification Pipeline
Asking an agent to pick one of 30 categories per image means re-running every image the moment a definition changes. Here is the reasoning and the implementation behind switching to closed-vocabulary tags plus a deterministic rule mapping.
Sorting wallpapers into thirty delivery categories is the kind of work that piles up quietly. You can tell at a glance which bucket an image belongs in, but there are thousands of them. As an indie developer shipping wallpaper apps, I have a steady supply of tasks shaped exactly like this, and handing the looking-at-images part to an agent felt like an obvious fit.
What tripped me up was not accuracy. After running the pipeline for a few weeks, I wanted to split night-time nature scenes out of the existing "Landscape" bucket. That single change put every previously classified image back in scope. The stored output was a list of image IDs and category names, and nothing in that list tells you whether an image was a dark forest or a sunrise over a ridge. The only way forward was to show every image to the model again.
That was the moment the design flaw became obvious. I had been asking the agent to hand me an interpretation.
Why adding one category meant redoing everything
The saved record was a single line per image: 210756 → Landscape. There is no trace of why that verdict was reached, so there is nothing to re-query when the definition of Landscape narrows.
Category definitions always move in production. You split a bucket because the delivery surface needs finer slots; you merge one because nobody taps it. Any structure that requires full re-inference on every definition change gets heavier as the library grows. My batches are capped at 50 images, twice a day, to stay under rate limits — so a full redo across a few thousand images burns weeks of wall clock time for a change that has nothing to do with the images themselves.
The pixels did not change. Only my bookkeeping did. Re-measuring something that has not moved is, in hindsight, plainly wasteful.
Let the agent observe, let a script interpret
The revised design is simple: the agent reports what is in the image and never names a category.
Layer
Owner
Output
On a definition change
Observation
Agent reading the image
Array of tags
Not re-run
Interpretation
Deterministic script
Category membership
Swap the rule table and re-run
Observations are facts about the image, so they can be captured once and reused. Interpretation is my bookkeeping, so it lives outside the model where it is cheap to change. Since the split, the number of inference calls required by a category change has been zero.
Reproducibility improved as a side effect. When the model chose one of thirty labels, adjacent categories drifted between runs — the same night cityscape came back as "Night View" one day and "Landscape" the next. Neither answer is wrong, which is exactly the problem: the ambiguity lived in my category boundaries and was getting baked into the output. Asking for night, city, skyline produced almost no such drift. The images were never the ambiguous part.
✦
Thank you for reading this far.
Continue Reading
What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.
WHAT YOU'LL LEARN
✦You will be able to draw the line in your own project between what an agent should judge and what belongs in a deterministic rule, assuming your categories will change
✦You will be able to design your way out of re-running inference over every image each time you add or merge a category
✦You will know how to resume a 50-item batch after a mid-run failure without dropping items or emitting duplicates
Secure payment via Stripe · Cancel anytime
✦
Unlock This Article
Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.
Free-form tags kill this design. The moment night, nighttime, dark, and evening all appear for the same kind of photo, no rule table can keep up.
So the vocabulary is closed explicitly, and the allowed list is handed to the agent verbatim.
{ "task": "classify_wallpaper_observation", "id": 210756, "allowed_tags": [ "night", "city", "skyline", "street", "mountain", "sea", "sky", "forest", "flower", "petal", "cat", "dog", "bird", "gradient", "geometric", "texture" ], "rules": [ "Emit nothing outside allowed_tags", "Omit any tag you are unsure about", "At most five tags per image" ], "output": { "id": 210756, "tags": ["night", "forest"] }}
Out-of-vocabulary tags still show up. Rather than rejecting them, I count them. The mapping script tallies unknown tags, and once a week I skim the list and promote the frequent ones into the official vocabulary. A recurring term like neon-cyberpunk often points at a slice of the catalog that actually has demand on the store side. Treating those as errors would have thrown that signal away.
The mapping script
Here is the version I run, trimmed to the essentials. all means every listed tag must be present, any means at least one, and none excludes.
#!/usr/bin/env python3"""tags.jsonl -> per-category ID lists (deterministic mapping)."""import json, sysfrom pathlib import PathCATEGORY_ORDER = ["Popular", "Landscape", "Night", "Flower", "Animal", "Abstract"]RULES = [ ("Night", {"all": ["night"], "any": ["city", "skyline", "street"]}), ("Landscape", {"any": ["mountain", "sea", "sky", "forest"]}), ("Flower", {"any": ["flower", "petal"]}), ("Animal", {"any": ["cat", "dog", "bird"]}), ("Abstract", {"any": ["gradient", "geometric", "texture"], "none": ["cat", "dog", "bird"]}), ("Popular", {"any": ["featured"]}),]VOCAB = {t for _, r in RULES for k in ("all", "any", "none") for t in r.get(k, [])}def match(rule, tags): if any(t not in tags for t in rule.get("all", [])): return False if rule.get("any") and not any(t in tags for t in rule["any"]): return False if any(t in tags for t in rule.get("none", [])): return False return Truedef main(path): buckets = {c: [] for c in CATEGORY_ORDER} unknown, unassigned = {}, [] for line in Path(path).read_text(encoding="utf-8").splitlines(): if not line.strip(): continue rec = json.loads(line) wid, tags = str(rec["id"]), set(rec["tags"]) for t in tags - VOCAB: # count, do not discard unknown[t] = unknown.get(t, 0) + 1 hit = [c for c, r in RULES if match(r, tags)] if not hit: unassigned.append(wid) # keep the gap visible for c in hit: buckets[c].append(wid) for c in CATEGORY_ORDER: # order is fixed by the template print(f"{c}: " + ", ".join(buckets[c])) print(f"[unassigned] {len(unassigned)} -> {unassigned}", file=sys.stderr) print(f"[unknown-tags] {sorted(unknown.items())}", file=sys.stderr)if __name__ == "__main__": main(sys.argv[1])
Three choices in there are deliberate.
Output order is fixed by the script. The delivery template dictates the order of categories. Expecting a model to honor ordering holds up right until a long batch starts drifting near the end. Order belongs in a rule.
Unassigned IDs go to stderr. Dropping them silently means chasing a total that does not add up, days later.
Unknown tags are tallied. As above, that tally is how the vocabulary grows.
The none clause on Abstract came out of production. Close-ups of cat fur kept picking up texture and landing in both Animal and Abstract. Fixing that took one line in the mapping — no prompt surgery, no re-inference.
Watching observation quality without ground truth
Hand-labelling a validation set turns into its own unending project. Instead of ground truth, I watch three numbers.
Signal
How it is measured
What a drop suggests
Tag stability
Read the same image twice, compare the tag sets
Vague instructions, or a vocabulary that is too fine-grained
Out-of-vocabulary rate
Share of images carrying at least one unknown tag
The vocabulary has fallen behind the catalog
Unassigned rate
Share of images landing in no category at all
A hole in the rules, or a category worth adding
All three are computable without knowing the right answer, which is the point. Print them to stderr on every batch and regressions announce themselves.
Spot-checking a handful of images with a second read has been enough in practice. When stability drops, the usual cause is a vocabulary with near-synonyms in it. I once had both sea and ocean available, and the same seascape alternated between them across runs. Removing one settled it. I now treat vocabulary size as something to keep small enough that no judgement call is required, rather than something to grow.
The unassigned rate cuts both ways. It exposes gaps in the rules, and it also nominates new categories: tally the tags on unassigned IDs and the same combination often keeps reappearing. Whether that combination deserves a slot on the delivery surface is a call I make, not one the number makes for me.
What actually runs when you add a category
Back to the change that started this. With the split in place, only the rule table is touched.
Running that against a seven-record sample — a night, city, skyline, bokeh cityscape, a mountain, sky, sunrise ridge, a night, forest scene, and four others — moved exactly one record from Landscape to Night Nature and left the rest identical. Not a single image was read again.
Change
Direct category selection
Tag-then-map
Inference required
Every image, again
None
Time to completion
Bounded by the 100 images/day cap
Script runtime
Diff surface
The entire output file
One rule table
Consistency with past runs
May drift on each re-run
Same tags always yield the same result
That last column is the real reason I made the switch. Not better accuracy — better tolerance for change.
Always eyeball the diff after a rule change
One line in a rule table reaches a long way. Whether only the intended record moved, or others came along, is not something you can read off the diff of the rules themselves. So a small script compares the old and new mappings and prints only the IDs whose membership changed.
#!/usr/bin/env python3"""Compare two mapping outputs and report which IDs changed category."""import subprocess, sys, collectionsdef load(script, data): out = subprocess.run([sys.executable, script, data], capture_output=True, text=True).stdout m = collections.defaultdict(set) for line in out.splitlines(): cat, _, ids = line.partition(":") for i in ids.split(","): if i.strip(): m[i.strip()].add(cat.strip()) return mold, new = load(sys.argv[1], sys.argv[3]), load(sys.argv[2], sys.argv[3])moved = 0for wid in sorted(set(old) | set(new)): a, b = old.get(wid, set()), new.get(wid, set()) if a != b: moved += 1 print(f"{wid}: {sorted(a) or ['-']} -> {sorted(b) or ['-']}")print(f"moved={moved} / total={len(set(old) | set(new))}")
Run against the Night Nature change from the previous section, it prints:
Exactly the one record I was after, and nothing else. total counts IDs that landed in at least one category under either rule set, so the record that stayed unassigned is not included.
This check earns its keep most when you add a none clause. Exclusions reach further than the person writing them tends to picture: a line meant to stop animal-and-abstract double registration can push out an unrelated group of images that happen to share a tag, and reading the rules will not reveal it. When moved comes back an order of magnitude larger than expected, I revert and rethink rather than inspect the fallout.
Resuming a 50-item batch after a failure
Splitting observation from interpretation does not spare the observation side from rate limits. My batches are 50 images, twice a day. When a read hits the ceiling, the run waits 60 seconds, then 120, then 180, and gives up on the remainder of the batch after the third failure.
My first mistake here was checkpoint granularity. Saving after every image felt safest, and it corrupted the output instead. Output is one file per 50 records in a fixed template order; a half-written file leaves the next run unable to tell whether that file is finished.
Write the output file exactly once, at batch completion. Partial results stay in memory
Record abandoned images in skipped_ids — not as a deletion, not as a hard failure
Select the next batch as "IDs above last_processed_id that are not in skipped_ids"
Backfill the skipped IDs later, as a separate run
Step three is what earns its keep. If a skipped ID lives outside the progress state, the next run either picks it up again and emits a duplicate, or skips it forever. Recording the failure as state prevents both at once.
Where the boundary sits on Antigravity background runs
Moving this onto an Antigravity background agent does not change where the line goes. The agent owns observation and nothing else.
The agent's artifact is limited to appended lines in tags.jsonl
The mapping script runs outside inference, deterministically, at the end of the turn
AGENTS.md states plainly: never emit a category name, never use a term outside allowed_tags
Progress-file updates belong to the script. State management is not the agent's job
One turn handles exactly one batch, and nothing is carried across turns
That last point grew on me over time. The longer a run goes, the more an agent leans on context from earlier turns and quietly skips steps it believes it already handled. Having the script hand it the work list each turn keeps turn fifty identical to turn one.
This pairs well with the idea of making agent work verifiable through concrete artifacts. A human can read tags.jsonl directly, and when a category assignment looks wrong you can tell whether the observation was off or the rule was. Back when both lived in one output, that question had no answer.
When not to bother
This is not a universal answer. In these cases, letting the model pick a category outright will get you there faster.
Condition
Reason
Around five categories, stable for the foreseeable future
Designing a vocabulary and rule table will not pay for itself
A one-off classification with no future updates
Re-run cost is not a concern you have
Category depends on context outside the image
If the deciding information cannot become a tag, the split loses its edge
If, on the other hand, you ship on the App Store or Google Play and your categories keep moving with the storefront, splitting early is worth it. I split late, and paid for it in images read twice.
A concrete first step: open your current classification output and check whether it records why each verdict was reached. If it does not, the next time you touch a category definition you will land in the same spot I did.
Share
Thank You for Reading
Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.