When an agent ends without a reason, I check the page's character encoding first

It happened on an afternoon spent digging through the specs of a site I had taken over for a client. Three old internal reference pages were lined up, and I was having the agent read them one after another. The first went through. The second went through. The third stopped mid-sentence.

All that was left on screen was a single line: Agent execution terminated due to error. Nothing about what it had tried to read, nothing about where it stopped, nothing about whether a retry would help.

For a while I blamed my own prompt. Too long, maybe. Too many files attached at once. I shortened the request, split the documents up, sent them again — and it ended at exactly the same place. Half a day later I finally saw it: what was failing was not my instruction, but the third page itself.

It was the page, not the prompt

The cause was that the page never declared its character encoding in the HTTP headers.

This is still an open report on the Antigravity CLI repository, filed as issue #818. When Content-Type carries no charset and the page announces itself only through something like <meta charset=iso-8859-1>, a failed decode is not caught — it terminates the agent run outright. As of September 15 it remains unresolved, and I have yet to find a write-up of it in Japanese.

That last part matters more than it sounds. Pages served as Shift_JIS or EUC-JP with no charset header are still common where I work. Old government notices, product spec pages, technical documents untouched for more than a decade. Take over an aging site as a contractor and you will meet one within the week.

So the people most likely to hit this failure are the ones reading pages in older regional encodings — and nothing on screen ever mentions encoding at all.

You can tell before you hand the page over

Once you know what to suspect, the check takes about a minute. There are only two places to look.

First, whether the HTTP header carries a charset at all.

# Fetch headers only and look at Content-Type
curl -sI https://example.com/old-spec.html | grep -i '^content-type'

This response is safe:

content-type: text/html; charset=UTF-8

This one is not. There is no charset anywhere:

content-type: text/html

When the header says nothing, pull just the top of the body and look for a meta declaration.

# Grab the first 2KB and search for meta charset
curl -s --range 0-2047 https://example.com/old-spec.html | grep -io 'charset=[a-z0-9_-]*'

If that turns up charset=shift_jis, charset=euc-jp, or charset=iso-8859-1, the page deserves one extra step before you hand it over. And if nothing turns up at all — no header, no meta — that is the variety that fails most quietly.

One detail worth keeping straight: the header wins over the meta tag. A page can carry <meta charset=shift_jis> in its markup while the server sends charset=UTF-8 in the response, and the header is what a conforming client follows. I have seen that mismatch on sites where the markup was written years before the hosting was migrated, and the markup was never revisited. Reading only the meta tag will mislead you on exactly those pages.

With no charset in the header, whatever receives the bytes has to decide for itself what they are. The historical HTTP default treats text/* as ISO-8859-1. Read multibyte content as ISO-8859-1 and something has to give. When that exception is not caught, it takes the whole agent run down with it.

Fetch it yourself, re-encode, then attach

Once you can spot the risky pages, the question becomes how to hand them over. What I settled on is simple: suspect pages never go to the agent as URLs. I fetch them, convert to UTF-8 locally, and attach the result as a file.

The script below takes one URL, tries the declared encoding first, then works through the candidates that actually show up in practice, and writes whichever succeeds out as UTF-8. Standard library only.

#!/usr/bin/env python3
"""Fetch a URL and save its contents as UTF-8.
 
Use it as the step before handing anything to an agent.
Attach the output file and the decoding is already settled on your side.
"""
import re
import sys
import urllib.request
 
# Ordered by what actually turns up in practice.
# cp932 is a superset of Shift_JIS, so it goes first.
FALLBACKS = ["utf-8", "cp932", "euc-jp", "iso-2022-jp", "latin-1"]
 
 
def detect_declared(headers, head_bytes):
    """Pull the declared charset from the HTTP header or a meta tag."""
    ctype = headers.get("Content-Type", "")
    m = re.search(r"charset=([\w-]+)", ctype, re.I)
    if m:
        return m.group(1).lower(), "header"
 
    # meta declarations stay within ASCII, so latin-1 scans them safely
    text = head_bytes.decode("latin-1", errors="replace")
    m = re.search(r'charset=["\']?([\w-]+)', text, re.I)
    if m:
        return m.group(1).lower(), "meta"
    return None, "none"
 
 
def fetch_as_utf8(url, out_path):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=30) as res:
        raw = res.read()
        declared, source = detect_declared(res.headers, raw[:2048])
 
    print(f"declared={declared or '(none)'} source={source} bytes={len(raw)}")
 
    # Try the declared encoding first, then fall through the candidates
    candidates = ([declared] if declared else []) + FALLBACKS
    for enc in candidates:
        try:
            # No errors= here, on purpose: broken characters must not pass silently
            text = raw.decode(enc)
        except (UnicodeDecodeError, LookupError):
            continue
        with open(out_path, "w", encoding="utf-8") as f:
            f.write(text)
        print(f"OK: decoded as {enc}, written to {out_path}")
        return 0
 
    print("FAILED: no candidate encoding worked. This may not be text at all.")
    return 1
 
 
if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("usage: fetch_utf8.py <URL> <output file>")
        sys.exit(2)
    sys.exit(fetch_as_utf8(sys.argv[1], sys.argv[2]))

A run looks like this:

$ python3 fetch_utf8.py https://example.com/old-spec.html spec.txt
declared=shift_jis source=meta bytes=48213
OK: decoded as cp932, written to spec.txt

One choice in there is deliberate: decode() is never given errors="replace".

Allow replacement and the script always succeeds — at the cost of leaving U+FFFD boxes scattered through the text. In material handed to an agent, the worst outcome is not a crash but content that passes through broken. A crash you notice. A spec sheet with boxes in it gets read, the missing pieces get filled in by inference, and that inference travels all the way to the end unless you happen to question it.

The mirror image shows up when an agent edits files that contain non-ASCII text, and I wrote about the checks I run for that in three byte-level checks I run before letting an agent edit files with non-ASCII text. Reading and writing turn out to need suspicion in opposite directions.

In unattended runs, fail before the handoff

The rest of this is about scheduled work that runs with nobody watching.

I run a set of Lab sites and a couple of WordPress blogs on my own as an indie developer, and most of that work happens on a timer. What hurts in unattended runs is rarely the failure itself. It is the failure arriving quietly.

A ReadURL call that kills the agent fits that shape exactly. The log records a termination and nothing about which page caused it.

So for any automation that touches URLs, I put the filter in front of the agent. The script above becomes the gate.

#!/usr/bin/env bash
# Only pass along what actually decoded
set -u
URL="$1"
OUT="/tmp/fetched_$(date +%s).txt"
 
if python3 fetch_utf8.py "$URL" "$OUT"; then
  agy -p "Read $OUT and summarize the spec changes as three bullet points"
else
  # Record the URL that failed, then finish cleanly
  echo "$(date '+%F %T') SKIP decode-failed $URL" >> fetch_failures.log
  exit 0
fi

The part that matters is exit 0. One undecodable page is no reason to mark the whole run as failed. Record what could not be read, and let the run continue. Since making that change, a glance at the log the next morning tells me exactly which pages fell out. The log has turned out to be useful in a second way as well. When the same host shows up in it week after week, that is a signal the source itself is worth replacing — an archived copy, a PDF export, anything I control the encoding of. A page that fails repeatedly is not a transient problem to retry around; it is a dependency I would rather not keep.

Worth noting alongside this: since CLI 1.1.28, fetching an external URL waits for approval by default unless you have allowed it in advance. In an unattended setup, stalling can arrive before crashing does. Where I draw that approval boundary is covered separately in three questions for routing work between Antigravity CLI and Claude Code.

Living with it until it is fixed

Issue #818 is still open, with no fix announced. It carries labels, but no reply from the maintainers yet.

So for now, noticing first is the job. The good news is that once you have decided how to notice, the cost is paid only on the first occurrence. Three habits is where I landed.

Looking back, the half day I lost was never really about character encoding. When no error is visible, I over-suspect my own side — that was the actual stumbling block. These days, when a run ends leaving no trace at all, I look at what I handed over before I look at how I asked.

For today, pick one of the old reference pages you hand over regularly and run curl -sI against it. If there is no charset in the response, you have just found the page that will stop you next.

If this spares even one person the half day it cost me, I will be glad. Thank you for reading.