Turning a Page With No Declared Charset Into UTF-8 Before

Someone asks an agent to read an older web page, and the whole run ends partway through. That is what a report on the Antigravity CLI GitHub tracker describes (issue 818).

According to the report, the server sends Content-Type: text/html with no charset, and the only hint is a <meta charset=iso-8859-1> inside the page. ReadURL decodes the body as UTF-8 anyway, hits UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 387, and the exception is not handled, so the agent run terminates. When I checked, I could not find a note saying it had been fixed.

While waiting on the tool, there is something you can do on your own side: convert the page to UTF-8 yourself and give the agent the result. This article builds a small script for that.

Reproduce the symptom first

I started a tiny local HTTP server that returns a Latin-1 page without a charset. Decoding it naively as UTF-8 on Python 3.10 gives this:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 50: invalid continuation byte

Only the byte and position differ from the issue. Whenever a server doesn't name its encoding and the reader assumes UTF-8, this is what you get.

Decide the order of evidence

Charset detection is manageable once the priority is fixed. This is a simplified version of what browsers follow (the WHATWG rules):

Order	Evidence	Why
1	Charset in the HTTP header	The server said it explicitly
2	BOM	Three leading bytes settle UTF-8
3	meta charset in the first 1024 bytes	Often the only declaration when the header is silent
4	Does it decode as strict UTF-8?	If it does, treat it as UTF-8
5	Otherwise assume cp1252	And always print a warning that this was a guess

One thing surprised me. When the meta tag says iso-8859-1, you should not decode with Python's latin-1. Decode with cp1252. Browsers do exactly that, and the difference shows up in the 0x80–0x9f range:

b = "<p>“quoted”</p>".encode("cp1252")
print(repr(b.decode("latin-1")))   # '<p>\x93quoted\x94</p>'
print(repr(b.decode("cp1252")))    # '<p>“quoted”</p>'

With latin-1, curly quotes become control characters. No error is raised, so you would quietly hand the agent damaged text.

The script

Standard library only. It takes a URL and saves a UTF-8 text file.

#!/usr/bin/env python3
"""fetch_utf8.py — 文字コード不明のページを UTF-8 のテキストにして保存する。"""
import codecs, re, sys, urllib.request
from html.parser import HTMLParser
 
MAX_BYTES = 2_000_000
META = re.compile(rb'''<meta[^>]+charset\s*=\s*["']?\s*([A-Za-z0-9_\-:.]+)''', re.I)
HDR = re.compile(r'charset\s*=\s*["\']?([A-Za-z0-9_\-:.]+)', re.I)
 
def normalize(label):
    """ラベルを Python のコーデック名へ。WHATWG 流に latin-1 は cp1252 として扱う。"""
    try:
        name = codecs.lookup(label.strip()).name
    except LookupError:
        return None
    return "cp1252" if name in ("iso8859-1", "latin-1") else name
 
def detect(body, content_type):
    m = HDR.search(content_type or "")
    if m and normalize(m.group(1)):
        return normalize(m.group(1)), "http-header"
    if body.startswith(codecs.BOM_UTF8):
        return "utf-8-sig", "bom"
    m = META.search(body[:1024])
    if m and normalize(m.group(1).decode("ascii", "ignore")):
        return normalize(m.group(1).decode("ascii", "ignore")), "meta"
    try:
        body.decode("utf-8")
        return "utf-8", "utf8-valid"
    except UnicodeDecodeError:
        return "cp1252", "fallback"
 
class Text(HTMLParser):
    def __init__(self):
        super().__init__(); self.out = []; self.skip = 0
    def handle_starttag(self, tag, attrs):
        if tag in ("script", "style"): self.skip += 1
    def handle_endtag(self, tag):
        if tag in ("script", "style") and self.skip: self.skip -= 1
        if tag in ("p", "div", "li", "h1", "h2", "h3", "br", "tr", "title"): self.out.append("\n")
    def handle_data(self, d):
        if not self.skip: self.out.append(d)
 
def main(url, dest):
    req = urllib.request.Request(url, headers={"User-Agent": "fetch-utf8/1.0"})
    with urllib.request.urlopen(req, timeout=20) as r:
        body = r.read(MAX_BYTES + 1)
        ctype = r.headers.get("Content-Type", "")
    if len(body) > MAX_BYTES:
        sys.exit("too large: " + url)
    enc, why = detect(body, ctype)
    html = body.decode(enc, errors="replace")
    p = Text(); p.feed(html)
    text = re.sub(r"\n{3,}", "\n\n", "".join(p.out)).strip()
    with open(dest, "w", encoding="utf-8") as f:
        f.write(text + "\n")
    if why == "fallback":
        print("warning: 文字コードを判定できず cp1252 と仮定しました。結果を目で確認してください", file=sys.stderr)
    print(f"encoding={enc} via={why} replaced={html.count(chr(0xFFFD))} -> {dest}")
 
if __name__ == "__main__":
    if len(sys.argv) != 3: sys.exit("usage: fetch_utf8.py URL OUT.txt")
    main(*sys.argv[1:])

Tags are stripped with HTMLParser, and the contents of script and style are dropped. Tables and links lose their structure, which I judged acceptable when the goal is simply to pass the content along without mojibake.

Check it locally

Here is a test server with one page per branch. /nometa is meant to be a page with no declaration at all.

from http.server import BaseHTTPRequestHandler, HTTPServer
PAGES = {
 "/latin1": ("text/html", "<html><head><meta charset=iso-8859-1><title>Asesoría</title></head><body><p>Año nuevo, señor.</p></body></html>".encode("iso-8859-1")),
 "/sjis":   ("text/html", "<html><head><meta http-equiv='Content-Type' content='text/html; charset=Shift_JIS'></head><body><p>壁紙アプリの更新履歴です。</p></body></html>".encode("shift_jis")),
 "/utf8":   ("text/html", "<html><body><p>日本語 UTF-8 そのまま</p></body></html>".encode("utf-8")),
 "/hdr":    ("text/html; charset=euc-jp", "<html><body><p>ヘッダ優先の確認</p></body></html>".encode("euc-jp")),
 "/nometa": ("text/html", "<html><body><p>Año sin宣言</p></body></html>".encode("cp1252", "replace")),
}
class H(BaseHTTPRequestHandler):
    def do_GET(self):
        ct, b = PAGES[self.path]
        self.send_response(200); self.send_header("Content-Type", ct); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
    def log_message(self, *a): pass
HTTPServer(("127.0.0.1", 8765), H).serve_forever()

python3 serve.py &
for p in latin1 sjis utf8 hdr nometa; do
  python3 fetch_utf8.py http://127.0.0.1:8765/$p out_$p.txt
done

The results:

[latin1]  encoding=cp1252 via=meta        -> Asesoría / Año nuevo, señor.
[sjis]    encoding=shift_jis via=meta     -> 壁紙アプリの更新履歴です。
[utf8]    encoding=utf-8 via=utf8-valid   -> 日本語 UTF-8 そのまま
[hdr]     encoding=euc_jp via=http-header -> ヘッダ優先の確認
[nometa]  encoding=cp1252 via=fallback    -> warning printed

All five branches behaved as intended. /hdr has no declaration in the body and only the header names euc-jp, which confirms the header wins.

The pitfall is the case you can't detect

The /nometa output read Año sin??. That wasn't a detection failure: my test page itself had been written with unencodable characters replaced by ?. Still, it points at something important. A page with no declaration that also isn't valid UTF-8 cannot be decoded with any guarantee, whatever library you use.

That's why the script warns when it reaches the fifth step. When you see the warning, glance at the saved text before handing it over.

Handing it to the agent

Attach the saved file to the conversation and ask for a summary. Because you never pass the URL itself, ReadURL's decoding is never involved.

The issue also lists curl -s URL | iconv -f iso-8859-1 -t utf-8 as a workaround. For a one-off fetch where you already know the encoding, that is enough. If you read different sites every time, automating the detection saves effort later.

Order of use

Save the page with fetch_utf8.py first
If there was no warning, attach it as is
If there was a warning, check the first few lines, then attach it

As an indie developer who runs a few small sites, I run into older pages during research fairly often. Prepare the input on your side before the agent has a chance to stop. I started by keeping this one script close at hand.

Turning a Page With No Declared Charset Into UTF-8 Before the Agent Reads It