◉ANTIGRAVITY LABJP
Articles/App Development
▣ App Development/2026-10-28Intermediate

I Fetched All 155 Sitemap URLs. Every One Returned 200, but 93% of the lastmod Dates Were the Same Day

A record of fetching every sitemap URL and cross-checking status, canonical, hreflang and lastmod. Full standard-library Python scripts and the mismatch I found.

sitemaphreflanglastmodPython25Cloudflare6indie development26site operations
✦ Premium Article

Does every URL I put in the sitemap really open the way I intend? As an indie developer, I found myself wondering about this on dolice.net, my own site.

Until now, I opened five or six pages in a browser and told myself it looked fine. A spot check covers a few percent of the whole.

So I wrote a small script that reads the sitemap and fetches every URL listed. Even if you hand this kind of audit to an agent, running it yourself once first gives you a way to judge whether the report that comes back is right.

The result first: all 155 URLs returned 200, and the hreflang references were reciprocal with no breaks. I expected nothing to fix. Then I lined up the lastmod values and stopped.

I fixed six checks before writing any code

Decide what to check before you write the script. Adding items afterward makes it easy to bend the criteria around the result.

CheckWhere it livesWhat goes wrong if it drifts
StatusResult of a GET on the URL404s and redirects in the sitemap waste crawl budget
canonicalThe page's link tagIf the page declares a different URL as canonical, one of the two gets ignored
html langThe html elementCatches an English path whose lang says ja
noindexmeta robotsListing a noindex URL in the sitemap is a contradiction
hreflangBoth the sitemap and the pageA one-way declaration can cause the language pairing to be ignored
lastmodThe sitemap and the Last-Modified headerIf the date can't be trusted, it is no signal for recrawling

For hreflang only, I fetch the sitemap declaration and the page tags separately and compare them. The same information is written in two places, so there are twice as many ways for it to drift.

The fetch script, in full

It runs on the standard library alone. It reads the sitemap, fetches every URL with six workers, and writes the results to JSON.

I paid attention to two things. First, it does not follow redirects. Python's urllib silently follows 301 and 302 by default, so a report of "200" can hide a redirect behind it. I plug in NoRedirect and count redirects as redirects.

Second, it never discards a URL that failed to fetch. If you keep status: 0 and the reason, you won't spend time later wondering why the counts don't add up.

#!/usr/bin/env python3
"""sitemap_audit.py — サイトマップの全 URL を実際に取得し、宣言と実体のずれを数える。
使い方: python3 sitemap_audit.py https://example.com/sitemap.xml > report.json
"""
import json, re, sys, time, urllib.request, urllib.error
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor
 
NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9",
      "x": "http://www.w3.org/1999/xhtml"}
UA = "sitemap-audit/1.0 (own-site check)"
 
class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *a, **k):
        return None  # 301/302 を自分で観測する（追従しない）
 
opener = urllib.request.build_opener(NoRedirect)
 
def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    t0 = time.time()
    try:
        r = opener.open(req, timeout=20)
        status, headers, body = r.status, r.headers, r.read()
    except urllib.error.HTTPError as e:
        status, headers, body = e.code, e.headers, b""
    except Exception as e:
        return {"url": url, "status": 0, "error": str(e)[:80]}
    ms = int((time.time() - t0) * 1000)
    html = body.decode("utf-8", "replace")
    def first(p):
        m = re.search(p, html, re.I | re.S)
        return m.group(1).strip() if m else None
    canon = first(r'<link[^>]+rel=["\']canonical["\'][^>]+href=["\']([^"\']+)')
    lang = first(r'<html[^>]+lang=["\']([^"\']+)')
    robots = first(r'<meta[^>]+name=["\']robots["\'][^>]+content=["\']([^"\']+)')
    alts = dict(re.findall(
        r'<link[^>]+rel=["\']alternate["\'][^>]+hreflang=["\']([^"\']+)["\'][^>]+href=["\']([^"\']+)', html))
    return {"url": url, "status": status, "ms": ms,
            "location": headers.get("Location"), "canonical": canon,
            "html_lang": lang, "robots": robots, "page_hreflang": alts,
            "bytes": len(body)}
 
def main(sm_url):
    xml = urllib.request.urlopen(
        urllib.request.Request(sm_url, headers={"User-Agent": UA}), timeout=20).read()
    root = ET.fromstring(xml)
    entries = {}
    for u in root.findall("s:url", NS):
        loc = u.find("s:loc", NS).text.strip()
        alts = {l.get("hreflang"): l.get("href")
                for l in u.findall("x:link", NS) if l.get("rel") == "alternate"}
        entries[loc] = {"sm_hreflang": alts,
                        "lastmod": (u.find("s:lastmod", NS).text if u.find("s:lastmod", NS) is not None else None)}
    with ThreadPoolExecutor(6) as ex:
        results = {r["url"]: r for r in ex.map(fetch, entries)}
    out = []
    for url, e in entries.items():
        r = results[url]; r.update(e); out.append(r)
    json.dump(out, sys.stdout, ensure_ascii=False, indent=1)
 
if __name__ == "__main__":
    main(sys.argv[1])

Run it like this. It finished in well under a minute for 155 URLs. Response time had a median of 170ms and a maximum of 267ms, and six workers against my own site caused no visible load.

python3 sitemap_audit.py https://dolice.net/sitemap.xml > report.json
✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN
✦You can cross-check status, canonical and hreflang for every URL in your sitemap with one script, and retire the habit of spot-checking a few pages by hand
✦You can verify hreflang reciprocity and the gap between sitemap and page tags by machine, and prevent language versions from swapping in search results before you publish
✦You can watch for lastmod collapsing onto a single day through an exit code, and notice sooner when your update signal has stopped meaning anything
Secure payment via Stripe · Cancel anytime
✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

or
Unlock all articles with Membership →
Share

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

  • ✦Copy-paste ready implementation code
  • ✦New advanced guides published daily
  • ✦$5/mo or $10 for lifetime access
View Membership →

Related Articles

▣ App Dev2026-10-18
Before Letting an Agent Touch My Config, I Found That .env* Doesn't Cover .dev.vars, and Wrote a Preflight Script
Auditing my own Cloudflare Workers repo showed that the .env* line in .gitignore never protected .dev.vars. Here is the full Python script that asks git itself where secrets can leak before an agent opens the repo.
▣ App Dev2026-09-19
I Asked for the Whole Quote PDF Tool at Once, Then Split It Into Four Requests
How I moved my client quote paperwork from copy-and-edit to a small PDF tool built with Antigravity, why the first single-shot request had to be thrown away, and the four smaller requests that replaced it, with working code.
✦ Tips2026-10-23
594 Articles Had No Body-Link Entrance. Counting the Related-Articles Box Left Only 6 Truly Orphaned
Counting only body links made half my MDX site look orphaned. Here is the full Python script that mirrors the related-articles scoring and also catches links the build strips.