When the Deploy Was Green but Users Still Saw the Old Build: Field Notes on a Gate That Verifies Your Shipped Commit Reached the Edge

After deploy, go fetch the real thing from the edge

With the stamp baked in, place a gate at the tail of the pipeline that, right after deploy, fetches production's version.json and waits until it matches the SHA you just shipped. If it doesn't match, that release either hasn't arrived yet or went somewhere else.

# What this verifies:
# After deploy, fetch production version.json with cache bypass and
# poll until it matches the expected SHA within a budget.
import sys, time, json, urllib.request
 
def fetch_live_sha(url: str) -> str:
    # Add a query to bypass intermediary caches and reach the origin.
    bust = f"{url}?t={int(time.time()*1000)}"
    req = urllib.request.Request(bust, headers={"Cache-Control": "no-cache"})
    with urllib.request.urlopen(req, timeout=10) as r:
        return json.loads(r.read())["sha"]
 
def verify_deploy(url: str, expected_sha: str, budget_sec: int = 180) -> bool:
    deadline = time.time() + budget_sec
    interval = 5
    while time.time() < deadline:
        try:
            live = fetch_live_sha(url)
        except Exception as e:
            print(f"probe error: {e}")
            live = None
        if live == expected_sha:
            print(f"converged: edge now serves {live}")
            return True
        print(f"still stale: edge={live} expected={expected_sha}")
        time.sleep(interval)
        interval = min(interval * 2, 30)
    return False
 
if __name__ == "__main__":
    url, expected = sys.argv[1], sys.argv[2]
    ok = verify_deploy(url, expected)
    sys.exit(0 if ok else 1)

Put this at the very end of the CI deploy stage and turn the whole pipeline red if verify_deploy fails. Only then does green recover its meaning: "the new code reached users."

Separate propagation lag from "shipped to the wrong place"

When verification doesn't converge within budget, the cause splits two ways. If it's plain propagation lag, the stamp eventually converges to the expected value. But if it stays on the old SHA indefinitely, it's likely not propagation — you shipped to the wrong target. Suspect a preview environment or a different branch, and check whether the production domain alias actually re-pointed.

As a heuristic, always log three things: whether it converged, the SHA the edge finally returned, and how many seconds it took. A steady old SHA points at the wrong target; wavering between old and new points at region skew or a cache layer.

The trap where caches return a stale stamp

Verification itself can be fooled by caches. If version.json is cached at the CDN, the origin is already new but the check keeps grabbing the old value — a false negative that merely looks like "not arrived."

Handle it in two layers. The fetch side bypasses caches with a query string and sends Cache-Control: no-cache. The serving side gives only version.json a short or zero TTL so this one path always hits the origin. The app bundle itself can keep its normal long-lived cache. The knack is deliberately placing just the verification stamp outside the cache.

Measured: what changed after adding the gate

Here's how incidents of an old build reaching users changed before and after the gate. Same app, roughly four months of releases.

Deploys turning red because of the gate went up. But that red is the red that stopped us at the moment of shipping instead of serving an old build to users. Five hours of noticing via support versus three minutes of stopping on the spot — that gap was where this paid off most.

Notes for putting it into production

Always tie the verification gate to a rollback trigger. If the gate fails, automatically revert to the last stable release, or at least steer new traffic to the old version. Sitting on a new release that hasn't arrived is the most dangerous outcome.

And set the verification budget from measurement, matched to your propagation characteristics. I'd recommend adopting this gate first — but too short a budget will wrongly red-flag normal propagation lag. Start generous, watch the distribution of convergence times, then tighten. In my setup about 95% of healthy releases aligned the stamp within 40 seconds, so I keep the budget at a comfortable 180 seconds. If you want to revisit the happy-path deploy first, deploying to Cloudflare Workers pairs well and makes it clearer where to slot the stamp in.

From trusting "it should have succeeded" to going and checking whether it arrived — it's one extra step, but the incidents of quietly serving old code dropped for good. If you're wrestling with the same mismatch, I hope this helps with tomorrow's next move. Thanks for reading.