A dependency without musllinux wheels doesn't fail on Alpine. It just gets slower

The morning I read that the Antigravity Python SDK had picked up PEP 656 musllinux wheels, the first thing that came to mind was a small job I wrote for myself. It collects article metadata across the Lab sites and counts a few things. I've kept that image deliberately small, so it runs on Alpine Linux.

At some point that job started taking noticeably longer, in a way I couldn't explain. I decided the upstream responses must be slow and left it alone for a while. Looking back, I skipped the cheapest thing I could have checked.

The cause was that one dependency inside the container was the pure-Python implementation rather than the C extension. pip never raised an error. The log said Successfully installed, right on schedule — the problem came back to me as a slowdown, not a failure.

pip doesn't say "no". It picks whatever fits

pip builds a list of compatible tags for the environment it's installing into, then takes the first distribution that matches. If no wheel matches at all, it falls back to the source distribution and tries to build it.

Printing that tag list is a good way to see how mechanical the choice really is.

python3 - <<'PY'
from packaging import tags
ts = list(tags.sys_tags())
print("total:", len(ts))
print("manylinux:", sum(1 for t in ts if "manylinux" in t.platform))
print("musllinux:", sum(1 for t in ts if "musllinux" in t.platform))
PY

On a glibc 2.35 Linux box running Python 3.10.12, I got 818 tags in total: 782 manylinux, zero musllinux. On Alpine, which uses musl libc, that entire manylinux block drops out. What's left decides which of three things happens.

One more detail about the tags themselves. The musllinux tag carries a musl version, and the SDK's new wheels are tagged musllinux_1_1 for x86_64 and aarch64, while cryptography publishes musllinux_1_2. pip treats a lower minor version as compatible with a newer musl runtime, so a _1_1 wheel installs on a _1_2 system but not the other way around. Publishing against the older tag is the conservative choice, and as a consumer it means you rarely have to think about the number.

The third row is the one that cost me time. A failed build tells you in red text. A silent downgrade to a pure-Python implementation tells you nothing at all.

I measured it by swapping implementations on one machine

I wanted a number I could keep. Rather than standing up an Alpine image to compare, I switched only the protobuf implementation on the same machine using an environment variable. What happens on musl is that this same switch flips without anyone asking for it.

# pb_bench.py
import time, sys
from google.protobuf.internal import api_implementation
from google.protobuf import struct_pb2
 
impl = api_implementation.Type()
N = int(sys.argv[1]) if len(sys.argv) > 1 else 300
 
s = struct_pb2.Struct()
s.update({f"key_{i}": {"n": i, "s": "x" * 20, "b": i % 2 == 0} for i in range(200)})
blob = s.SerializeToString()
 
t0 = time.perf_counter()
for _ in range(N):
    s.SerializeToString()
t1 = time.perf_counter()
for _ in range(N):
    m = struct_pb2.Struct()
    m.ParseFromString(blob)
t2 = time.perf_counter()
 
print(f"impl={impl} bytes={len(blob)} "
      f"serialize={(t1 - t0) / N * 1e3:.3f}ms parse={(t2 - t1) / N * 1e3:.3f}ms")

python3 pb_bench.py 300
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python3 pb_bench.py 300

Python 3.10.12, protobuf 7.35.1, glibc 2.35, x86_64. A Struct with 200 keys that serializes to 13,690 bytes, 300 iterations each way.

That's roughly 840x on serialization and 285x on parsing. My first attempt used N = 2,000, and the pure-Python run hit a 60-second timeout before it could print anything. When the gap is that wide, the measurement setup breaks before the measurement does.

It's worth knowing why the gap is that large rather than merely noticeable. The C extension encodes and decodes fields in compiled code and hands Python a thin wrapper over the result. The pure-Python path walks every field, every nested message, and every map entry through the interpreter. My test message has 200 nested structs, so the interpreter overhead is paid 200 times per call, twice over — once building the wire format and once taking it apart.

That shape matters when you're deciding whether to care. A job that sends a handful of small requests per run will never notice. A job that parses a few thousand messages in a loop turns a sub-second step into a coffee break, and the profile will point at your own code rather than at the dependency, because that's where the time appears to be spent.

Checking which one is loaded takes a single line.

python3 -c "from google.protobuf.internal import api_implementation as a; print(a.Type())"
# upb    -> C extension, as expected
# python -> pure Python, which is the trap here

The question isn't whether the install succeeded. It's what actually got installed. That's the one check I try not to skip, even on a rushed day.

Take inventory of your wheels before you build the image

The PyPI JSON API will tell you which wheels a given release publishes. Running it once by hand, before wiring anything into CI, is usually enough to settle the question.

# wheel_audit.py
import json, urllib.request, sys
 
for pkg in sys.argv[1:]:
    d = json.load(urllib.request.urlopen(f"https://pypi.org/pypi/{pkg}/json", timeout=15))
    files = [f["filename"] for f in d["urls"] if f["filename"].endswith(".whl")]
    musl = [f for f in files if "musllinux" in f]
    pure = [f for f in files if f.endswith("py3-none-any.whl")]
    verdict = "musl wheel" if musl else ("pure python" if pure else "sdist build")
    print(f"{pkg:22} {d['info']['version']:10} whl={len(files):4d} "
          f"musl={len(musl):3d} pure={len(pure)}  -> {verdict}")

Here are the interesting rows from a run on September 3, 2026.

Two rows deserve a second look. Cryptography publishes only six musllinux wheels and still doesn't cause trouble, because a cp39-abi3 stable-ABI wheel covers everything from 3.9 up. Reading the count alone and concluding "too few, must be risky" gets it backwards.

Tokenizers is the opposite shape. It does ship musllinux wheels, but only for cp310, so the 3.12 and 3.13 interpreters in newer Alpine images find no match and fall through to the sdist, which wants a Rust toolchain. That one fails loudly, which makes it the easier problem.

Protobuf ships zero musllinux wheels while also publishing py3-none-any — so it never fails. And because it tends to arrive as a transitive dependency around gRPC, it slips in without being noticed.

A more direct approach is to stop counting and let pip make the choice, then read back the filenames it picked. --dry-run installs nothing, and --report writes the resolution out as JSON.

pip install --dry-run --ignore-installed --report /tmp/rep.json \
    google-genai protobuf grpcio
python3 - <<'PY'
import json
d = json.load(open("/tmp/rep.json"))
for it in d["install"]:
    name = it["metadata"]["name"]
    fn = it.get("download_info", {}).get("url", "").rsplit("/", 1)[-1]
    print(f"{name:22} {fn}")
PY

On glibc, that prints filenames like protobuf-7.36.1-cp310-abi3-manylinux2014_x86_64.whl. Run the same command inside the Alpine image and look for the line that has become protobuf-7.36.1-py3-none-any.whl. That turned out to be the shortest check I have. Even with 28 resolved dependencies, all you compare is filenames.

Make it visible at startup

Passing --only-binary :all: forbids sdist builds, but a pure-Python wheel is a perfectly legitimate distribution, so it sails right through. What the flag can't cover, you have to look at yourself.

I now put about ten lines at the entry point of the job. It fails in CI and only warns in production.

# runtime_guard.py
import os, sys, logging
 
def check_protobuf_impl(strict: bool = False) -> str:
    from google.protobuf.internal import api_implementation
    impl = api_implementation.Type()
    if impl != "upb":
        msg = f"protobuf implementation is '{impl}' (expected 'upb')"
        if strict:
            raise RuntimeError(msg)
        logging.warning("%s - serialization may be orders of magnitude slower", msg)
    return impl
 
if __name__ == "__main__":
    print(check_protobuf_impl(strict=os.environ.get("CI") == "true"))
    sys.exit(0)

I only turn on strict=True in CI. Stopping a production job over an implementation detail turns "slower" into "not running at all," and there are days when staying up matters more than being fast — I'd rather keep both behaviors and let the environment decide which one applies.

The base-image question got simpler for me too. If every dependency at the center of the job has musllinux wheels, Alpine stays. If even one of them doesn't, I move to python:3.12-slim and accept the extra few dozen megabytes.

If you do stay on Alpine with a dependency that needs building, the usual escape hatch is a multi-stage Dockerfile: install the toolchain in a builder stage, run pip wheel to produce wheels into a directory, and copy only those into the runtime stage with --no-index --find-links. It keeps the final image small without asking the runtime image to carry a compiler. That works well for the loud failures. It does nothing for the quiet one, because there was never anything to build.

One habit came out of this that has nothing to do with protobuf. When a container job gets slower and the code hasn't changed, I now check the environment before I check the network. The environment is cheap to inspect and it changes underneath you — a base image tag moves, a dependency cuts a release, a resolver picks a different file. The network is expensive to inspect and usually innocent.

Start by printing one line

Digging back through container build logs takes longer than asking a running job to print api_implementation.Type(). If the string comes back upb, you can forget this article entirely. If it comes back python, that's where the measuring starts.

I was blaming the network for time my own container was spending. Thank you for reading.