Running Gemma 4 on a Self-Hosted vLLM Server from Antigravity — A Practical Guide to Building High-Throughput Inference

The smallest vLLM that Antigravity can actually talk to

The first thing to build is not a production-ready stack. It is the minimum that lets you verify "Antigravity sends a prompt, tokens come back from a self-hosted server." Everything else builds on that verification.

Drop the following docker-compose.yml and .env into the same directory. The setup assumes a Linux host with a single NVIDIA GPU of 24 GB or more.

# docker-compose.yml
# Minimum viable vLLM serving Gemma 4 9B with an OpenAI-compatible API
# - Exposes http://localhost:8000/v1
# - Caches model weights under ./cache so subsequent starts are fast
# - Requires an NVIDIA runtime; CPU-only hosts need a different --device/--dtype
 
services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm-gemma4
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - ./cache:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command:
      - --model
      - google/gemma-4-9b-it
      - --served-model-name
      - gemma-4-9b
      - --dtype
      - bfloat16
      - --max-model-len
      - "8192"
      - --gpu-memory-utilization
      - "0.90"
      - --api-key
      - ${VLLM_API_KEY}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 15s
      timeout: 5s
      retries: 10
      start_period: 120s

# .env
HF_TOKEN=hf_YOUR_HUGGINGFACE_TOKEN
VLLM_API_KEY=sk-local-dev-ANY_RANDOM_STRING

Bring it up with:

docker compose up -d
docker compose logs -f vllm   # wait for "Application startup complete"

What to expect: around two minutes to pull roughly 18 GB of weights the first time, then a healthy /health response and a working /v1/chat/completions:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-dev-ANY_RANDOM_STRING" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-9b",
    "messages": [{"role": "user", "content": "Introduce yourself in one sentence."}],
    "max_tokens": 64
  }'

One detail that is easy to miss: always set --api-key. Without it, vLLM serves requests unauthenticated, which is fine on an isolated host but becomes a liability the moment you expose the port to the rest of the network. Even in development, it's worth getting into the habit.

I tend to keep my .env and docker-compose.yml checked into a private repository, with environment-specific overrides layered on top via docker-compose.override.yml. That split makes it obvious what is shared configuration and what is environment-specific, and it lets me share the base setup with a teammate without also handing over my Hugging Face token. If you plan to share this kind of setup across a team, investing five minutes in this structure now saves an afternoon of cleanup later.

Pointing Antigravity at the vLLM endpoint

Because vLLM exposes an OpenAI-compatible surface, Antigravity's configuration is straightforward. In Model Providers, add a Custom Provider with:

• Provider Name: local-vllm-gemma4
• Base URL: http://localhost:8000/v1 (swap in your domain for remote hosting)
• API Key: whatever you set for VLLM_API_KEY in .env
• Model ID: gemma-4-9b

If everything lines up, the Antigravity model picker will show gemma-4-9b, and chats and agents will route through your own hardware. The mistake I made the first time was forgetting the /v1 suffix on Base URL. OpenAI-compatible clients concatenate base_url + /chat/completions, and without the suffix you get a confusing 404.

A small verification script you can run outside Antigravity pays dividends whenever something goes wrong:

# verify_vllm.py
# Mirror the Antigravity configuration from Python to isolate where a failure lives:
# - reachability
# - authentication
# - served model name
import os
from openai import OpenAI
from openai import APIConnectionError, AuthenticationError, NotFoundError
 
client = OpenAI(
    base_url=os.environ.get("VLLM_BASE_URL", "http://localhost:8000/v1"),
    api_key=os.environ["VLLM_API_KEY"],
)
 
def verify() -> None:
    try:
        resp = client.chat.completions.create(
            model="gemma-4-9b",
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=8,
            timeout=30,
        )
        print("[OK] response:", resp.choices[0].message.content)
    except APIConnectionError as e:
        # Most commonly: the container is not up, or the URL is wrong.
        print("[NG] connection failure:", e)
    except AuthenticationError:
        # API key in .env and the client differ.
        print("[NG] auth failure — check VLLM_API_KEY vs. --api-key")
    except NotFoundError:
        # served-model-name and client-side model do not match.
        print("[NG] model not found — check --served-model-name")
 
if __name__ == "__main__":
    verify()

Keeping this in the repository turned out to be the single most useful operational tool I added. When Antigravity fails to connect, a three-line log tells me whether it is the frontend configuration, the vLLM process, or the network path that needs attention.

Measuring quantization trade-offs on your own workload

Once the basic path is working, the obvious next question is what happens if you quantize the model: how much faster does it get, and how much quality do you lose? The honest answer is that the only numbers that matter are the ones you measure on your own hardware with prompts that look like your real workload.

Here is a minimal async benchmark that fires a fixed number of requests at a controlled concurrency level and reports throughput and latency percentiles. Run it against bfloat16, W8A8, and AWQ 4-bit builds in turn and compare.

# vllm_benchmark.py
# Drive a configurable number of concurrent requests at vLLM and report
# throughput (tokens/sec) and latency percentiles for the run.
# - Set CONCURRENCY to your expected peak
# - If you see OOM, reduce --gpu-memory-utilization or --max-model-len on the server
import os
import time
import asyncio
import statistics
from openai import AsyncOpenAI
 
PROMPTS = [
    "Explain PagedAttention in three sentences.",
    "Write a haiku about self-hosted LLMs.",
    "List 5 reasons to use vLLM.",
    "Summarize why continuous batching matters.",
    "Translate the phrase 'production operation' into natural Japanese.",
]
CONCURRENCY = int(os.environ.get("CONCURRENCY", "8"))
TOTAL = int(os.environ.get("TOTAL", "80"))
 
client = AsyncOpenAI(
    base_url=os.environ["VLLM_BASE_URL"],
    api_key=os.environ["VLLM_API_KEY"],
)
 
async def one_call(prompt: str) -> tuple[float, int]:
    start = time.perf_counter()
    resp = await client.chat.completions.create(
        model="gemma-4-9b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=128,
        temperature=0.7,
    )
    elapsed = time.perf_counter() - start
    tokens = resp.usage.completion_tokens if resp.usage else 0
    return elapsed, tokens
 
async def main() -> None:
    semaphore = asyncio.Semaphore(CONCURRENCY)
 
    async def guarded(i: int):
        async with semaphore:
            return await one_call(PROMPTS[i % len(PROMPTS)])
 
    t0 = time.perf_counter()
    results = await asyncio.gather(*[guarded(i) for i in range(TOTAL)])
    wall = time.perf_counter() - t0
 
    latencies = [r[0] for r in results]
    total_tokens = sum(r[1] for r in results)
    p50 = statistics.median(latencies)
    p95 = statistics.quantiles(latencies, n=20)[18]
 
    print(f"concurrency: {CONCURRENCY} / total: {TOTAL}")
    print(f"wall: {wall:.2f}s  throughput: {total_tokens/wall:.1f} tok/s")
    print(f"latency p50: {p50:.2f}s  p95: {p95:.2f}s")
 
if __name__ == "__main__":
    asyncio.run(main())

Representative output on an RTX 4090 24 GB with Gemma 4 9B at CONCURRENCY=8, TOTAL=80:

concurrency: 8 / total: 80
wall: 22.14s  throughput: 362.4 tok/s
latency p50: 1.94s  p95: 3.71s

From my runs comparing bfloat16, W8A8, and AWQ 4-bit on Gemma 4 9B, the patterns I saw were:

• bfloat16 — best quality, peak VRAM around 19 GB (barely fits on a 24 GB card), throughput as the reference line.
• W8A8 — throughput roughly on par or marginally higher, VRAM drops to about 60% of bfloat16, quality loss visible mainly in the fluency of non-English output.
• AWQ 4-bit — throughput 1.3–1.5x higher, VRAM drops to about 7 GB, but on my summarization workload I saw small facts getting dropped. I did not ship it to production.

These are conclusions for my workload, not universal ones. That is why this benchmark matters: quantization is not a free win, and the right tier depends on what you are willing to lose in exchange for what you gain. If you skip this step, you will not notice the drop until you ship it and someone complains two weeks later about wrong answers.

What I find helpful here is to define one or two "golden prompts" that represent the failure modes you care most about. For me that has meant a long structured summarization prompt (where I want to see whether facts get dropped) and a pair-programming prompt (where I want to see whether the code it writes is subtly broken). Running those same two prompts against every quantization tier and eyeballing the outputs takes maybe twenty minutes and tells you more than any automated benchmark score.

The other piece of nuance is that quantization choices interact with context length. AWQ 4-bit frees up a lot of VRAM, which you can either bank as safety margin or spend on longer --max-model-len. If your workload includes long documents, it can be worth choosing a slightly lower quantization tier purely because it lets you stretch the context window further. Production decisions like this are where the benchmark numbers stop being the point and start being the tool. Put differently, the benchmark's job is not to tell you which tier to pick but to rule out the tiers that are clearly unacceptable. A tier that drops 15% of the relevant facts from your summarization output is out, no matter how fast it is. Among the tiers that pass that bar, the choice comes down to which combination of VRAM freedom, throughput, and latency aligns with the rest of your stack.

To serve a quantized model, extend the command block in docker-compose.yml:

    command:
      - --model
      - google/gemma-4-9b-it
      - --quantization
      - awq              # choose from "awq", "fp8", "bitsandbytes", "gptq", and others
      - --dtype
      - auto

For the 27B model, plain bfloat16 essentially requires an 80 GB-class GPU. Individuals running 27B usually rely on community AWQ 4-bit builds that fit on a 24 GB card. If you do go this route, I strongly recommend comparing 27B AWQ 4-bit against 9B bfloat16 on your own prompts before committing to it — the quality delta is often smaller than intuition suggests.

What the numbers say about cost

The economics of self-hosting deserve a direct look, because it is easy to spend a weekend building a beautiful inference stack and then discover it is more expensive than the managed API you were avoiding.

On a rough basis, a consumer-grade 24 GB GPU in a cloud environment costs somewhere between 0.50 and 1.00 USD per hour, depending on provider and commitment. Running Gemma 4 9B at the throughput numbers from the benchmark earlier, that translates to roughly 0.001 to 0.003 USD per 1,000 output tokens, assuming the GPU is kept saturated. A managed API for a comparable model, for reference, runs in the neighborhood of 0.10 to 0.30 USD per 1,000 output tokens at the time of writing.

The catch is the word "saturated." Those numbers only hold when the GPU is busy most of the time. A GPU that is idle 80% of the day flips the economics sharply against self-hosting. The threshold that tends to matter for individual developers is around 20–30% average utilization — below that, a managed API is cheaper and simpler. Above it, the self-hosted route starts to pay off and keeps pulling further ahead as utilization rises.

I have shipped projects on both sides of this line. The decision almost never comes down to a single factor. Privacy constraints, latency sensitivity, and the cost of the engineering time to maintain the stack all matter at least as much as the per-token dollar figure. What the numbers do clarify is whether you are in the ballpark where the question is worth asking in the first place.

Taking the stack from "works" to "stays up"

The docker-compose above gets you a functioning server. It does not protect you against a single client with an accidentally huge prompt locking up the whole system. A small FastAPI proxy in front of vLLM addresses most of the operational pain with modest code.

# proxy.py
# Antigravity -> proxy -> vLLM, centralizing rate limits and timeouts.
# Notes:
# - Keep vLLM on an internal address; do not expose it directly.
# - Only Authorization values in CLIENT_KEYS are allowed through.
# - Always respond with Retry-After on 429 so Antigravity's retry logic stays healthy.
import os
import time
import asyncio
from collections import defaultdict, deque
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import StreamingResponse
import httpx
 
VLLM_UPSTREAM = os.environ["VLLM_UPSTREAM"]   # e.g. http://vllm:8000
VLLM_API_KEY = os.environ["VLLM_API_KEY"]
CLIENT_KEYS = set(os.environ["CLIENT_KEYS"].split(","))
RATE_PER_MIN = int(os.environ.get("RATE_PER_MIN", "60"))
MAX_PROMPT_CHARS = int(os.environ.get("MAX_PROMPT_CHARS", "32000"))
UPSTREAM_TIMEOUT = float(os.environ.get("UPSTREAM_TIMEOUT", "60"))
 
app = FastAPI()
bucket: dict[str, deque[float]] = defaultdict(deque)
lock = asyncio.Lock()
 
def _check_and_record(key: str) -> None:
    now = time.time()
    window = bucket[key]
    while window and now - window[0] > 60:
        window.popleft()
    if len(window) >= RATE_PER_MIN:
        retry = int(60 - (now - window[0]))
        raise HTTPException(429, "rate limit", headers={"Retry-After": str(retry)})
    window.append(now)
 
async def _guard(request: Request) -> str:
    auth = request.headers.get("Authorization", "")
    if not auth.startswith("Bearer ") or auth[7:] not in CLIENT_KEYS:
        raise HTTPException(401, "bad client key")
    key = auth[7:]
    body = await request.body()
    if len(body) > MAX_PROMPT_CHARS * 4:
        raise HTTPException(413, "prompt too large")
    async with lock:
        _check_and_record(key)
    return key
 
@app.post("/v1/chat/completions")
async def proxy(request: Request):
    await _guard(request)
    headers = {
        "Authorization": f"Bearer {VLLM_API_KEY}",
        "Content-Type": "application/json",
    }
    body = await request.body()
    async with httpx.AsyncClient(timeout=UPSTREAM_TIMEOUT) as client:
        try:
            r = await client.post(
                f"{VLLM_UPSTREAM}/v1/chat/completions",
                content=body,
                headers=headers,
            )
        except httpx.ReadTimeout:
            raise HTTPException(504, "upstream timeout")
    return StreamingResponse(
        iter([r.content]),
        media_type=r.headers.get("content-type", "application/json"),
        status_code=r.status_code,
    )
 
@app.get("/health")
async def health():
    async with httpx.AsyncClient(timeout=5) as client:
        r = await client.get(f"{VLLM_UPSTREAM}/health")
        return {"vllm": r.status_code}

Register the proxy URL in Antigravity and lock down vLLM to internal access only. Two benefits appear immediately. First, you can issue a different CLIENT_KEY per person or per agent, which makes it obvious who is saturating the queue when 429s start showing up. Second, retries become well-behaved because Retry-After tells clients exactly when to come back.

On health checks: a 200 response from /health only proves the process is running. To confirm that the model has finished loading and can actually produce tokens, hit /v1/models and verify the served model ID is listed. For production monitoring, add a synthetic probe that sends a trivial prompt at an interval and asserts the response content is non-empty. A liveness check that never touches the model is the most common way a silent inference outage goes undetected.

One pattern I have adopted for the proxy layer is request tagging. Every incoming request gets a X-Request-Source header that identifies which Antigravity workspace or which agent generated it. Those tags show up in logs and in rate-limit decisions, which means I can answer "why did we hit 429 at 3pm yesterday" by filtering to the tag that was responsible. The proxy is a convenient place to enforce this consistently, and Antigravity lets you attach custom headers per custom provider configuration if you want to segment by workspace.

Where things actually break, and what to do about it

These are the issues I and other individual developers have actually tripped over while operating vLLM. Each has a matching fix.

Pitfall 1: CUDA out-of-memory appears only after dozens of requests

The start is clean, and the crash arrives much later. The culprit is usually an over-generous --max-model-len combined with an aggressive --gpu-memory-utilization. Size --max-model-len to about 1.5x the longest prompt you will realistically serve, and leave the GPU memory target around 0.85–0.88. Those two changes solve this specific class of issue for most individual deployments. The underlying reason is that vLLM pre-allocates KV cache slots based on the maximum model length you specify. An overly large --max-model-len reserves so many slots that there is no headroom for the continuous-batching scheduler to slot in new requests once the buffer fills, and the server either drops requests or OOMs. Treating --max-model-len as "the longest prompt I will actually accept" rather than "the biggest number the model could theoretically handle" is the mental shift that makes everything downstream easier to reason about.

# Too eager — KV cache has no headroom
--max-model-len 32768
--gpu-memory-utilization 0.95
 
# Better — sized to realistic prompts, leaves slack for request bursts
--max-model-len 8192
--gpu-memory-utilization 0.88

Pitfall 2: Chat templates go sideways and outputs include special tokens

Gemma's tokenizer includes non-trivial role delimiters (<start_of_turn>, <end_of_turn>). If vLLM ends up applying the wrong chat template, you will see those tokens leak into generations. The reliable path is to let vLLM read tokenizer_config.json from the model repo untouched and to avoid passing a custom --chat-template unless you have validated the Jinja2 thoroughly.

Pitfall 3: Requests from Antigravity come back only in English

When a system message is missing or too generic, Gemma 4 will often default to English even for Japanese prompts. In Antigravity's custom provider settings, add a default system prompt like "Respond in Japanese." and it will apply consistently for everyone on the team.

Pitfall 4: Streaming responses truncate behind a reverse proxy

nginx and Cloudflare are happy to buffer SSE responses into silence. On nginx, add proxy_buffering off; and proxy_read_timeout 300s; to the /v1/chat/completions location. On a CDN like Cloudflare, bypass caching for that path and send Cache-Control: no-cache from clients.

Pitfall 5: Rolling restarts briefly route traffic to nothing

During a rolling restart, there is a window where the old container's /health has flipped to failing but the new container's /health has not yet flipped to passing. Push the docker-compose start_period up to 120–180 seconds and relax your load balancer's unhealthy_threshold so the brief dip does not snowball into half your fleet being marked bad.

Routing requests to the right LoRA adapter

As you use Gemma 4 more seriously, specialized adapters inevitably appear — a code-review LoRA, a summarization LoRA, maybe a domain-specific rewriter. vLLM supports loading multiple LoRA adapters into a single server and routing requests to them by name.

Extend docker-compose.yml to enable LoRA and register the adapters you want resident:

    command:
      - --model
      - google/gemma-4-9b-it
      - --enable-lora
      - --max-loras
      - "4"
      - --max-lora-rank
      - "32"
      - --lora-modules
      - review=/adapters/code-review-lora
      - summary=/adapters/summary-lora
    volumes:
      - ./cache:/root/.cache/huggingface
      - ./adapters:/adapters

The key change on the client side is that the model field of a request now selects the adapter. Leaving it as the base model (gemma-4-9b) uses the original weights; naming an adapter (review or summary) applies that LoRA.

# lora_switch.py
# Same base model, different personalities, selected per request.
from openai import OpenAI
import os
 
client = OpenAI(
    base_url=os.environ["VLLM_BASE_URL"],
    api_key=os.environ["VLLM_API_KEY"],
)
 
def review(code: str) -> str:
    resp = client.chat.completions.create(
        model="review",  # routes to code-review-lora
        messages=[
            {"role": "system", "content": "You are a rigorous but polite code reviewer."},
            {"role": "user", "content": f"Review the following code:\n\n{code}"},
        ],
        max_tokens=512,
        temperature=0.2,
    )
    return resp.choices[0].message.content or ""
 
def summary(text: str) -> str:
    resp = client.chat.completions.create(
        model="summary",
        messages=[{"role": "user", "content": f"Summarize in three bullets:\n{text}"}],
        max_tokens=256,
        temperature=0.4,
    )
    return resp.choices[0].message.content or ""
 
if __name__ == "__main__":
    print(review("def add(a, b): return a - b"))

A small operational note: set --max-loras to exactly the number of adapters you plan to keep warm. Exceeding that number forces load/unload cycles on each request and causes latency to spike. All adapters must share the same base model — you cannot mix 9B and 27B adapters in one container. If you need to span multiple base models, run them as separate services.

In practice, my LoRA routing strategy has evolved into a loose rule: one adapter per coherent task, not one per prompt template. That is, a single "code-review" adapter trained on plenty of review examples beats three hyper-specialized adapters for "Python review", "TypeScript review", and "Go review" — the generalization across languages tends to make the single adapter stronger, and the latency savings from not swapping are real. The exception is when the task itself really does have two different distributions (a writing coach and a code reviewer, for instance) — those should be separate adapters.

Observability that catches silent failures

A vLLM process that is "up" is not the same as a vLLM process that is serving well. The failure modes I have most often seen in the wild are not outright crashes — they are slow drift. Token generation gets subtly slower week over week as the model cache fills up. A small memory leak in a sidecar quietly steals GPU memory until a specific prompt length starts OOMing.

The minimum observability kit that has actually helped me:

• Throughput (tokens/sec) and p95 latency, scraped from a synthetic probe that sends the same trivial prompt every minute. This is the best single signal that the server is healthy under a specific, controlled load.
• Queue depth, which vLLM exposes via its metrics endpoint. If the queue starts sustaining nonzero values, you are under-provisioned for your current traffic and the next step is either horizontal scaling or tightening request limits upstream.
• Success rate of the synthetic probe, as a percentage over 1 hour. Sudden dips are usually caused by upstream timeouts or proxy-level misconfigurations, not vLLM itself.

None of these require expensive infrastructure. A Prometheus exporter and a three-panel Grafana dashboard will get you most of the value. The point is to build the habit of looking at those three numbers before shipping, so you can see trouble coming instead of explaining it after it arrives.

The one step I would take first

Even after all of this, trying everything at once is the wrong strategy. What I would actually do on day one is the smallest possible thing: boot the minimum docker-compose from earlier on a GPU-equipped host, register it as a custom provider in Antigravity, and send one or two prompts through. That exercise alone makes the difference from Ollama obvious, especially under concurrent load. The reason I recommend it over jumping straight to the proxy or the LoRA configuration is that every downstream decision becomes easier once you have a single concrete data point from your own environment. Is the throughput acceptable? Do the timeouts feel right? Are there warnings in the logs that need investigation? You cannot answer any of these questions in the abstract, and the quickest way to answer them concretely is to get a minimum viable server in front of Antigravity and poke at it.

Once the single-host setup is real, running the benchmark on your own prompts — and choosing a quantization tier from actual numbers rather than intuition — is the next honest step. The rest of the stack builds from there, and you end up with an inference layer you can explain, not just one that happens to work.

One last thought about when to stop self-hosting. I do not believe that every team benefits from running their own inference. If your team is under five people, your workload is bursty, and your primary constraint is engineering time rather than dollars, a managed API is almost certainly the right call. Self-hosting makes sense when either the privacy requirements of your data preclude a third-party API, or when your utilization is high enough that a GPU running 12 hours a day is cheaper than per-token pricing. As an indie developer covering this on my own budget, I found that most individual developers sit somewhere between those two poles, and the honest answer is that a hybrid approach — a managed API for ad-hoc calls, a self-hosted vLLM for your heaviest batch workloads — often wins on both cost and latency. Antigravity's custom provider setting makes that kind of routing easy to experiment with: you can toggle between the two backends at the model picker level and compare real-world performance side by side before making a commitment.

The point of this whole exercise is not to build a perfect inference platform on your first try. It is to build one you understand well enough to change later. Self-hosted infrastructure only pays off when you can evolve it as your needs evolve, and the shape of this guide — start small, measure, add constraints deliberately — is the shape that makes evolution possible. Every piece I recommended, from the minimum docker-compose to the FastAPI proxy to the LoRA routing, exists because I saw a specific failure mode in production and wanted to avoid hitting it again. Your failure modes will be different. The posture of "small and measured, then add on" is the part worth copying.