Running Multiple Gemma 4 LoRAs in Production — A Practical Guide to Merging and Dynamic Adapter Switching

Four ways to merge LoRAs — and the order to try them in

If you pick Strategy A, the choice of merge algorithm quietly decides the quality ceiling of your final model. Four methods dominate in 2026:

• Weighted (linear) merge. Classic. new_W = α * W_A + β * W_B. Works beautifully when the source LoRAs barely interfere with each other, which tends to be true when tasks are similar.
• TIES (Trim, Elect Sign, Merge). Trims near-zero parameters, resolves sign conflicts by majority vote, then averages. This is my default for three or more LoRAs — the quality floor is noticeably higher than naive weighted merge.
• DARE (Drop And REscale). Randomly drops parameters before merging, then rescales. Combined with TIES (DARE-TIES), it handles five-plus adapters gracefully.
• SLERP (Spherical Linear Interpolation). Interpolates between two models on a sphere. Great for style blending — formal voice × casual voice — but doesn't extend past two sources cleanly.

The decision tree is simpler than the method names suggest. Two sources: try Weighted, try SLERP. Three or more: go TIES. Five or more: go DARE-TIES. Rather than picking one blind, mergekit lets you try all four in a few minutes, so I routinely benchmark several candidates before committing.

Setting up the environment with Antigravity alongside

Time to run code. My setup is either Ubuntu 22.04 with CUDA 12.4 and a single A100 40GB, or an M3 Max 64GB MacBook. I've written the examples to work on both.

When I ask Antigravity's agent "set up peft and mergekit for Gemma 4 merging, and give me a minimal sanity-check script," it produces a full scaffold. Always sanity-check the versions it suggests — the peft and transformers APIs break on a near-monthly cadence, and a snippet from a blog post two months ago often no longer runs.

# requirements.txt
# torch>=2.3.0
# transformers>=4.44.0
# peft>=0.13.0
# mergekit>=0.0.5
# accelerate>=0.34.0
# safetensors>=0.4.5
 
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
 
# Log in to HuggingFace (required to fetch Gemma 4 weights)
huggingface-cli login
 
# Quick sanity check: Gemma 4 base loads and produces output
python - <<'PY'
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
 
model_id = "google/gemma-4-2b-it"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
prompt = "Summarize in one sentence: Antigravity is Google's new IDE built around Gemini 3 Pro."
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
PY
# Expected: a one-sentence summary like "Antigravity is Google's new IDE."

The subtle trap here is trusting Antigravity's version pins blindly. In my experience, about one in three suggestions needs a small bump or downgrade to match the exact CUDA/driver combination on the host. Treating the scaffold as a starting point rather than a finished script has saved me hours of debugging obscure CUDA errors.

Implementation 1: Weighted merge for a summary × translation dual-task model

The smallest useful example locks in the mental model for everything that follows. Here I take two trained adapters — lora_summary and lora_translate — and merge them at a 60:40 ratio.

# merge_weighted.py
# Purpose: Merge a summary LoRA and a translation LoRA at a weighted ratio,
#          then save as a single HF-format model so vLLM can serve it directly.
 
from pathlib import Path
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
 
BASE = "google/gemma-4-2b-it"
LORA_A = "./artifacts/lora_summary"
LORA_B = "./artifacts/lora_translate"
OUT_DIR = Path("./artifacts/merged_weighted_6_4")
WEIGHTS = {"summary": 0.6, "translate": 0.4}
 
def merge_weighted():
    tok = AutoTokenizer.from_pretrained(BASE)
    base = AutoModelForCausalLM.from_pretrained(
        BASE, torch_dtype=torch.bfloat16, device_map="auto"
    )
 
    # Load adapters with stable names
    model = PeftModel.from_pretrained(base, LORA_A, adapter_name="summary")
    model.load_adapter(LORA_B, adapter_name="translate")
 
    # Linear combination. API became stable in peft>=0.13.
    model.add_weighted_adapter(
        adapters=list(WEIGHTS.keys()),
        weights=list(WEIGHTS.values()),
        adapter_name="merged",
        combination_type="linear",
    )
    model.set_adapter("merged")
 
    # Fold the merged adapter into base weights so no adapter is needed at serve time
    merged = model.merge_and_unload()
    OUT_DIR.mkdir(parents=True, exist_ok=True)
    merged.save_pretrained(OUT_DIR, safe_serialization=True)
    tok.save_pretrained(OUT_DIR)
    print(f"Saved merged model to {OUT_DIR}")
 
if __name__ == "__main__":
    merge_weighted()

Once merge_and_unload() has run, the output is indistinguishable from a plain Gemma4ForCausalLM. A one-liner like vllm serve ./artifacts/merged_weighted_6_4 --dtype bfloat16 turns it into a production endpoint. That simplicity is Strategy A's single biggest virtue.

Why pick Weighted here? Honestly, because there are only two sources. Interference between two adapters is typically small enough that a linear combination holds up. The moment I add a third, I move to TIES.

One counterintuitive lesson: the weights aren't about task importance. They're about compensating for the source LoRAs' capacity differences. A summary LoRA with rank 16 and a translation LoRA with rank 32 need different coefficients — the higher-rank one tends to dominate linearly, so I lean the weights slightly against it. Running three candidates (0.5:0.5, 0.6:0.4, 0.7:0.3) with the same eval set is usually enough to see the trend.

Implementation 2: TIES merge for four LoRAs without catastrophic interference

The moment you add a third LoRA, weighted merges start visibly degrading. The reason is plain: LoRAs trained for different tasks sometimes encode parameters with opposite signs, and a linear average can cancel them both.

TIES resolves this with a concrete three-step ritual:

• Trim — drop the parameters with the smallest absolute values in each LoRA (the ones the training barely moved).
• Elect Sign — for each parameter, decide the sign by majority vote across sources.
• Merge — average only the parameters that agree with the elected sign.

Put plainly: it keeps the "core" of each adapter and only averages the vectors that point the same way. Even with four or five LoRAs, quality degrades gracefully rather than cliff-falling.

mergekit is the pragmatic tool. I prefer its YAML form — declarative configuration is easier to diff, and diffs are what you need during benchmarking.

# merge_ties.yaml
# Purpose: Merge four LoRAs via TIES.
# Run: mergekit-yaml merge_ties.yaml ./artifacts/merged_ties_4 --cuda
models:
  - model: google/gemma-4-2b-it
    # Base model — differences are computed relative to this
  - model: ./artifacts/lora_summary
    parameters:
      density: 0.7   # keep the top 70% most-important parameters
      weight: 0.3
  - model: ./artifacts/lora_translate
    parameters:
      density: 0.7
      weight: 0.3
  - model: ./artifacts/lora_code_review
    parameters:
      density: 0.6
      weight: 0.2
  - model: ./artifacts/lora_sql
    parameters:
      density: 0.6
      weight: 0.2
 
merge_method: ties
base_model: google/gemma-4-2b-it
parameters:
  int8_mask: true
  normalize: true
dtype: bfloat16

density is the trim threshold. Set it to 1.0 and TIES collapses back to a weighted merge; lower it and you keep only the "core" of each adapter. Tasks that are semantically far apart want lower density; tasks close to each other tolerate higher density. The weight values conventionally sum to 1.0, but that's a habit rather than a constraint.

One piece of advice I wish I'd internalized sooner: do not expect the merged model to match each individual LoRA's benchmark. Accept a 5–10% drop on single-task metrics as the price of running three tasks on one model. If you measure without that acceptance, you'll tune forever.

Implementation 3: A dynamic-LoRA inference server for one shared base

Strategy B takes more code but produces the most stable quality when tasks are independent. Here's a FastAPI server that keeps Gemma 4 loaded once and swaps adapters per request.

# serve_dynamic.py
# Purpose: Keep Gemma 4 base loaded. Per request, switch LoRA and generate.
# Note: Adapter switching is exclusive. Concurrent requests are serialized via asyncio.Lock.
#       For true parallelism, you want Strategy C (multi-adapter).
 
import asyncio
from contextlib import asynccontextmanager
 
import torch
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
 
BASE = "google/gemma-4-2b-it"
ADAPTERS = {
    "summary":     "./artifacts/lora_summary",
    "translate":   "./artifacts/lora_translate",
    "code_review": "./artifacts/lora_code_review",
    "sql":         "./artifacts/lora_sql",
}
 
state = {"model": None, "tok": None, "current": None, "lock": asyncio.Lock()}
 
@asynccontextmanager
async def lifespan(app: FastAPI):
    tok = AutoTokenizer.from_pretrained(BASE)
    base = AutoModelForCausalLM.from_pretrained(
        BASE, torch_dtype=torch.bfloat16, device_map="auto"
    )
    # Loading the first adapter promotes `base` to a PeftModel
    first_name, first_path = next(iter(ADAPTERS.items()))
    model = PeftModel.from_pretrained(base, first_path, adapter_name=first_name)
    for name, path in list(ADAPTERS.items())[1:]:
        model.load_adapter(path, adapter_name=name)
    model.set_adapter(first_name)
    state.update(model=model, tok=tok, current=first_name)
    print(f"Loaded {len(ADAPTERS)} adapters; active={first_name}")
    yield
 
app = FastAPI(lifespan=lifespan)
 
class Req(BaseModel):
    task: str
    prompt: str
    max_new_tokens: int = 256
 
@app.post("/generate")
async def generate(req: Req):
    if req.task not in ADAPTERS:
        raise HTTPException(400, f"unknown task: {req.task}")
    async with state["lock"]:
        if state["current"] != req.task:
            # ~50ms switch cost. Bursty switching kills throughput — see note below.
            state["model"].set_adapter(req.task)
            state["current"] = req.task
        inputs = state["tok"](req.prompt, return_tensors="pt").to(state["model"].device)
        try:
            out = state["model"].generate(
                **inputs, max_new_tokens=req.max_new_tokens, do_sample=False
            )
        except torch.cuda.OutOfMemoryError as e:
            raise HTTPException(503, "GPU OOM — reduce max_new_tokens") from e
        text = state["tok"].decode(out[0], skip_special_tokens=True)
        return {"task": req.task, "text": text}
 
# Example client call:
# curl -X POST http://localhost:8000/generate \
#   -H "Content-Type: application/json" \
#   -d '{"task":"summary","prompt":"..."}'
# Response: {"task":"summary","text":"..."}

The switch itself finishes in about 50ms, but the lock is where the real subtlety sits. Interleaved requests for different tasks cause constant adapter flipping and throughput collapses. The production pattern is to bucket requests by task, batch them within a short window, and only call set_adapter between batches.

When I asked Antigravity to add batching, it produced a clean asyncio.Queue scaffold with a time-window aggregator within a few minutes. But a scaffold is a scaffold — you still need to run a real load test. I start with a 30-second spike from k6 or locust to find the adapter-flip rate that breaks throughput.

Benchmarks — real numbers from my own production workload

Theory only gets you so far, so here are numbers from a small SaaS I run. Setup: one A100 40GB, Gemma 4 2B base, four LoRAs (summary, translation, code review, SQL generation) all at rank 16. Median prompt length 640 tokens, average output 180 tokens, 18,000 mixed-task requests over 24 hours.

Rough comparison across the three strategies:

• Strategy A (TIES merge, single model): 42 req/s throughput, 780ms P50, 1,450ms P99, average 7% drop on single-task evals, 14 GB GPU memory.
• Strategy B (dynamic swap, 4 LoRAs hot-loaded): 28 req/s throughput, 920ms P50, 2,300ms P99, no single-task eval drop, 16 GB GPU memory — but on days when the task changes every request, P99 spikes above 3,500ms.
• Strategy C (S-LoRA multi-adapter parallelism): 38 req/s throughput, 840ms P50, 1,700ms P99, no single-task eval drop, 22 GB GPU memory, roughly 3x the implementation effort.

My conclusion from these numbers: unless your tasks are genuinely independent and you can't afford even a 1% single-task accuracy drop, TIES merging (Strategy A) is the cost-performance winner. Strategy B shines in chat-heavy workloads where task transitions are sparse. Strategy C pays off cleanly when premium-tier differentiation ("the Pro plan is noticeably more accurate") is worth paid engineering time.

The numbers are sensitive to traffic pattern and hardware. Run the measurement on your own workload. Building a repeatable benchmark harness once pays off every time you revisit the merge question later. When I ask Antigravity to "write a benchmark that compares TIES and dynamic-switch on these 100 Golden Prompts," I get a usable first script immediately — not perfect, but enough to start iterating.

A brief look at Strategy C — when multi-adapter parallelism is actually worth it

I'm deliberately keeping Strategy C brief, but the decision criteria are worth stating clearly. Consider it only when all three of these hold:

• Task boundaries shift within a single response (a reply that mixes summarization and translation in one turn).
• Any single-task accuracy drop has direct business impact (medical, legal, or financial domains where the tolerance is effectively zero).
• You have the appetite to run a custom inference stack — vLLM alone won't get you there.

The reference implementations are projects like S-LoRA and Punica that apply multiple LoRAs at the CUDA-kernel level simultaneously. Memory use rises, but batching efficiency and resilience to shifting task mixes are in another league.

For solo developers and small teams, this path is usually too heavy. I tried it once and spent three weekends debugging CUDA kernels before concluding that Strategy A plus a monthly TIES retuning was the right fit for my workload. Not chasing the bleeding edge is itself a respectable operational decision when continuity matters.

Five pitfalls I actually hit in production

LoRA plumbing hides a lot of sharp edges. These five cost me anywhere from an afternoon to a full day to diagnose:

• Forgetting set_adapter() before merge_and_unload(). If it's unclear which adapter is active when you unload, only the most recently loaded one gets folded in. The correct sequence is add_weighted_adapter → set_adapter("merged") → merge_and_unload(). I lost a full day the first time because my eval numbers looked "merged" but the merge had never actually happened.
• Leaving density unspecified on TIES. mergekit silently defaults to 1.0, which is identical to a weighted merge. The logs say TIES but the behavior is Weighted. Always set density explicitly.
• Merging LoRAs with mismatched rank. A rank-8 and a rank-64 LoRA don't combine cleanly — the larger one dominates. Matching rank at training time is ideal. If you can't, compensate via weight roughly inversely proportional to √rank. That heuristic has kept me out of the worst outcomes.
• Saving with torch_dtype=float16. Gemma 4 expects bfloat16 in many code paths. Saving in float16 leads to subtle underflow and broken outputs. I always set safe_serialization=True and torch_dtype=torch.bfloat16 explicitly at save time.
• Calling PeftModel.merge_and_unload() inside the dynamic-switching server. Once called, adapters can no longer be swapped. Never call it in your serve script. Put a comment and a runbook note to remind future you.

Observability and rollback — the parts that matter when it matters

Quality regressions often show up without any code change. The model hasn't moved; the traffic distribution has, and the LoRA's strong zones don't cover what's arriving now. Catching that early is not optional.

I track three signals at minimum:

• Per-task median response length and latency. Sudden shifts (abrupt shortening, runaway generation) are the clearest early sign of a broken adapter. Prometheus plus Grafana with 1-minute buckets will surface it by morning.
• Golden-prompt regression diffs. I keep 30–50 anchor prompts per task and run them after every release, scoring against the previous release with ROUGE-L and chrF++. If a threshold is crossed, an automated rollback triggers.
• User-side signals. Thumbs up/down, regenerate-button rate. These lead quantitative metrics more often than I expected.

Rollback must be close to instant. For merged models I swap a tagged safetensors file; for dynamic switching I swap a symlink to the adapter directory. Both paths complete inside ten seconds. The broader operational picture — dashboards, alerting topology, deployment gates — is covered in my Antigravity LLMOps production monitoring guide, which pairs well with this article.

What I'd ask you to try this week

If something here made you want to experiment, my one request: take two LoRAs you already have, and run exactly two Weighted merges — one at 0.5:0.5 and one at 0.7:0.3 — before the week ends. That single exercise locks in the intuition that everything else builds on.

TIES and dynamic switching are easier to absorb after you've done the smallest thing first. At least that was true for me. Finishing the first comparison on a Saturday afternoon, then spending the following week on benchmarks, turned out to be a surprisingly efficient learning cadence.

Production multi-LoRA is not a one-size solution. Pick what fits the shape of your product. If you land on a combination that works — or one that breaks in unexpected ways — I'd love to hear about it. For a broader view of running Gemma 4 in production with Antigravity, the Gemma 4 local-LLM production guide covers the surrounding infrastructure questions this article deliberately sets aside.