Gemma 4 Production Fine-tuning Complete Guide — Dataset Design, QLoRA Optimization, Evaluation Pipeline & Cloudflare Workers Deployment

"I finished the LoRA fine-tuning tutorial. The code ran. But the model quality isn't good enough for production."

If that sounds familiar, you're not alone. I ran into exactly this wall when building a custom Gemma 4 model — the tutorial code worked perfectly, but applying it to my actual use case gave inconsistent results. And when I tried to deploy it, the latency didn't meet production requirements.

This article bridges the gap between "a fine-tuning that technically runs" and "a model that creates real value in production." We'll cover dataset design patterns, Gemma 4-optimized QLoRA settings, evaluation pipelines, and deployment to Cloudflare Workers AI — all using Antigravity as our development environment throughout.

Three Walls That Block Production Fine-tuning

Before diving into code, it's worth understanding the root causes of why fine-tuning often fails to reach production quality.

Wall 1: Data quality. Official tutorials use toy datasets. When you switch to your own domain data, poor results usually come from data quality problems, not quantity. Noisy samples, inconsistent output formatting, and skewed domain distribution are the biggest culprits — and they're invisible until you know what to look for.

Wall 2: Hyperparameter combinations. With learning_rate, lora_rank, lora_alpha, batch_size, and more, the search space is enormous. The trap is "use whatever settings worked once and never change them." Gemma 4 has specific optimal ranges that differ from other models.

Wall 3: Evaluation-to-value disconnect. Perplexity goes down, but real outputs don't improve. Benchmark scores look great, but users don't notice a difference. The gap between metrics and business value is something you only discover post-deployment — unless you set up the right evaluation pipeline upfront.

Environment Setup with Antigravity

The main advantage of using Antigravity here is that you can let AI agents assist across the design → train → evaluate → deploy cycle without context-switching between tools.

# Recommended GPU: A100 80GB or H100 (Google Colab Pro+ / Vertex AI Workbench)
pip install transformers==4.47.0 \
            peft==0.14.0 \
            trl==0.13.0 \
            bitsandbytes==0.45.0 \
            datasets==3.2.0 \
            wandb
 
# Pin versions — silent version mismatches can degrade accuracy without errors
python -c "import transformers; print(transformers.__version__)"

Set up your project structure in Antigravity's terminal:

gemma4-finetune/
├── data/
│   ├── raw/          ← original data
│   ├── processed/    ← after quality checks
│   └── final/        ← ready for training
├── scripts/
│   ├── prepare_data.py
│   ├── train.py
│   └── evaluate.py
├── configs/
│   └── qlora_config.yaml
└── AGENTS.md         ← instructions for Antigravity agents

Your AGENTS.md guides Antigravity agents working on this project:

# Gemma 4 Fine-tuning Agent Instructions
 
## Role
Assist with data preprocessing, training config optimization, and evaluation result analysis.
 
## Key Constraints
- Always reference `configs/qlora_config.yaml` for quantization settings
- Warn when valid sample count drops below 1,000
- After each evaluation run, append results to summary.md
 
## Common Tasks
- Validate dataset: `python scripts/prepare_data.py --validate`
- Start training: `python scripts/train.py --config configs/qlora_config.yaml`
- Run evaluation: `python scripts/evaluate.py --model-dir output/`

Designing Production-Quality Datasets

This is the core of the article. A good dataset matters more than model architecture choices.

Gemma 4's Instruction Format

Gemma 4 uses <start_of_turn> / <end_of_turn> tokens for instruction tuning. Using the wrong format means your fine-tuning completes without errors but fails at inference time.

# prepare_data.py — Gemma 4 instruction format
 
def format_for_gemma4(sample: dict) -> str:
    """
    Convert a sample to Gemma 4 instruction tuning format.
    Input: {"instruction": "...", "output": "..."}
    Output: "<start_of_turn>user\n{instruction}<end_of_turn>\n<start_of_turn>model\n{output}<end_of_turn>"
    """
    instruction = sample.get("instruction", "").strip()
    output = sample.get("output", "").strip()
    
    if not instruction or not output:
        return None  # exclude empty samples
    
    formatted = (
        f"<start_of_turn>user\n{instruction}<end_of_turn>\n"
        f"<start_of_turn>model\n{output}<end_of_turn>"
    )
    return formatted
 
def validate_dataset(samples: list) -> dict:
    """
    Quantitatively assess dataset quality.
    Run this before every training run.
    """
    stats = {
        "total": len(samples),
        "empty_removed": 0,
        "too_short": 0,      # output < 20 chars
        "too_long": 0,       # approx tokens > 1024
        "duplicate": 0,
        "valid": 0,
    }
    
    seen_instructions = set()
    valid_samples = []
    
    for s in samples:
        formatted = format_for_gemma4(s)
        if formatted is None:
            stats["empty_removed"] += 1
            continue
        
        output_len = len(s.get("output", ""))
        if output_len < 20:
            stats["too_short"] += 1
            continue
        
        approx_tokens = len(formatted) // 3  # rough estimate; use tokenizer in production
        if approx_tokens > 1024:
            stats["too_long"] += 1
            continue
        
        # Exact dedup on first 100 chars of instruction
        instruction_key = s["instruction"].strip().lower()[:100]
        if instruction_key in seen_instructions:
            stats["duplicate"] += 1
            continue
        seen_instructions.add(instruction_key)
        
        valid_samples.append({"text": formatted})
        stats["valid"] += 1
    
    stats["valid_ratio"] = stats["valid"] / stats["total"] if stats["total"] > 0 else 0
    return stats, valid_samples
 
if __name__ == "__main__":
    import json
    
    with open("data/raw/samples.jsonl") as f:
        raw = [json.loads(line) for line in f]
    
    stats, valid = validate_dataset(raw)
    print(f"Dataset Validation Results:")
    print(f"  Total samples: {stats['total']}")
    print(f"  Empty removed: {stats['empty_removed']}")
    print(f"  Too short removed: {stats['too_short']}")
    print(f"  Too long removed: {stats['too_long']}")
    print(f"  Duplicates removed: {stats['duplicate']}")
    print(f"  Valid samples: {stats['valid']} ({stats['valid_ratio']:.1%})")
    
    if stats["valid"] < 500:
        print("⚠️  WARNING: Under 500 valid samples. Generalization may suffer.")
    
    with open("data/processed/train.json", "w") as f:
        json.dump(valid, f, ensure_ascii=False, indent=2)
    
    print(f"\n✅ Written to data/processed/train.json ({stats['valid']} samples)")

Using Antigravity Agents for Data Synthesis

In practice, having Antigravity agents assist with sample generation is far more efficient than manual creation. Here's a prompt pattern for generating customer support QA pairs:

# Prompt pattern passed via AGENTS.md
GENERATION_PROMPT = """
Generate 20 customer support QA pairs based on the product documentation below.
 
Constraints:
- Questions should use natural user language (avoid jargon)
- Answers should be 100-300 words
- No repeated question patterns
- Output as JSON: {"instruction": "...", "output": "..."}
 
Reference Documentation:
{document_content}
"""

One critical rule: never use LLM-generated data without human review. Training an LLM on LLM-generated data amplifies errors. Review at least 10% of synthesized samples manually before including them in training.

QLoRA Configuration — Gemma 4-Specific Optimal Values

Here are QLoRA settings that work well for Gemma 4, based on practical experimentation:

# configs/qlora_config.yaml
 
model:
  name: "google/gemma-4-4b-it"   # or 12b-it / 27b-it
  torch_dtype: "bfloat16"         # bfloat16 recommended over float16
 
quantization:
  load_in_4bit: true
  bnb_4bit_compute_dtype: "bfloat16"
  bnb_4bit_use_double_quant: true
  bnb_4bit_quant_type: "nf4"      # nf4 outperforms fp4 for Gemma 4
 
lora:
  r: 16                # 8-32 works well for Gemma 4; higher uses more memory
  lora_alpha: 32       # alpha = 2 * r is a good starting point
  lora_dropout: 0.05
  bias: "none"
  target_modules:
    - "q_proj"
    - "k_proj"
    - "v_proj"
    - "o_proj"
    - "gate_proj"
    - "up_proj"
    - "down_proj"
 
training:
  output_dir: "./output"
  num_train_epochs: 3
  per_device_train_batch_size: 2   # for A100 40GB
  gradient_accumulation_steps: 8   # effective batch size = 16
  learning_rate: 2.0e-4            # Gemma 4 sweet spot; explore 1e-4 to 5e-4
  warmup_ratio: 0.03
  lr_scheduler_type: "cosine"
  max_seq_length: 512
  logging_steps: 10
  save_strategy: "steps"
  save_steps: 100
  evaluation_strategy: "steps"
  eval_steps: 100
  load_best_model_at_end: true
  report_to: "wandb"

# train.py — production-quality training script
 
from transformers import (
    AutoTokenizer, AutoModelForCausalLM,
    BitsAndBytesConfig, TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset
import torch, yaml, sys
 
def setup_model_and_tokenizer(config: dict):
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=config["quantization"]["load_in_4bit"],
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=config["quantization"]["bnb_4bit_use_double_quant"],
        bnb_4bit_quant_type=config["quantization"]["bnb_4bit_quant_type"],
    )
    
    model = AutoModelForCausalLM.from_pretrained(
        config["model"]["name"],
        quantization_config=bnb_config,
        device_map="auto",
        torch_dtype=torch.bfloat16,
        trust_remote_code=False,
    )
    model = prepare_model_for_kbit_training(model)
    
    tokenizer = AutoTokenizer.from_pretrained(config["model"]["name"])
    tokenizer.padding_side = "right"  # required for Gemma 4
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
    
    return model, tokenizer
 
def main(config_path: str):
    with open(config_path) as f:
        config = yaml.safe_load(f)
    
    model, tokenizer = setup_model_and_tokenizer(config)
    
    lora_config = LoraConfig(
        r=config["lora"]["r"],
        lora_alpha=config["lora"]["lora_alpha"],
        lora_dropout=config["lora"]["lora_dropout"],
        bias=config["lora"]["bias"],
        target_modules=config["lora"]["target_modules"],
        task_type="CAUSAL_LM",
    )
    model = get_peft_model(model, lora_config)
    
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    total = sum(p.numel() for p in model.parameters())
    print(f"Trainable: {trainable:,} / {total:,} ({100 * trainable / total:.2f}%)")
    # For Gemma 4-4B, expect ~0.5-1%. Much higher → review target_modules
    
    dataset = load_dataset("json", data_files={
        "train": "data/processed/train.json",
        "validation": "data/processed/val.json",
    })
    
    training_args = TrainingArguments(
        output_dir=config["training"]["output_dir"],
        num_train_epochs=config["training"]["num_train_epochs"],
        per_device_train_batch_size=config["training"]["per_device_train_batch_size"],
        gradient_accumulation_steps=config["training"]["gradient_accumulation_steps"],
        learning_rate=config["training"]["learning_rate"],
        warmup_ratio=config["training"]["warmup_ratio"],
        lr_scheduler_type=config["training"]["lr_scheduler_type"],
        logging_steps=config["training"]["logging_steps"],
        save_strategy=config["training"]["save_strategy"],
        save_steps=config["training"]["save_steps"],
        evaluation_strategy=config["training"]["evaluation_strategy"],
        eval_steps=config["training"]["eval_steps"],
        load_best_model_at_end=config["training"]["load_best_model_at_end"],
        report_to=config["training"]["report_to"],
        bf16=True,
        gradient_checkpointing=True,
        optim="paged_adamw_32bit",
        dataloader_num_workers=2,
    )
    
    trainer = SFTTrainer(
        model=model, tokenizer=tokenizer,
        train_dataset=dataset["train"],
        eval_dataset=dataset["validation"],
        args=training_args,
        dataset_text_field="text",
        max_seq_length=config["training"]["max_seq_length"],
        packing=False,
    )
    
    trainer.train()
    model.save_pretrained(f"{config['training']['output_dir']}/lora_adapter")
    tokenizer.save_pretrained(f"{config['training']['output_dir']}/lora_adapter")
    print(f"✅ Adapter saved to {config['training']['output_dir']}/lora_adapter")
 
if __name__ == "__main__":
    config_path = sys.argv[1] if len(sys.argv) > 1 else "configs/qlora_config.yaml"
    main(config_path)

Building the Evaluation Pipeline

"Loss went down, so the model must be better" is a dangerous assumption. Always evaluate fine-tuned models from multiple angles.

# evaluate.py
 
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch, json
 
def load_finetuned_model(base_model_name: str, adapter_path: str):
    tokenizer = AutoTokenizer.from_pretrained(adapter_path)
    base_model = AutoModelForCausalLM.from_pretrained(
        base_model_name, torch_dtype=torch.bfloat16, device_map="auto"
    )
    model = PeftModel.from_pretrained(base_model, adapter_path)
    model = model.merge_and_unload()  # merge adapter for inference
    model.eval()
    return model, tokenizer
 
def generate_response(model, tokenizer, instruction: str, max_new_tokens: int = 256) -> str:
    prompt = (
        f"<start_of_turn>user\n{instruction}<end_of_turn>\n"
        f"<start_of_turn>model\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs, max_new_tokens=max_new_tokens,
            temperature=0.1, do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    ).strip()
 
if __name__ == "__main__":
    import sys
    model_dir = sys.argv[1] if len(sys.argv) > 1 else "output/lora_adapter"
    model, tokenizer = load_finetuned_model("google/gemma-4-4b-it", model_dir)
    
    with open("data/processed/test.json") as f:
        test_data = json.load(f)
    
    results = []
    for sample in test_data[:50]:
        generated = generate_response(model, tokenizer, sample["instruction"])
        keywords = [w for w in sample["output"].split() if len(w) > 3][:10]
        score = sum(1 for k in keywords if k in generated) / len(keywords) if keywords else 0
        results.append({"keyword_match": score, "generated": generated[:200]})
    
    avg = sum(r["keyword_match"] for r in results) / len(results)
    print(f"Keyword match score (avg): {avg:.2%}")
    
    if avg >= 0.7:
        print("✅ Ready for deployment")
    elif avg >= 0.5:
        print("⚠️  Consider additional training")
    else:
        print("❌ Review your dataset quality")

Common Pitfalls and How to Avoid Them

Pitfall 1: Learning rate too high → catastrophic forgetting. If your fine-tuned model loses capabilities it had before training, the learning rate is almost certainly too high. Start at 2.0e-4 and go as low as 5.0e-5 for small datasets (under 500 samples). Always include out-of-domain samples in your evaluation set to catch generalization regression.

Pitfall 2: Missing pad_token configuration. Gemma 4 doesn't have a pad token by default. Without tokenizer.pad_token = tokenizer.eos_token, you'll either get cryptic errors early in training or silently poor results. Always log tokenizer.pad_token_id and verify it's not None before training starts.

Pitfall 3: Deploying the raw adapter without merging. Saving with model.save_pretrained() outputs a LoRA adapter, not a standalone model. Uploading this directly to Cloudflare Workers AI will fail. Always run model.merge_and_unload() first, then convert to GGUF format before deployment.

Deploying to Cloudflare Workers AI

After merging your adapter into the base model, convert to GGUF and deploy:

# scripts/convert_to_gguf.py
 
import subprocess
from pathlib import Path
 
def convert_to_gguf(model_dir: str, output_name: str, quantization: str = "Q4_K_M"):
    """
    Convert to GGUF using llama.cpp.
    quantization options: Q4_K_M (balanced) / Q5_K_M (accuracy) / Q8_0 (high quality, large)
    """
    merged_path = Path(model_dir)
    output_path = merged_path.parent / f"{output_name}.gguf"
    
    cmd = [
        "python", "-m", "llama_cpp.convert",
        str(merged_path),
        "--outfile", str(output_path),
        "--outtype", quantization.lower(),
    ]
    
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode \!= 0:
        raise RuntimeError(f"GGUF conversion failed: {result.stderr}")
    
    file_size_mb = output_path.stat().st_size / (1024 * 1024)
    print(f"✅ Converted: {output_path} ({file_size_mb:.1f} MB)")
    
    if file_size_mb > 10_000:
        print("⚠️  File too large for Cloudflare Workers AI. Try Q4_K_M.")
    
    return str(output_path)
 
if __name__ == "__main__":
    convert_to_gguf("output/merged_model", "gemma4-custom-q4", "Q4_K_M")

Cloudflare Workers AI pricing as of April 2026 is approximately $0.011 per 1K input tokens and $0.044 per 1K output tokens — at 100K monthly requests with ~500 output tokens average, you're looking at roughly $2,200/month. For lower volumes or cost-sensitive use cases, a managed GPU instance (A10G, ~$700/month) becomes more economical, though you'll trade infrastructure management time for cost savings.

Next Steps

Production fine-tuning is not a one-time event. As usage patterns shift, model performance drifts. The best thing you can do today is run your existing dataset through the validation script and see what your current valid sample ratio looks like. That single metric will tell you whether your next improvement should be data quality, hyperparameter tuning, or architecture changes — and in my experience, it's almost always data quality first.