Gemma 4 Fine-Tuning in Practice: Preventing Data Starvation, Overfitting, and Quality Problems

Most Gemma 4 fine-tuning tutorials stop at environment setup. They show you how to run the training loop, but they don't help you answer the questions that actually matter: How much data do I really need? When is my model overfitting? Which checkpoint should I deploy? How do I know if it's actually better?

Before You Fine-Tune: Check Whether You Need To

Fine-tuning isn't always the right tool. The right question is whether prompt engineering can get you close enough.

import os
from google import genai
 
client = genai.Client(api_key=os.getenv("GOOGLE_API_KEY"))
 
def test_with_prompting(
    example_inputs: list[str],
    example_outputs: list[str],
    model: str = "gemini-2.5-pro"
) -> float:
    """Measure how well few-shot prompting handles the task before fine-tuning."""
    
    # Build a few-shot prompt from the first 5 examples
    examples = [
        f"Input: {inp}\nOutput: {out}"
        for inp, out in zip(example_inputs[:5], example_outputs[:5])
    ]
    
    system = f"""Follow this exact format:
 
Examples:
{chr(10).join(examples)}
 
When given an input, produce output in exactly the same format."""
 
    correct = 0
    test_inputs = example_inputs[5:]
    test_outputs = example_outputs[5:]
    
    for inp, expected in zip(test_inputs, test_outputs):
        response = client.models.generate_content(
            model=model,
            contents=f"Input: {inp}",
            config={"system_instruction": system}
        )
        if response.text.strip() == expected:
            correct += 1
    
    return correct / max(len(test_inputs), 1)
 
# If accuracy >= 0.80, prompting may be sufficient — skip fine-tuning

Fine-tuning makes sense when you need: a specific style or tone that prompting can't reliably reproduce, privacy requirements that prevent sending data to a hosted API, latency requirements that need a smaller deployed model, or cost optimization at high request volumes.

Data Preparation: Quality Over Quantity

Training results are roughly 90% determined by data quality. A few hundred well-curated examples consistently outperform thousands of noisy ones.

Data Requirements by Task Type

2–5 class classification: 100–500 samples per class
Text generation / style transfer: 500–2,000 samples
Domain-specific Q&A: 1,000–5,000 pairs
Code generation: 2,000–10,000 samples

Below these thresholds, consider data augmentation or prompting instead.

Validate Before Training

from collections import Counter
 
def validate_training_data(data: list[dict]) -> dict:
    """Run quality checks on training data before using it."""
    
    issues = {
        "duplicates": 0,
        "empty_inputs": 0,
        "empty_outputs": 0,
        "too_short": 0,
        "too_long": 0,
        "format_errors": 0,
    }
    
    inputs_seen = []
    for i, sample in enumerate(data):
        if "input" not in sample or "output" not in sample:
            issues["format_errors"] += 1
            continue
        
        inp = sample["input"].strip()
        out = sample["output"].strip()
        
        if not inp: issues["empty_inputs"] += 1
        if not out: issues["empty_outputs"] += 1
        if len(inp) < 10: issues["too_short"] += 1
        if len(inp) > 4000 or len(out) > 2000: issues["too_long"] += 1
        
        inputs_seen.append(inp)
    
    counts = Counter(inputs_seen)
    issues["duplicates"] = sum(c - 1 for c in counts.values() if c > 1)
    
    total_issues = sum(issues.values())
    quality_score = max(0, 1 - (total_issues / len(data)))
    
    return {
        "total_samples": len(data),
        "issues": {k: v for k, v in issues.items() if v > 0},
        "quality_score": quality_score,
        "avg_input_length": sum(len(s.get("input", "")) for s in data) / len(data),
    }
 
stats = validate_training_data(training_data)
print(f"Quality score: {stats['quality_score']:.1%}")
if stats["issues"]:
    print(f"Issues found: {stats['issues']}")

Data Augmentation for Small Datasets

import json
import random
 
def augment_data(
    original: list[dict],
    target_size: int,
    model: str = "gemini-2.0-flash"
) -> list[dict]:
    """Expand a small dataset using model-generated variations."""
    
    augmented = list(original)
    examples = random.sample(original, min(5, len(original)))
    examples_text = "\n".join(f"Input: {e['input']}\nOutput: {e['output']}" for e in examples)
    
    existing_inputs = {d["input"] for d in augmented}
    attempts = 0
    max_attempts = (target_size - len(original)) * 3
    
    while len(augmented) < target_size and attempts < max_attempts:
        attempts += 1
        prompt = f"""Generate one new training sample in the same style as these examples.
 
Existing examples:
{examples_text}
 
Requirements:
- Same domain, format, and difficulty level
- NOT a duplicate of any existing example
- Introduce variety in topic or phrasing
 
Respond with only valid JSON: {{"input": "...", "output": "..."}}"""
        
        try:
            response = client.models.generate_content(model=model, contents=prompt)
            text = response.text.strip()
            if "```" in text:
                text = text.split("```")[1].replace("json", "").strip()
            
            new_sample = json.loads(text)
            if "input" in new_sample and new_sample["input"] not in existing_inputs:
                augmented.append(new_sample)
                existing_inputs.add(new_sample["input"])
        except Exception:
            pass
    
    return augmented
 
# Always human-review augmented samples before training

LoRA vs QLoRA: Decision Framework

Full parameter fine-tuning of Gemma 4 requires hardware most people don't have. LoRA and QLoRA make fine-tuning accessible.

When to use each:

VRAM ≥ 24 GB: LoRA (higher accuracy, full precision)
VRAM 8–24 GB: QLoRA (good balance of quality and memory)
VRAM < 8 GB: QLoRA with 4-bit quantization (the only option)
Maximum accuracy matters: LoRA
Speed matters more: QLoRA

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
import torch
 
def setup_model(
    model_name: str = "google/gemma-4-12b",
    quantize: bool = True,
    lora_rank: int = 16,
    lora_alpha: int = 32,
) -> tuple:
    """Configure a Gemma 4 model for LoRA or QLoRA fine-tuning."""
    
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token
    
    if quantize:
        quant_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=torch.bfloat16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type="nf4",
        )
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            quantization_config=quant_config,
            device_map="auto",
            torch_dtype=torch.bfloat16,
        )
    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_name, device_map="auto", torch_dtype=torch.bfloat16
        )
    
    lora_config = LoraConfig(
        task_type=TaskType.CAUSAL_LM,
        r=lora_rank,
        lora_alpha=lora_alpha,
        lora_dropout=0.05,
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                        "gate_proj", "up_proj", "down_proj"],
        bias="none",
    )
    
    model = get_peft_model(model, lora_config)
    model.print_trainable_parameters()
    # Example output: trainable params: 41,943,040 || all params: 12.3B || trainable%: 0.34%
    
    return model, tokenizer
 
# Rank selection guide:
# r=4:  Style adaptation with very small datasets
# r=8:  Small-to-medium task adaptation
# r=16: General-purpose fine-tuning (start here)
# r=32: Complex tasks with large datasets
# r=64: Near full fine-tuning effect (high VRAM requirement)

Preventing Overfitting

Training Configuration

from transformers import TrainingArguments, Trainer, EarlyStoppingCallback
 
def get_training_args(output_dir: str, batch_size: int = 4) -> TrainingArguments:
    return TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=3,
        
        # Evaluate every 100 steps, not just at epoch end
        eval_strategy="steps",
        eval_steps=100,
        save_strategy="steps",
        save_steps=100,
        
        # Effective batch size = 4 × 4 = 16 (without needing 16 samples in memory)
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size,
        gradient_accumulation_steps=4,
        
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        warmup_ratio=0.05,
        
        # Regularization
        weight_decay=0.01,
        max_grad_norm=0.3,
        
        # Load the best checkpoint (by eval_loss) at the end
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        
        # Memory efficiency
        bf16=True,
        gradient_checkpointing=True,
        logging_steps=10,
    )
 
def train(model, tokenizer, train_ds, eval_ds, output_dir="./output"):
    trainer = Trainer(
        model=model,
        args=get_training_args(output_dir),
        train_dataset=train_ds,
        eval_dataset=eval_ds,
        callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
    )
    
    trainer.train()
    
    # Check for overfitting in the log history
    logs = trainer.state.log_history
    eval_losses = [l["eval_loss"] for l in logs if "eval_loss" in l]
    if len(eval_losses) >= 2 and eval_losses[-1] > eval_losses[-2]:
        print(f"⚠️ Eval loss increased — overfitting. Best checkpoint: {trainer.state.best_model_checkpoint}")
    
    return trainer

Selecting the Right Checkpoint

import os, json
 
def find_best_checkpoint(checkpoint_dir: str) -> str:
    """Find the checkpoint with the lowest eval_loss."""
    
    best_loss = float("inf")
    best_path = None
    
    for name in os.listdir(checkpoint_dir):
        if not name.startswith("checkpoint-"):
            continue
        
        state_file = os.path.join(checkpoint_dir, name, "trainer_state.json")
        if not os.path.exists(state_file):
            continue
        
        with open(state_file) as f:
            state = json.load(f)
        
        step = int(name.split("-")[1])
        for entry in reversed(state.get("log_history", [])):
            if "eval_loss" in entry and entry.get("step") == step:
                if entry["eval_loss"] < best_loss:
                    best_loss = entry["eval_loss"]
                    best_path = os.path.join(checkpoint_dir, name)
                break
    
    print(f"Best checkpoint: {best_path} (eval_loss: {best_loss:.4f})")
    return best_path

Pre-Deployment Evaluation

Never deploy a fine-tuned model based on training metrics alone. Measure against held-out test data.

from bert_score import score as bert_score
from rouge_score import rouge_scorer
import numpy as np, torch
 
def evaluate_model(model, tokenizer, test_data: list[dict], device: str = "cuda") -> dict:
    """Quantitatively evaluate a fine-tuned model on held-out test data."""
    
    predictions, references = [], []
    scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
    
    for sample in test_data:
        inputs = tokenizer(
            sample["input"], return_tensors="pt",
            truncation=True, max_length=512
        ).to(device)
        
        with torch.no_grad():
            outputs = model.generate(
                **inputs, max_new_tokens=256, temperature=0.1, do_sample=False
            )
        
        generated = tokenizer.decode(
            outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
        )
        predictions.append(generated.strip())
        references.append(sample["output"].strip())
    
    rouge = [scorer.score(r, p) for r, p in zip(references, predictions)]
    _, _, bert_f1 = bert_score(predictions, references, lang="en", device=device)
    
    return {
        "rouge1": np.mean([s["rouge1"].fmeasure for s in rouge]),
        "rouge2": np.mean([s["rouge2"].fmeasure for s in rouge]),
        "rougeL": np.mean([s["rougeL"].fmeasure for s in rouge]),
        "bert_score_f1": bert_f1.mean().item(),
        "samples": len(test_data),
    }
 
# Compare fine-tuned vs base model
# base_metrics = evaluate_model(base_model, tokenizer, test_data)
# ft_metrics = evaluate_model(finetuned_model, tokenizer, test_data)
# delta = {k: ft_metrics[k] - base_metrics[k] for k in ft_metrics if isinstance(ft_metrics[k], float)}
# print("Improvement:", delta)

A fine-tuned model that doesn't show measurable improvement on test data—that it never saw during training—shouldn't go to production, regardless of how low the training loss got.

The pattern I've seen in failed fine-tuning attempts is almost always the same: training started too quickly, data quality wasn't checked, and overfitting wasn't monitored. The model learned the training examples well and nothing else.

Start with a small dataset (200–300 samples), run the complete pipeline from data validation through quality evaluation, and verify the numbers before investing in a larger training run. Getting the pipeline right on a small scale before scaling up saves significantly more time than optimizing for training speed from the start.