Since Gemma 4 was released, "I want to fine-tune it on my own data" has become a common goal. Getting started is harder than it should be — GPU requirements are unclear, the setup feels intimidating, and the gap between "hello world" examples and something actually useful is wide.
This guide closes that gap. It covers Gemma 4 fine-tuning using QLoRA on Google Colab's free T4 GPU tier, then scales up to Vertex AI for production workloads. Everything includes working code.
When Fine-Tuning Is Worth It (and When It Isn't)
Before investing time in fine-tuning, it's worth confirming it's the right tool. Fine-tuning is valuable when you need the model to consistently produce outputs in a specific format or style, when your domain has vocabulary and patterns underrepresented in the base model's training data, or when prompt engineering alone can't reach the accuracy level your application requires.
For many use cases, few-shot prompting or carefully designed system instructions will get you 80% of the way there with a fraction of the effort. Try those first. If they fall short after genuine optimization, fine-tuning becomes the right investment.
QLoRA: Fine-Tuning With a Fraction of the Memory
Full fine-tuning of Gemma 4 requires GPU memory that most developers don't have access to. QLoRA (Quantized Low-Rank Adaptation) solves this by combining two techniques: quantizing the base model's weights to 4-bit (roughly 1/8 the memory of 32-bit), and limiting the weight updates to small "adapter" layers with far fewer parameters than the full model.
The practical result: a 16GB GPU can fine-tune a 7B-parameter model. A free Colab T4 can fine-tune Gemma 4 2B.
Setup: Google Colab Environment
# Install required libraries
\!pip install transformers datasets peft trl bitsandbytes accelerate -q
import torch
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import Dataset
# Verify GPU
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB")Loading Gemma 4 With 4-Bit Quantization
MODEL_NAME = "google/gemma-4-2b-it" # 2B model — fits on Colab free tier
# 4-bit quantization configuration
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NF4: recommended for QLoRA
bnb_4bit_compute_dtype=torch.bfloat16, # Compute in bfloat16
bnb_4bit_use_double_quant=True, # Double quantization saves ~0.4 bits/param
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
model = prepare_model_for_kbit_training(model)
print("Model loaded successfully")LoRA Adapter Configuration
lora_config = LoraConfig(
r=16, # Rank: lower = fewer parameters, higher = more capacity
lora_alpha=32, # Scaling factor — typically 2x the rank
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj", # Attention layers
"gate_proj", "up_proj", "down_proj" # FFN layers
],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
# Show what fraction of parameters will be trained
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable parameters: {trainable:,} ({100*trainable/total:.2f}%)")
# Typical output: Trainable parameters: 3,407,872 (0.17%)Start with r=16. If the fine-tuned model lacks quality after training, try r=32. If GPU memory becomes the constraint, try r=8. The rank controls the trade-off between model capacity and memory usage.
Dataset Preparation
The training data format matters. Gemma 4 Instruct models expect a specific conversation template:
# Example training data — in practice, load from CSV or JSON
raw_data = [
{
"instruction": "What is your return policy?",
"response": "We accept returns within 30 days of purchase. Please contact us through the support form to initiate a return."
},
{
"instruction": "How long does shipping take?",
"response": "Standard shipping takes 3-5 business days. Express shipping delivers by the next business day."
},
# ... you'll want at least 500-1000 examples for meaningful fine-tuning
]
def format_for_gemma(example: dict) -> str:
"""Format as Gemma 4 Instruct conversation"""
return (
f"<start_of_turn>user\n"
f"{example['instruction']}<end_of_turn>\n"
f"<start_of_turn>model\n"
f"{example['response']}<end_of_turn>"
)
formatted = [{"text": format_for_gemma(item)} for item in raw_data]
dataset = Dataset.from_list(formatted)
split = dataset.train_test_split(test_size=0.2, seed=42)
train_dataset = split["train"]
eval_dataset = split["test"]
print(f"Training examples: {len(train_dataset)}")
print(f"Validation examples: {len(eval_dataset)}")Running the Fine-Tuning
training_args = TrainingArguments(
output_dir="./gemma4-finetuned",
num_train_epochs=3,
per_device_train_batch_size=4, # For T4 16GB
per_device_eval_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size = 4 × 4 = 16
learning_rate=2e-4,
weight_decay=0.001,
fp16=True, # T4 does not support bfloat16
bf16=False, # switch to bf16=True on A100/L4 (Ampere+)
max_grad_norm=0.3,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
evaluation_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100,
logging_steps=25,
load_best_model_at_end=True,
report_to="none",
)
trainer = SFTTrainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
peft_config=lora_config,
dataset_text_field="text",
max_seq_length=1024,
tokenizer=tokenizer,
args=training_args,
packing=False,
)
trainer.train()
trainer.save_model("./gemma4-finetuned-final")
print("Fine-tuning complete")One easily missed detail: Colab's free-tier T4 GPU does not support bfloat16 (bf16 requires Ampere-generation hardware or newer). Running with bf16=True on a T4 either errors out or silently degrades. Use fp16=True on T4, and flip back to bf16=True only when you move to an A100 or L4. For the same reason, set bnb_4bit_compute_dtype to torch.float16 on T4.
A second hard-earned tip: free Colab sessions disconnect without warning. Mount Google Drive and point output_dir there, so resume_from_checkpoint=True can pick up where the session died. I lost two hours of training to a local-only output_dir once — set Drive checkpointing before the first long run, not after.
from google.colab import drive
drive.mount('/content/drive')
training_args.output_dir = "/content/drive/MyDrive/gemma4-finetuned"
trainer.train(resume_from_checkpoint=True)Inference With the Fine-Tuned Model
from peft import PeftModel
# Load and merge the adapter into the base model
base_model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
quantization_config=bnb_config,
device_map="auto"
)
finetuned_model = PeftModel.from_pretrained(base_model, "./gemma4-finetuned-final")
finetuned_model = finetuned_model.merge_and_unload() # Merge adapter weights
def generate(prompt: str, max_new_tokens: int = 256) -> str:
formatted = f"<start_of_turn>user\n{prompt}<end_of_turn>\n<start_of_turn>model\n"
inputs = tokenizer(formatted, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = finetuned_model.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=0.7,
do_sample=True,
pad_token_id=tokenizer.eos_token_id
)
generated_ids = outputs[0][inputs["input_ids"].shape[1]:]
return tokenizer.decode(generated_ids, skip_special_tokens=True)
print(generate("What is your return policy?"))Scaling to Vertex AI for Production
For larger datasets, bigger models (9B+), or production reliability, Vertex AI removes the resource constraints of Colab:
from google.cloud import aiplatform
aiplatform.init(project="YOUR_PROJECT_ID", location="us-central1")
job = aiplatform.CustomTrainingJob(
display_name="gemma4-fine-tuning",
script_path="train.py", # Save the training code above as train.py
container_uri="us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.2-2:latest",
requirements=["transformers", "peft", "trl", "bitsandbytes", "datasets"]
)
# A100 80GB supports Gemma 4 9B fine-tuning
job.run(
machine_type="a2-highgpu-1g",
accelerator_type="NVIDIA_TESLA_A100",
accelerator_count=1,
replica_count=1,
args=[
"--model_name", "google/gemma-4-9b-it",
"--epochs", "3",
"--output_dir", "gs://YOUR_BUCKET/gemma4-finetuned"
]
)Practical Resource Reference
For planning purposes: on Colab (free T4 GPU), expect 1–2 hours for Gemma 4 2B with 500 training examples over 3 epochs, at no cost (subject to Colab's usage limits). On Vertex AI (A100 80GB), expect 3–5 hours for Gemma 4 9B with 5,000 examples over 3 epochs, at roughly $15–$30 at current Vertex AI pricing.
Start with Colab to validate that your dataset and training setup produce a measurable improvement before committing to Vertex AI costs. A quick 100-example pilot run is usually enough to confirm the training direction before scaling up.
For more on working with Gemma 4 models in general, the Antigravity articles cover base model usage, API integration, and deployment patterns.
Common Issues and How to Fix Them
CUDA out of memory: Reduce per_device_train_batch_size to 2 or 1, increase gradient_accumulation_steps proportionally to maintain the same effective batch size. Also try reducing max_seq_length if your training examples are long.
Training loss plateaus early: Your learning rate may be too high. Try 1e-4 instead of 2e-4. Alternatively, if you have fewer than 200 training examples, the dataset may be too small to show meaningful improvement — add more data first.
Fine-tuned model ignores the new format or style: This usually means the training data doesn't consistently represent the target behavior, or the model has insufficient examples to override its base behavior for the given pattern. Review the training data for inconsistencies and consider increasing the number of epochs to 5.
Inference is slow after merging: If you've merged the adapter with merge_and_unload(), the model is now full precision. For faster inference, load with quantization without merging — just load the PeftModel directly without calling merge_and_unload().
What Makes a Good Training Dataset
The quality of the fine-tuning result is almost entirely determined by the quality of the training data. A few principles that consistently matter:
Consistency is more important than volume. 300 consistently formatted, high-quality examples will produce better results than 2,000 inconsistent ones. Before scaling up data collection, establish a clear specification for what "correct" output looks like and apply it uniformly.
Represent the edge cases. If your application needs to handle unusual inputs gracefully (ambiguous requests, out-of-scope questions, sensitive topics), include those in the training data with the desired responses. If they're absent, the model will fall back to base model behavior for those inputs.
Include negative examples sparingly. For tasks where the model needs to decline certain requests, including a small number of correctly-handled refusal examples helps. Don't over-index on refusals — the model already has base training in this area.
Match the inference prompt format exactly. If your production code wraps user input in a specific template before sending to the model, make sure your training data uses exactly that same template. Format mismatches between training and inference are a consistent source of degraded performance.
Fine-tuning Gemma 4 is more accessible than it was even a year ago, largely because of QLoRA's memory efficiency improvements. The workflow here — Colab for validation, Vertex AI for production — gives you a cost-effective path from experiment to deployment without needing dedicated GPU infrastructure. As an indie developer I have come to treat training as a routine loop of fixing data and re-running, not a one-off ritual, and this setup is built around making that loop cheap.
If you work on Apple Silicon, there is also a fully local path that skips Colab entirely — covered in the Gemma 4 fine-tuning on Mac with MLX guide. Having both cloud and local options lets you route each dataset by its sensitivity.