Fine-Tuning Gemma 4 on a Mac — A Colab-Free Training Setup with Apple Silicon + MLX

Fine-tuning Gemma 4 on Google Colab is well-documented. But Colab has friction: free tier sessions time out, Pro sessions still have limits, and there's something uncomfortable about not knowing when your training run will get interrupted.

Running it on a Mac with Apple Silicon turns out to be a realistic alternative — at least for certain use cases. I tested this on an M3 Max with 96GB unified memory, and the experience was different from Colab in ways I didn't fully expect. Here's what I found. For an indie developer, having a training environment that adds zero recurring cloud cost is a quietly significant shift. In the second half, we'll connect the trained adapter all the way through Ollama into Antigravity.

Why MLX Over PyTorch + MPS

MLX is Apple's open-source machine learning framework designed specifically for Apple Silicon's unified memory architecture. Because the GPU and CPU share the same memory pool, a 96GB M3 Max can handle the full Gemma 4 27B model without quantization — barely, but it works.

PyTorch with MPS is an option too, but MLX proved more memory-efficient in practice, and the mlx-lm library provides fine-tuning utilities that integrate cleanly with the Hugging Face ecosystem.

Hardware note: if you're on an M1/M2/M3 chip with less than 64GB unified memory, use the 4-bit quantized version of Gemma 4 (approximately 14GB). Larger models on smaller machines are better handled by Colab.

Realistic Configurations by Unified Memory

What you can do locally is determined almost entirely by unified memory. Based on my own runs and configurations reported in the MLX community:

• 96GB+ (M3 Max / M4 Max high-end): 27B at 8-bit quantization + LoRA is comfortably stable. Start with --batch-size 4 and --lora-layers 16
• 64GB: 27B at 4-bit quantization + LoRA. Dropping to --batch-size 2 helps avoid swap
• 32-36GB: a 12B-class model at 4-bit is the realistic ceiling. 27B may run inference, but training gradients will exhaust memory
• 16-24GB: experiment with 4B-class models, or prepare datasets locally and hand training off to Colab

A rough planning formula that has served me well: quantized model size × 2 + 8GB equals your training memory budget. LoRA freezes the base model and trains only small adapter matrices, so it is far lighter than full fine-tuning — but gradients and optimizer state still add real overhead.

If you're wondering how much quality the quantization bit depth actually costs, Gemma 4 on Antigravity: Picking Q4 vs Q5 — What I Found After a Week on M2 Mac is a useful companion read. It compares inference rather than training, but the intuition about where quantization bites carries over.

Environment Setup

# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
 
# Python 3.11+ recommended
brew install python@3.11
 
# Create and activate virtual environment
python3.11 -m venv gemma4-mlx-env
source gemma4-mlx-env/bin/activate
 
# Install MLX and related libraries
pip install mlx-lm
pip install huggingface-hub datasets
 
# Log into Hugging Face (required to download Gemma 4)
huggingface-cli login

Before downloading Gemma 4, visit huggingface.co and accept the usage terms. Without this, the download returns a 401 error — easy to miss if you skip directly to the command.

Preparing the Dataset

Training data uses Alpaca format (instruction / input / output). For custom data, prepare a JSON file in the same structure:

# custom_dataset.json example
[
    {
        "instruction": "Find the bug in this code",
        "input": "def add(a, b):\n    return a - b",
        "output": "The operator is wrong. It should be `a + b`, not `a - b`."
    }
]

# Convert to MLX training format
import json
from pathlib import Path
 
def convert_to_mlx_format(input_file: str, output_dir: str):
    """Convert Alpaca-format JSON to MLX JSONL training format."""
    with open(input_file, "r", encoding="utf-8") as f:
        data = json.load(f)
    
    converted = []
    for item in data:
        prompt = f"<start_of_turn>user\n{item['instruction']}"
        if item.get("input"):
            prompt += f"\n\n{item['input']}"
        prompt += "<end_of_turn>\n<start_of_turn>model\n"
        response = item["output"] + "<end_of_turn>"
        converted.append({"text": prompt + response})
    
    # 90/10 train/validation split
    split = int(len(converted) * 0.9)
    
    out = Path(output_dir)
    out.mkdir(exist_ok=True)
    
    with open(out / "train.jsonl", "w") as f:
        for item in converted[:split]:
            f.write(json.dumps(item, ensure_ascii=False) + "\n")
    
    with open(out / "valid.jsonl", "w") as f:
        for item in converted[split:]:
            f.write(json.dumps(item, ensure_ascii=False) + "\n")
    
    print(f"Done: train={split}, valid={len(converted)-split}")
 
convert_to_mlx_format("custom_dataset.json", "data")

One mistake I made early on, so you can skip it: if you train on raw text without Gemma's chat template tags (<start_of_turn> / <end_of_turn>), loss goes down but conversations fall apart at inference time. When your base model is the instruction-tuned variant (-it), your training data must use the same template.

Running the LoRA Fine-Tune

# Download the model (first run only, ~14GB for 4-bit quantized version)
python -c "
from huggingface_hub import snapshot_download
snapshot_download('google/gemma-4-27b-it', local_dir='./gemma-4-27b-it')
"
 
# Start LoRA fine-tuning
mlx_lm.lora \
  --model ./gemma-4-27b-it \
  --train \
  --data ./data \
  --iters 1000 \
  --batch-size 4 \
  --lora-layers 16 \
  --learning-rate 1e-4 \
  --adapter-path ./adapters \
  --save-every 100

On M3 Max, 1,000 iterations took about 40 minutes. That's roughly 1.5-2x slower than an A100 on Colab Pro+, but the run completed without interruption — which matters more than you'd think when you're iterating on dataset quality.

With --save-every 100, adapter checkpoints are written every 100 iterations. For overnight runs this is essential: if the process dies, you can resume from the last checkpoint with --resume-adapter-file instead of starting over.

Choosing a Realistic Learning Rate and Iteration Count

Treat the sample command's --learning-rate 1e-4 as a value tuned for datasets in the thousands of examples. When I ran a few-hundred-example dataset at 1e-4, loss dropped fast — too fast. The model slid into memorization, parroting phrasings straight out of the training data. Dropping to 5e-5 kept the responses natural while still pulling them toward the target style. My rule now: start at 5e-5, and only raise it if loss barely moves.

For iteration count, my starting point is "dataset size × 2-3 epochs ÷ batch size." With 300 examples and --batch-size 4, one epoch is roughly 75 iterations, so 150-225 iterations is a sensible first run. There's no need to default to 1,000.

Validation loss is the signal that matters. When the valid loss printed at each checkpoint stops improving, further iterations only deepen memorization. Watching train loss alone is the trap — it keeps "improving" convincingly, and I wasted several early runs exactly this way.

Setting Up Your Mac for Overnight Runs

The Mac's biggest advantage over Colab is running while you sleep. But run unprepared and you'll wake up to a machine that went to sleep mid-training. This is the setup I settled on:

# Block idle sleep for exactly as long as training runs
caffeinate -i mlx_lm.lora \
  --model ./gemma-4-27b-it \
  --train \
  --data ./data \
  --iters 1000 \
  --batch-size 4 \
  --lora-layers 16 \
  --learning-rate 5e-5 \
  --adapter-path ./adapters \
  --save-every 100 2>&1 | tee train_log.txt

• caffeinate -i blocks idle sleep only while the command is alive. No system settings to change, nothing to remember to switch back. The display can sleep — that's fine. You only need to stop system sleep
• Plug into power. On battery, macOS leans toward power saving and training speed becomes unstable
• tee writes the log to a file, so in the morning you can review the full loss curve. Relying on terminal scrollback alone means losing the middle of the run
• Temporarily pause macOS automatic updates that involve restarts, at least on nights you train

Testing the Fine-Tuned Model

from mlx_lm import load, generate
 
model, tokenizer = load(
    "./gemma-4-27b-it",
    adapter_path="./adapters"
)
 
prompt = """<start_of_turn>user
Find any issues with this Python code:
 
def calculate_average(numbers):
    total = sum(numbers)
    return total / len(numbers)
<end_of_turn>
<start_of_turn>model
"""
 
response = generate(model, tokenizer, prompt=prompt, max_tokens=512, temp=0.7)
print(response)

A practical testing tip: prompt the model with paraphrases of your training questions, not the originals. Answering memorized inputs proves nothing — generalization shows when the response style holds up under rewording. Small datasets of a few hundred examples overfit quickly; if validation loss starts climbing mid-run, reduce --iters or add data.

Fusing the Adapter for Ollama / Antigravity

Once you are happy with the adapter, fuse it into a standalone model:

mlx_lm.fuse \
  --model ./gemma-4-27b-it \
  --adapter-path ./adapters \
  --save-path ./gemma-4-27b-custom

The fused model works directly with mlx_lm.generate, and if you convert it to GGUF with llama.cpp's conversion script, it loads into Ollama. From there, Antigravity's local LLM connection can call it like any other model — meaning a completion model trained on your own codebase's conventions can live inside your IDE, all built on a single Mac. The Antigravity-side connection steps are covered in Setting Up Local LLMs in Antigravity for Practical Use.

# After GGUF conversion, register with Ollama
cat > Modelfile <<'EOF2'
FROM ./gemma-4-27b-custom.gguf
PARAMETER temperature 0.7
EOF2
ollama create gemma4-custom -f Modelfile
ollama run gemma4-custom

One more thing worth bookmarking: if long responses start getting cut off once you're running through Ollama, Fixing Mid-Stream Cutoffs and Long-Run Freezes When Antigravity Talks to Ollama covers the fixes. The symptom isn't specific to custom models — it affects Ollama integrations in general.

Common Errors and Fixes

Errors I actually hit, or reproduced while testing:

• Out of memory right after training starts: halve --batch-size first. If it still dies, drop --lora-layers from 16 to 8 or switch to the 4-bit base model. Sometimes just closing the browser is enough. If the same class of error appears later on the inference side — when calling the model from Antigravity — see How to Fix Out of Memory Errors When Using Gemma 4 in Antigravity
• 401 Unauthorized on download: you have not accepted the Gemma license on Hugging Face, or your CLI token lacks read permission
• Training progresses but responses are broken: the chat template mismatch described above. Pull one line out of train.jsonl and visually confirm the template tags are present
• Sudden slowdown mid-run: memory pressure has pushed you into swap. If Activity Monitor's memory pressure turns yellow, a smaller batch size finishes faster overall

When to Use Mac vs. Colab

The choice comes down to dataset size and model scale:

Use Mac (MLX) when:

• Dataset is under ~1,000 examples (fast iteration, low stakes)
• You need overnight or multi-hour runs without session limits
• You want to avoid cloud costs entirely
• You have 64GB+ unified memory

Use Colab when:

• Dataset is 10,000+ examples and training time matters
• You're working with models larger than Gemma 4 27B
• Your Mac has less than 32GB unified memory

The practical pattern I've settled on: prototype and iterate on Mac, then run the final production training pass on Colab Pro+ where speed matters. MLX made the Mac genuinely useful for fine-tuning — not just as an inference machine, but as a real part of the training workflow.

Start with a small dataset of around 100 examples and push one full cycle through the pipeline — convert, train, fuse, register with Ollama. Once the pipeline works end to end, you can spend your energy where it matters: improving the data. I hope this helps if you are building on the same setup.