Running Gemma 4 Locally: A Developer's Guide to On-Device AI Integration

Google announced Gemma 4 on April 2, 2026, marking a significant shift in how developers think about AI. The key advantage for engineers: it runs entirely locally, is free to use commercially under Apache 2.0 license, and eliminates dependence on cloud APIs.

For developers seeking to embed AI directly into their applications, Gemma 4 is the obvious choice.

What's New in Gemma 4

The jump from Gemma 3 to Gemma 4 is substantial. Performance improvements include +330% on AIME (mathematical reasoning), +175% on LiveCodeBench (code generation), and a striking +1200% improvement when using agent tools.

This reflects architectural innovations, higher-quality training data, and native multimodal capabilities. Four model sizes are available for different use cases. E2B (2B MoE) targets phones and edge devices with 128K context and voice support. E4B (4B MoE) suits mobile and tablets. 26B MoE is designed for consumer GPUs—despite the size, Mixture of Experts means only 3.8B parameters are active at once. 31B Dense serves cloud deployments and high-precision applications with 256K token context.

With MoE, you load only the expert networks you need. 26B MoE runs comfortably on a RTX 3080.

Fastest Setup: Ollama

Ollama strips away configuration complexity. One command handles everything.

brew install ollama
ollama pull gemma4:2b
ollama run gemma4:2b

Gemma 4 2B downloads automatically and a REST API starts listening on localhost:11434. Call it from Python:

import requests
 
URL = "http://localhost:11434/api/generate"
 
def generate(prompt, temperature=0.7):
    payload = {
        "model": "gemma4:2b",
        "prompt": prompt,
        "temperature": temperature,
        "stream": False
    }
    response = requests.post(URL, json=payload)
    return response.json().get("response", "")
 
output = generate("List practical use cases for local LLMs")
print(output)

JSON responses are consistent, no API keys needed—the on-premises advantage.

Fine-Grained Control: llama.cpp

Ollama prioritizes simplicity. For precise parameter tuning (temperature, top-p, repeat penalty), use llama.cpp.

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make
 
wget https://huggingface.co/google/gemma-4/resolve/main/gemma-4-2b.Q4_K_M.gguf
 
./main -m gemma-4-2b.Q4_K_M.gguf -p "Your prompt here" --temp 0.3 -n 256

llama.cpp excels at GGUF quantization—compress models to one-quarter size without accuracy loss. 26B MoE quantized fits on a 12GB GPU.

Agent Capabilities and Tool Calling

Gemma 4's biggest leap is native agent functionality. Function Calling lets the model autonomously invoke external tools—databases, APIs, filesystems.

import json
 
TOOLS = [
    {
        "name": "fetch_data",
        "description": "Retrieve data from external API",
        "parameters": {
            "type": "object",
            "properties": {
                "endpoint": {"type": "string"}
            },
            "required": ["endpoint"]
        }
    }
]
 
prompt = f"""
Available tools:
{json.dumps(TOOLS, indent=2)}
 
Fetch the latest user statistics from /api/stats
"""
 
output = generate(prompt)
print(output)

Gemma 4 embeds tool calls in its response text. Parse them, execute the actual tools, and feed results back to the model.

Multimodal Capabilities

26B and 31B models natively process images and video. E2B and E4B handle audio.

import base64
 
with open("chart.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode()
 
payload = {
    "model": "gemma4:26b",
    "prompt": "Analyze this chart and summarize key trends",
    "images": [image_data],
    "stream": False
}
 
response = requests.post(URL, json=payload)
print(response.json()["response"])

Use cases: automated screenshot analysis, design document interpretation, robot vision processing.

Running on Raspberry Pi and Edge Devices

Gemma 4 E2B is confirmed to run on Raspberry Pi 5—proof that edge AI inference is now practical.

curl -sSL https://ollama.ai/install.sh | sh
ollama pull gemma4:2b-q2

Pi 5 inference: roughly 30 seconds per token. Not suitable for real-time chat, but fine for batch processing and data analysis. IoT devices, home robots, and offline learning systems now have viable AI options.

Production Deployment with Docker

For team consistency, containerize Ollama and integrate into CI/CD.

FROM nvidia/cuda:12.2-runtime-ubuntu22.04
 
RUN apt-get update && apt-get install -y curl git python3 pip
RUN curl -sSL https://ollama.ai/install.sh | sh
RUN ollama pull gemma4:2b
 
EXPOSE 11434
CMD ["ollama", "serve"]

The same API serves local machines, CI/CD servers, and production clusters uniformly.

Next Steps

Local Gemma 4 is production-ready—no longer experimental. Try E2B with Ollama in 15 minutes. Move to 26B MoE and test agent capabilities with your own tool definitions. Deploy via Docker or Kubernetes for high availability.

The local LLM era isn't coming. It's here. The question isn't whether to adopt it, but when.