Memory Budget Design for Embedding Gemma 4 in Mobile Apps

The night I tried adding a small on-device model to a wallpaper app, I looked at the file size, decided it would fit, and moved on. It ran in the simulator. On a real phone, a few seconds into inference, the app simply vanished.

The cause wasn't the model's size. It was peak memory. File size and runtime footprint are different things: weights, KV cache, activations, and the OS and runtime reserve all have to fit inside what the device will actually grant you, at the same moment.

Whether it fits is decided by the peak, not the file. I've built every budget since around that one sentence.

Start with the published numbers

The on-device Gemma 4 models are E2B and E4B. Google's documentation lists approximate memory for the static weights.

Two caveats are easy to miss. These figures cover static weights only, not the KV cache or supporting software. And the same page says they vary by inference tool and environment. E2B and E4B also use Per-Layer Embeddings, so what gets loaded is larger than the "effective" parameter count suggests.

So the table is only the first row of your budget. The remaining rows you fill in by measuring on your own devices.

The four memory regions

An app that is running inference occupies roughly four regions.

• Model weights: the quantized tensors themselves (static).
• KV cache: Key/Value buffers for past tokens (grows with context length).
• Activations: intermediate tensors during inference.
• OS and runtime reserve: GPU driver, ML framework, and your app's own UI and caches.

Judge by weights alone and the other three regions will overrun you.

Estimate the KV cache with a formula

The KV cache can be estimated from layer count, KV head count, head dimension, context length, and element size.

KV bytes ≈ 2 × layers × kv_heads × head_dim × context_length × bytes_per_element

The leading 2 is for Key and Value. Layer and head counts differ per model, so I won't quote numbers here; read them from the model card or the runtime log. I chose context lengths by gut feeling at first and kept lowering the limit afterwards.

The takeaway is the linear relationship: double the context and this term doubles. You can't move the weights, but you can move this.

Filling in a budget table once, with placeholder numbers

The official page mentions a 128K context window. That is a ceiling, not a budget. Plug placeholder values into the formula above (24 layers, 4 KV heads, head dimension 128, 2 bytes for fp16) and the KV cache per context length falls out. These are not any real model's values; they are there to give you the order of magnitude.

The arithmetic: 2 × 24 × 4 × 128 × 4,096 × 2 bytes = 201,326,592 bytes, which is 192 MiB. Every other row is just that value doubled or quadrupled.

I skipped this table at first and spent a while thinking "it supports 128K, so I can go long." Being able to go long and surviving at that length are different questions. What belongs in your budget is not the model's ceiling but the largest value you measured on your own device without a crash.

Usable memory is smaller than physical memory

Classifying devices by total RAM alone is risky, because what your app may actually use is much smaller.

On iOS there is an API that returns what is left for your app right now.

import os
 
func availableBytesForApp() -> UInt64 {
    // iOS 13+. Roughly how much more the app can allocate.
    return UInt64(os_proc_available_memory())
}
 
func chooseProfile() -> ModelProfile {
    let gb = Double(availableBytesForApp()) / 1_073_741_824
 
    if gb >= 4.0 {
        return .e4bFullContext
    } else if gb >= 2.0 {
        return .e4bShortContext
    } else {
        return .e2bShortContext
    }
}

The 4.0 and 2.0 thresholds are placeholders. Derive yours from your model's measured peak plus a margin.

On Android, availMem and lowMemory from ActivityManager.getMemoryInfo() play the same role. If isLowRamDevice() returns true, I'd rather start from the smaller model.

import android.app.ActivityManager
import android.content.Context
 
fun chooseProfile(context: Context): ModelProfile {
    val am = context.getSystemService(Context.ACTIVITY_SERVICE) as ActivityManager
    val info = ActivityManager.MemoryInfo().also { am.getMemoryInfo(it) }
 
    // While lowMemory is true, or on low-RAM devices, pin to the smaller model
    if (info.lowMemory || am.isLowRamDevice) return ModelProfile.E2B_SHORT_CONTEXT
 
    val availGb = info.availMem / 1_073_741_824.0
    return when {
        availGb >= 4.0 -> ModelProfile.E4B_FULL_CONTEXT
        availGb >= 2.0 -> ModelProfile.E4B_SHORT_CONTEXT
        else -> ModelProfile.E2B_SHORT_CONTEXT
    }
}

availMem reports free memory for the whole device, not a per-app ceiling like iOS's os_proc_available_memory(). Set the Android thresholds more conservatively than the iOS ones.

I split by remaining memory rather than device name because the same phone has different headroom depending on what is running in the background.

Sliding-window KV cache

When you want a long context but a lower peak, switch the KV cache from "keep everything" to "keep the prefix and slide a window." The system prompt stays at the front; the middle of old conversation is dropped.

class SlidingKVCache:
    """Keep the first keep_prefix tokens; slide the rest as a window."""
 
    def __init__(self, max_tokens: int, keep_prefix: int):
        self.max_tokens = max_tokens
        self.keep_prefix = keep_prefix
        self.tokens: list[int] = []
 
    def append(self, token_ids: list[int]) -> bool:
        """Returns True if trimming happened; the caller rebuilds the KV."""
        self.tokens.extend(token_ids)
        if len(self.tokens) <= self.max_tokens:
            return False
        overflow = len(self.tokens) - self.max_tokens
        head = self.tokens[: self.keep_prefix]
        tail = self.tokens[self.keep_prefix + overflow :]
        self.tokens = head + tail
        return True

The trade-off is forgetting whatever leaves the window. That rarely matters for FAQ-style chat, and it's a poor fit for long story generation or code editing. I'd make it switchable per use case.

Measure load and inference separately

Memory behaves differently while the model loads and while it runs. Some implementations briefly use nearly double during load, so you need both measurements to catch the peak.

import os.signpost
 
func measureLoadAndInference() {
    let log = OSLog(subsystem: "app.model", category: "memory")
 
    let before = os_proc_available_memory()
    os_signpost(.begin, log: log, name: "model_load")
    let model = loadModel()
    os_signpost(.end, log: log, name: "model_load")
    let afterLoad = os_proc_available_memory()
 
    os_signpost(.begin, log: log, name: "inference")
    _ = model.generate("Hello")
    os_signpost(.end, log: log, name: "inference")
    let afterRun = os_proc_available_memory()
 
    print("consumed by load:", before - afterLoad)
    print("remaining after inference:", afterRun)
}

Lining up signposts in Instruments tells you whether a crash happened during load or during inference. When a "it crashes in production" report arrives, that split saves a lot of time.

Be ready for memory warnings

When memory gets tight, the OS sends a warning and, if things don't improve, terminates the app. With a large model, design so you can unload the moment a warning arrives.

NotificationCenter.default.addObserver(
    forName: UIApplication.didReceiveMemoryWarningNotification,
    object: nil, queue: .main
) { _ in
    ModelManager.shared.unloadIfIdle(seconds: 5)
}

If the model is idle, release it and reload on next use. A few seconds of waiting beats losing the whole app.

On Android, onTrimMemory plays the same role.

// In Application or an Activity: the higher the level, the more the OS wants memory back
override fun onTrimMemory(level: Int) {
    super.onTrimMemory(level)
    if (level >= ComponentCallbacks2.TRIM_MEMORY_RUNNING_LOW) {
        ModelManager.unloadIfIdle(seconds = 5)
    }
}

Three assumptions that bit me

• Stronger quantization doesn't solve everything. Activations and the KV cache shrink less than weights, and accuracy drops first.
• Physical RAM isn't what you get. As above, the app's share is smaller and shifts with conditions.
• Published or benchmark numbers won't appear as-is. Camera, networking, and background work compete with you, so re-measure in your real app.

What to try next

The smallest useful step: log os_proc_available_memory() at three points on a real device, before load, after load, and after inference. Writing those differences into rows two onward of your budget turns "it should fit" into numbers.

I no longer classify devices without those three readings.