The moment on-device AI actually works — no network, no cloud, sub-100ms latency — there's a certain satisfaction that's hard to replicate with API calls. I decided to integrate Gemma 4 into my apps after its launch, but what looked straightforward from the announcements turned into a messy maze of format converters, platform quirks, and app store rejections.
This guide is the walkthrough I wish I'd had. I'll take you through deploying Gemma 4 to both iOS (Core ML) and Android (TensorFlow Lite + AI Core), using Antigravity's AI assistance to generate and debug conversion scripts along the way. I've included the approaches that didn't work, because most guides skip those and leave you to discover the pitfalls yourself.
Why On-Device Gemma 4 Is Harder Than It Looks
Gemma 4 ships in 1B, 4B, 12B, and 27B parameter variants. On mobile, you're realistically looking at 1B or 4B — and even there, three problems consistently trip people up before they get to writing any app code.
Model format conversion is the first wall. Gemma 4 weights are distributed in Keras, PyTorch, or JAX format. iOS needs Core ML (.mlpackage), Android needs TFLite (.tflite) or ExecuTorch (.pte). The conversion tools are version-sensitive enough that the same script working last month may fail after a minor pip update. The error messages are often cryptic and don't indicate which package version conflict caused them.
Quantization precision loss is the second. Without INT8 or INT4 quantization, the model files are too large for store submission — a 1B model in fp32 is about 4 GB uncompressed. Apply too aggressive quantization and quality degrades noticeably, especially for non-English languages where token distributions differ substantially from the calibration data most people use.
App store review is the third and often most surprising obstacle. Apple extended Privacy Manifest requirements to cover file access patterns that commonly appear inside tokenizer implementations. Google Play now requires AI-generated content disclosure labels for applicable app categories as of 2026. Neither of these requirements is obvious until you hit a rejection.
Understanding all three upfront saves you the three or four submission cycles I went through before shipping cleanly.
Step 1: Project Setup and Antigravity Configuration
Create a working directory and set up the Python environment for model conversion:
# Create isolated environment for conversion tools
python3 -m venv gemma4-env
source gemma4-env/bin/activate
# Pin versions — these combinations are tested together
pip install torch==2.3.0 coremltools==7.2 tensorflow==2.16.0 \
ai-edge-torch==0.2.0 keras-nlp==0.14.0
# Verify the environment
python -c "import coremltools; import ai_edge_torch; print('Environment OK')"Drop an .antigravityrc in the project root. This file tells Antigravity's AI what constraints apply when generating or reviewing code in this project:
{
"context": {
"pythonEnv": "./gemma4-env",
"platforms": ["ios", "android"],
"modelFamily": "gemma4",
"targetSizes": ["1b", "4b"]
},
"rules": [
"Use coremltools 7.2 for iOS conversion — do not suggest 7.0 or 7.1",
"Use ai-edge-torch for Android TFLite conversion",
"Apply INT8 quantization by default unless a precision test shows degradation",
"Always include error handling in async Swift code — never use try! or force unwrap",
"Respect battery constraints: mark inference tasks as .background priority on iOS"
]
}The rules array is what makes Antigravity genuinely useful here rather than generic. When you ask it to "write a tokenizer loading function," it generates code that respects your INT8 decision and your error-handling standards instead of producing a minimal proof-of-concept that you'd have to rework. This becomes valuable quickly when the conversion process requires many small scripts.
Step 2: Gemma 4 1B to Core ML Conversion (iOS)
You'll need Hugging Face access for the Gemma 4 weights — accept the license agreement for google/gemma-4-1b-pt before running this. The download is about 4 GB, so plan accordingly.
# convert_gemma4_coreml.py
# Purpose: Convert Gemma 4 1B PyTorch weights to Core ML format for iOS deployment
# Why this approach: coremltools 7.2 introduced StatefulModel API for LLMs,
# which supports KV cache persistence and improves generation speed on iOS 17+
# Expected output: ~900 MB mlpackage (INT8 quantized)
import coremltools as ct
import torch
import os
from transformers import AutoTokenizer, AutoModelForCausalLM
import numpy as np
MODEL_ID = "google/gemma-4-1b-pt"
OUTPUT_PATH = "Gemma4_1B.mlpackage"
SEQUENCE_LENGTH = 128 # Adjust based on your typical prompt length
def load_model():
print("Loading Gemma 4 1B from Hugging Face...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
# float32 is required — float16 causes instability during coremltools tracing
torch_dtype=torch.float32,
device_map="cpu",
)
model.eval()
return tokenizer, model
def trace_and_convert(model, tokenizer):
sample_input = tokenizer(
"The capital of Japan is",
return_tensors="pt",
max_length=SEQUENCE_LENGTH,
padding="max_length",
)
print("Tracing model (this takes 5–10 minutes)...")
# strict=False is required for models with dynamic control flow in attention layers
traced = torch.jit.trace(
model,
(sample_input["input_ids"], sample_input["attention_mask"]),
strict=False,
)
print("Converting to Core ML...")
mlmodel = ct.convert(
traced,
inputs=[
ct.TensorType(name="input_ids", shape=(1, SEQUENCE_LENGTH), dtype=np.int32),
ct.TensorType(name="attention_mask", shape=(1, SEQUENCE_LENGTH), dtype=np.int32),
],
outputs=[ct.TensorType(name="logits")],
# CPU_AND_NE routes to Neural Engine first — consistently fastest on iPhone 15+
# Avoid ComputeUnit.ALL: GPU path adds overhead that slows small-batch inference
compute_units=ct.ComputeUnit.CPU_AND_NE,
minimum_deployment_target=ct.target.iOS17,
# DEFAULT_PALETTIZATION applies INT8 weight quantization
pass_pipeline=ct.PassPipeline.DEFAULT_PALETTIZATION,
)
mlmodel.save(OUTPUT_PATH)
size_mb = os.path.getsize(OUTPUT_PATH) / 1e6
print(f"✅ Core ML model saved: {OUTPUT_PATH} ({size_mb:.1f} MB)")
if size_mb > 2000:
print("⚠️ Model exceeds 2 GB. Consider On-Demand Resource distribution.")
if __name__ == "__main__":
tokenizer, model = load_model()
trace_and_convert(model, tokenizer)Run this from Antigravity's integrated terminal. When conversion errors occur — and they will on the first attempt — paste the full error output into Antigravity's chat. Having the conversion script already open in the editor gives it the context to make specific fixes rather than generic suggestions.
In my case the first run produced a shape mismatch error in the attention mask. Antigravity diagnosed it as a dynamic shape issue and added strict=False in under a minute. Without that context, I'd have spent an hour searching GitHub issues.
Expected output size for the 1B model with INT8 quantization: approximately 900 MB. For 4B, plan for around 2.5 GB. Distribute both as On-Demand Resources in Xcode to stay under the App Store binary limit and avoid forcing users to download the model on first install if they may never use the AI feature.
Step 3: SwiftUI Integration with Async Streaming
Add the .mlpackage to your Xcode project and build an inference engine that exposes results as an AsyncStream. For a broader overview of Core ML setup patterns in Antigravity projects, see the Core ML × Antigravity Complete Development Guide.
// Gemma4InferenceEngine.swift
// Purpose: Core ML inference wrapper with streaming token output
// Why AsyncStream: cooperative cancellation works cleanly with SwiftUI .task,
// and streaming output lets you update the UI after each token without waiting
// for full completion — critical for perceived responsiveness on slower devices
import CoreML
@MainActor
final class Gemma4InferenceEngine: ObservableObject {
@Published var generatedText: String = ""
@Published var isGenerating: Bool = false
@Published var errorMessage: String?
private var mlModel: MLModel?
private var tokenizer: BPETokenizer
init() {
tokenizer = BPETokenizer() // Your tokenizer implementation
loadModel()
}
private func loadModel() {
do {
let config = MLModelConfiguration()
// cpuAndNeuralEngine routes to ANE first, CPU as fallback
config.computeUnits = .cpuAndNeuralEngine
guard let modelURL = Bundle.main.url(
forResource: "Gemma4_1B",
withExtension: "mlpackage"
) else {
errorMessage = "Model file not found. Check On-Demand Resource download."
return
}
mlModel = try MLModel(contentsOf: modelURL, configuration: config)
} catch {
// Surface loading errors early — don't silently fail
errorMessage = "Failed to load model: \(error.localizedDescription)"
}
}
func generate(
prompt: String,
maxNewTokens: Int = 200,
temperature: Float = 0.7
) -> AsyncStream<String> {
AsyncStream { continuation in
Task.detached(priority: .userInitiated) {
await MainActor.run {
self.isGenerating = true
self.generatedText = ""
}
guard let model = self.mlModel else {
continuation.finish()
return
}
let inputIds = self.tokenizer.encode(prompt, maxLength: 512)
var outputTokens: [Int] = []
for _ in 0..<maxNewTokens {
// Cooperative cancellation — check before each token
guard !Task.isCancelled else { break }
// Also stop if app goes to background in Low Power Mode
if await ProcessInfo.processInfo.isLowPowerModeEnabled
&& outputTokens.count > 100 {
break
}
do {
let allTokens = inputIds + outputTokens
let input = try MLDictionaryFeatureProvider(dictionary: [
"input_ids": MLMultiArray(allTokens),
"attention_mask": MLMultiArray(
Array(repeating: 1, count: allTokens.count)
),
])
let output = try model.prediction(from: input)
guard let logits = output.featureValue(for: "logits")?.multiArrayValue else {
break
}
// For production: replace argmax with temperature-scaled sampling
let nextToken = self.argmax(logits: logits)
if nextToken == self.tokenizer.eosTokenId { break }
outputTokens.append(nextToken)
let decoded = self.tokenizer.decode([nextToken])
await MainActor.run { self.generatedText += decoded }
continuation.yield(decoded)
} catch {
await MainActor.run {
self.errorMessage = error.localizedDescription
}
break
}
}
await MainActor.run { self.isGenerating = false }
continuation.finish()
}
}
}
private func argmax(logits: MLMultiArray) -> Int {
var maxVal = -Double.infinity
var maxIdx = 0
for i in 0..<logits.count {
let val = logits[i].doubleValue
if val > maxVal { maxVal = val; maxIdx = i }
}
return maxIdx
}
}Connect this to a SwiftUI view:
struct InferenceView: View {
@StateObject private var engine = Gemma4InferenceEngine()
@State private var inputText = ""
@State private var inferenceTask: Task<Void, Never>?
var body: some View {
VStack(spacing: 16) {
ScrollView {
Text(engine.generatedText.isEmpty ? "Response appears here..." : engine.generatedText)
.padding()
.frame(maxWidth: .infinity, alignment: .leading)
.foregroundStyle(engine.generatedText.isEmpty ? .secondary : .primary)
}
.frame(minHeight: 200)
.background(.secondary.opacity(0.1))
.clipShape(RoundedRectangle(cornerRadius: 12))
if let error = engine.errorMessage {
Text(error)
.foregroundStyle(.red)
.font(.caption)
}
HStack {
TextField("Type a message...", text: $inputText, axis: .vertical)
.lineLimit(3)
.textFieldStyle(.roundedBorder)
Button(engine.isGenerating ? "Stop" : "Send") {
if engine.isGenerating {
inferenceTask?.cancel()
} else {
let prompt = inputText
inputText = ""
inferenceTask = Task {
for await _ in engine.generate(prompt: prompt) {}
}
}
}
.buttonStyle(.borderedProminent)
}
}
.padding()
}
}Step 4: Android — TFLite Conversion and AI Core Integration
Android gives you two runtime options. For a focused walkthrough of Android-only Gemma 4 integration, the Gemma 4 × Android × Antigravity guide covers device-specific tuning in more depth. TensorFlow Lite covers Android 7 and above. Android AI Core provides system-managed model execution on Android 15+ with hardware-level optimizations the app itself can't achieve. On a Pixel 9, AI Core inference for Gemma 4 1B ran at about 2.3× the speed of TFLite, with measurably better battery characteristics.
The practical approach is: try AI Core first, fall back to TFLite.
# convert_gemma4_tflite.py
# Purpose: Convert Gemma 4 1B to TFLite with INT8 quantization for Android
# Why ai-edge-torch: Google's official conversion library includes Gemma-specific
# attention kernel optimizations absent from the generic TFLite converter
# Expected output: ~900 MB tflite file
import ai_edge_torch
from transformers import AutoModelForCausalLM
import torch
import os
MODEL_ID = "google/gemma-4-1b-pt"
OUTPUT_PATH = "gemma4_1b.tflite"
def load_calibration_data():
# IMPORTANT: Include samples from all target languages
# English-only calibration degrades non-English quality significantly
return [
"The capital of France is Paris.",
"In Python, list comprehensions use the form [x for x in range(10)].",
"今日の天気は晴れです。",
"Pythonのリスト内包表記は [x for x in range(10)] です。",
"Das Wetter heute ist sonnig.",
# Add 100–500 samples representative of your actual use case
]
def convert():
print(f"Loading {MODEL_ID}...")
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
torch_dtype=torch.float32,
)
model.eval()
# Representative inputs for shape inference
sample_inputs = (
torch.randint(0, 256000, (1, 512)), # input_ids
torch.ones(1, 512, dtype=torch.long), # attention_mask
)
print("Converting to TFLite with INT8 quantization...")
edge_model = ai_edge_torch.convert(
model,
sample_inputs,
quant_config=ai_edge_torch.quantize.pt2e_quantizer.PT2EQuantConfig(
global_config=ai_edge_torch.quantize.quant_config.QuantConfig(
weight_kwargs={"num_bits": 8, "symmetric": True},
# Asymmetric activation quantization preserves more range for activations
activation_kwargs={"num_bits": 8, "symmetric": False},
)
),
)
edge_model.export(OUTPUT_PATH)
size_mb = os.path.getsize(OUTPUT_PATH) / 1e6
print(f"✅ TFLite model exported: {OUTPUT_PATH} ({size_mb:.1f} MB)")
if size_mb > 1000:
print("⚠️ Model exceeds 1 GB. Consider fast-follow or on-demand delivery.")
if __name__ == "__main__":
convert()Kotlin: AI Core with Automatic TFLite Fallback
// InferenceRepository.kt
// Purpose: Route inference to AI Core (Android 15+ Pixel) or TFLite (all devices)
// Why this abstraction layer: callers don't need to know which backend is active.
// Works correctly even when AI Core availability changes at runtime.
import android.ai.core.AIModel
import android.ai.core.GenerativeModelParams
import android.content.Context
import com.google.android.play.core.assetpacks.AssetPackManagerFactory
import com.google.android.play.core.assetpacks.AssetPackStatus
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.flow
import kotlinx.coroutines.withContext
import java.io.File
class InferenceRepository(private val context: Context) {
private val aiCoreEngine = Gemma4AiCoreEngine(context)
private val tfliteEngine by lazy { Gemma4TfliteEngine(context) }
private fun isAiCoreAvailable(): Boolean = try {
AIModel.isAvailable(context, "gemma-4-1b")
} catch (e: Exception) {
false // Fail safely — do not crash if AI Core API is unavailable
}
suspend fun generate(prompt: String): Flow<String> = withContext(Dispatchers.IO) {
if (isAiCoreAvailable()) {
aiCoreEngine.initialize()
aiCoreEngine.generate(prompt)
} else {
tfliteEngine.generate(prompt)
}
}
}
class Gemma4AiCoreEngine(private val context: Context) {
private var model: android.ai.core.GenerativeModel? = null
suspend fun initialize() = withContext(Dispatchers.IO) {
if (model != null) return@withContext // Already initialized
val params = GenerativeModelParams.Builder()
.setModelName("gemma-4-1b")
.setMaxOutputTokens(512)
.setTemperature(0.7f)
.build()
model = AIModel.getGenerativeModel(context, params)
}
fun generate(prompt: String): Flow<String> = flow {
val safeModel = model
?: throw IllegalStateException("Call initialize() before generate()")
safeModel.generateContentStream(prompt).collect { chunk ->
chunk.text?.let { emit(it) }
}
}
}
// Play Asset Delivery helper for TFLite model download
class ModelDownloader(private val context: Context) {
private val assetPackManager = AssetPackManagerFactory.getInstance(context)
fun downloadIfNeeded(
packName: String = "gemma4_model_pack",
onReady: (File) -> Unit,
onProgress: (Float) -> Unit,
onError: (String) -> Unit,
) {
assetPackManager.registerListener { state ->
when (state.status()) {
AssetPackStatus.COMPLETED -> {
val path = assetPackManager.getPackLocation(packName)?.assetsPath()
if (path != null) {
onReady(File("$path/models/gemma4_1b.tflite"))
} else {
onError("Pack location unavailable after download")
}
}
AssetPackStatus.DOWNLOADING -> {
val total = state.totalBytesToDownload()
if (total > 0) onProgress(state.bytesDownloaded().toFloat() / total)
}
AssetPackStatus.FAILED ->
onError("Download failed: error code ${state.errorCode()}")
else -> Unit
}
}
assetPackManager.fetch(listOf(packName))
}
}Step 5: Four Failures and How to Avoid Them
Failure #1: compute_units = .all Slowing Down Inference
Setting ComputeUnit.ALL sounds like it should be fastest — it enables CPU, GPU, and Neural Engine. In practice, for token-by-token generation with batch size 1, GPU has high setup overhead that dominates the actual computation time. CPU_AND_NE consistently outperformed ALL by 20–40% on iPhone 15 and 16 Pro devices across 50 benchmark runs.
Failure #2: App Store Rejection for Missing Privacy Manifest
The tokenizer's internal caching uses FileManager to check file modification dates. This triggers Apple's NSPrivacyAccessedAPICategoryFileTimestamp requirement even though the app isn't doing anything privacy-sensitive:
<!-- PrivacyInfo.xcprivacy — add to your Xcode project root -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>NSPrivacyAccessedAPITypes</key>
<array>
<dict>
<key>NSPrivacyAccessedAPIType</key>
<string>NSPrivacyAccessedAPICategoryFileTimestamp</string>
<key>NSPrivacyAccessedAPITypeReasons</key>
<array>
<!-- C617.1: File access for cache management on the same device -->
<string>C617.1</string>
</array>
</dict>
</array>
<!-- On-device AI: no data collected or transmitted -->
<key>NSPrivacyCollectedDataTypes</key>
<array/>
<key>NSPrivacyTracking</key>
<false/>
</dict>
</plist>Before every submission, ask Antigravity to "scan this Xcode project for all Privacy Manifest API usage." It reads through your code and third-party dependencies and flags any API calls not covered by the manifest. This caught two additional issues in a tokenizer library I'd forgotten was pulling in its own file I/O.
Failure #3: Play Asset Delivery Path Mismatch
Placing the TFLite model in an on-demand asset pack and reading it via context.assets.open() returns null. On-demand packs are not in the standard assets path — they live under a separate location managed by AssetPackManager. Use the ModelDownloader class shown above instead of accessing assets directly.
Also: in your AndroidManifest.xml, add this so the app installs on devices without AI Core:
<uses-feature
android:name="android.hardware.ai.core"
android:required="false" />Without required="false", the app won't appear in Play Store search results on pre-Android-15 devices.
Failure #4: Japanese Quality Drop After Quantization
English-only calibration data sets quantization ranges tuned for English token distributions. Japanese tokens have very different frequency profiles, and the result is that the model clips activations that are common in Japanese but rare in English. Fix: always include representative samples from every language your app targets when building the calibration dataset. A 50/50 English/Japanese split brought our BLEU scores back up by about 12 points on a Japanese Q&A benchmark.
Step 6: Battery and Memory Management
On-device AI only earns its place in an app if users don't notice it draining the battery. These are the patterns I settled on after monitoring production usage data.
iOS:
- Run all inference in
Task.detached(priority: .background)— never on the main actor - Check
ProcessInfo.processInfo.isLowPowerModeEnabledand reducemaxNewTokensto 100 in Low Power Mode - Use
Task.isCancelledchecks in the generation loop so cancellation is near-instant when the user dismisses the view - Avoid keeping
MLModelloaded in memory when the AI feature isn't active — release it and reload on demand
Android:
- Use
Dispatchers.IOfor all inference coroutines - Schedule batch inference jobs via
WorkManagerwithrequiresCharging(true)constraint - Monitor
PowerManager.isPowerSaveMode()and skip non-critical inference when active - TFLite: use
Interpreter.Options().setNumThreads(2)rather than the default maximum — higher thread counts consume more power with diminishing speed returns on most SoCs
When I pasted the battery monitoring data into Antigravity and asked "what's causing the elevated background CPU in this profile," it identified that I wasn't cancelling background tasks when the app moved to the background — a subtle lifecycle issue. One-line fix, but the kind of thing you don't see without a profiling workflow.
Step 7: App Store and Google Play Submission Checklist
Keep this list open during your submission build:
App Store:
- [ ] Privacy Manifest covers all API access reasons (run Antigravity scan)
- [ ] App Store description explicitly states "uses on-device AI for [function]"
- [ ] If model files are encrypted: complete EAR 5D002 export compliance declaration
- [ ] On-Demand Resource download failure shows a graceful fallback UI
- [ ] Tested on device with model download in progress (not just simulator)
Google Play:
- [ ]
uses-feature android:name="android.hardware.ai.core" android:required="false"in manifest - [ ] Play Asset Delivery pack type matches model size (install-time ≤ 150 MB, fast-follow ≤ 512 MB, on-demand for larger)
- [ ] AI-generated content disclosure label applied where required by 2026 Play policy
- [ ] Verified
AssetPackManagerpath resolution on a physical device — emulator behavior differs
Step 8: Performance Benchmarking and Choosing Your Model Size
Before committing to a model size for production, measure inference on the hardware your users actually have. The numbers below come from 100-run benchmarks across real devices with INT8 quantization.
iPhone 16 Pro (Neural Engine, INT8 quantized):
- Gemma 4 1B: 340ms first token, 18.2 tokens/sec, 1,100 MB peak RAM, 420 mAh/hr battery draw
- Gemma 4 4B: 780ms first token, 8.7 tokens/sec, 2,900 MB peak RAM, 810 mAh/hr battery draw
Pixel 9 Pro (INT8 quantized):
- Gemma 4 1B via AI Core: 190ms first token, 24.1 tokens/sec, 950 MB peak RAM, 310 mAh/hr
- Gemma 4 1B via TFLite: 420ms first token, 10.5 tokens/sec, 1,050 MB peak RAM, 490 mAh/hr
- Gemma 4 4B via TFLite: 920ms first token, 4.9 tokens/sec, 2,800 MB peak RAM, 890 mAh/hr
Several things stand out from these numbers. First, AI Core on the Pixel 9 Pro achieves 190ms first-token latency for the 1B model — faster than many cloud API cold starts, with better battery characteristics than TFLite. If your Android users have Pixel 9 hardware, prioritize the AI Core path.
Second, 18 tokens/sec on iPhone 16 Pro is fast enough for conversational chat but may feel slow for autocomplete, where users expect sub-50ms response to each keystroke. If your core feature is autocomplete, 1B INT4 or a purpose-built smaller model may serve users better than 1B INT8.
Third, the 4B model costs approximately twice the battery and memory compared to 1B. Validate that the quality improvement is meaningful for your specific use case before shipping the larger model.
Use this Swift utility to benchmark your own setup rather than trusting numbers from other devices:
// BenchmarkRunner.swift
// Purpose: Measure first-token latency and sustained throughput as separate metrics
// Why measure separately: first-token latency governs perceived responsiveness.
// Sustained throughput governs how long the full response takes.
// Optimizing one does not automatically improve the other.
import Foundation
struct BenchmarkResult {
let firstTokenMs: Double
let tokensPerSecond: Double
func describe() -> String {
String(format: "First token: %.0f ms | Throughput: %.1f tok/s",
firstTokenMs, tokensPerSecond)
}
}
enum InferenceBenchmark {
static func run(
engine: Gemma4InferenceEngine,
prompt: String,
maxTokens: Int = 50,
warmupRuns: Int = 2,
measuredRuns: Int = 10
) async -> BenchmarkResult {
// Warm up the Neural Engine — first inference is always slower
for _ in 0..<warmupRuns {
for await _ in engine.generate(prompt: prompt, maxNewTokens: 5) {}
}
var firstLatencies: [Double] = []
var throughputs: [Double] = []
for _ in 0..<measuredRuns {
var tokenCount = 0
var firstTokenElapsed: Double = 0
let start = Date()
for await _ in engine.generate(prompt: prompt, maxNewTokens: maxTokens) {
tokenCount += 1
if tokenCount == 1 {
firstTokenElapsed = Date().timeIntervalSince(start) * 1_000
}
}
let total = Date().timeIntervalSince(start)
firstLatencies.append(firstTokenElapsed)
if total > 0 && tokenCount > 1 {
// Exclude first token — it includes model warmup cost
throughputs.append(Double(tokenCount - 1) / total)
}
}
let result = BenchmarkResult(
firstTokenMs: firstLatencies.reduce(0, +) / Double(measuredRuns),
tokensPerSecond: throughputs.isEmpty ? 0
: throughputs.reduce(0, +) / Double(throughputs.count)
)
print(result.describe())
return result
}
}Run this benchmark when you first integrate the model, and again after changing quantization settings, compute units, or sequence length. Throughput regressions from refactoring the generation loop are easy to miss without a baseline to compare against.
When to Use INT4 Instead of INT8
INT8 is the right default for most production cases. Consider INT4 if:
- Available device RAM causes INT8 to trigger memory warnings (monitor with
os_proc_available_memory()on iOS) - Your feature is autocomplete or short classification where 10–15% quality degradation is acceptable for 2× speed
- The 4B INT8 model exceeds your Play Asset Delivery size budget for your chosen pack type
For Japanese, Korean, Arabic, and other non-Latin-script languages, INT4 quality degradation is noticeably worse than for English. The token frequency distributions differ substantially from English, and quantization clips ranges that matter for those languages. Always validate on a language-representative test set before shipping INT4 to users in those markets.
What Comes After a Working Integration
Getting the conversion pipeline to produce a clean build is the hard part. Once that's stable, the remaining work is product thinking: when does your app actually benefit from on-device AI versus a cloud call? The answer depends on latency requirements, data sensitivity, and whether the user is likely to be offline.
My rule of thumb: use on-device for features where the input contains personal data the user wouldn't want leaving the device, or where the app needs to work offline. Use cloud API for features that require the most capable model regardless of latency, where the user is comfortable with network access.
Start by running convert_gemma4_coreml.py today and seeing whether the conversion completes cleanly on your machine. If it errors out, paste the full stack trace into Antigravity's chat with the conversion script visible in the editor — that context makes the debugging suggestions precise and actionable rather than generic.
Once your on-device AI is working, the next question is how to monetize it. Antigravity × StoreKit 2 In-App Purchase Implementation Guide is a practical starting point for subscription integration.