Calling Local LLMs from Antigravity — Ollama and LM Studio Integration in Practice

Concrete Setup — Ollama

To call Ollama from Antigravity, add it as a custom provider in Antigravity's settings, with an OpenAI-compatible endpoint.

Provider Name: Ollama (local)
Base URL: http://localhost:11434/v1
API Key: ollama  (any string is fine)
Model: gemma3:27b  (the model you previously ollama-pulled)

The trailing /v1 on the Base URL matters. Without it, Antigravity's OpenAI Chat Completions request hits a 404.

The API Key isn't actually used by Ollama, but Antigravity's validation rejects empty values. Any string works — ollama, dummy, anything.

To reach Ollama on a different machine, allow external connections on the Ollama side. Default is localhost-only, so launch with OLLAMA_HOST=0.0.0.0:11434 ollama serve. Within the same LAN, you can then point Antigravity at http://192.168.1.10:11434/v1.

For security: do not expose this to external LAN or WAN. At minimum, route via VPN or SSH tunnel. Anything less means anyone can borrow your machine for free inference.

Concrete Setup — LM Studio

In LM Studio, the "Local Server" tab launches the API server. The default port is 1234 and it also speaks OpenAI-compatible API.

Provider Name: LM Studio (local)
Base URL: http://localhost:1234/v1
API Key: lm-studio
Model: name of the currently-loaded model

A handy LM Studio feature: server-launch system instructions are configurable in the GUI. Forcing Japanese responses, tweaking quantization behavior — all without changing prompts on the Antigravity side.

The catch: the LM Studio API server only runs while the app is open, so it's not for long-running production. A common pattern is "LM Studio for daytime experimentation, Ollama for overnight batches."

Model Selection for Antigravity Use

The models I'm currently using from Antigravity. Optimal choice varies by your hardware, but as a reference:

Main driver: Gemma 3 27B. On an M2 Pro Mac mini with 32GB RAM, the Q4_K_M quantization runs comfortably. Code completion, prose editing, summarization — quality approaches cloud LLM levels for general tasks.

For long contexts: quantized Llama 3.3 70B. Noticeably slower, so not for interactive use. Reserved for batch jobs that can take their time.

For lightweight tasks: Phi-3.5 mini (3.8B). Filename normalization, simple classification, JSON formatting — "mechanical transformation" tasks. Fast enough to run as a constantly-available helper agent.

The most common selection mistake: "go as big as possible." A model that just fits your RAM swaps mid-inference and crawls. Pick a model size around 70% of effective RAM for safety.

Routing Between Cloud and Local

One of Antigravity's strengths is letting providers coexist. I rule-encode which provider to use per project.

Projects with sensitive data: all inference pinned to local LLMs via Ollama. With the default provider set to Ollama at the Antigravity project level, accidental cloud sends are prevented.

Low-sensitivity OSS projects: default to cloud (Gemini or Claude), with local taking lighter tasks like code completion. Cost optimization meets quality.

A useful technique across projects: write the defaultProvider into Antigravity's project config (e.g., .antigravity/config.json — paths vary by version). Opening the project auto-selects the right provider.

Local LLM Constraint 1: Context Length

The classic local-LLM limit. Cloud LLMs support 1M-token contexts; local LLMs typically max out at 8k–32k. Some Gemma 3 variants extend to 128k, but RAM consumption explodes — practical range is limited.

Three workarounds:

Use Antigravity's context-splitting features. When operating on a long file, Antigravity has a "extract relevant portions and pass only those to the model" mode. Enable it and you stay inside local-LLM context limits more often.

Pre-summarize before sending to local. For long meeting-notes summarization: summarize chapter by chapter on cloud LLM, then merge on local LLM. Antigravity lets you mix providers, so this two-stage approach is easy to wire.

Concentrate context-light tasks on local. "Refactor this single function" needs less context than "review this whole codebase." Mind your task granularity and the context limit becomes much less of a wall.

Local LLM Constraint 2: Function Calling Quality

Cloud function calling has gotten dramatically smarter; local LLMs lag here. Even Gemma 3 27B struggles with complex tool chains.

Two responses:

Don't ask local LLMs to do complex function calling. Keep agentic patterns in the cloud; local handles "single-shot response generation."

Use a function-calling alternative. Instead of OpenAI-style tool calls, use "force JSON output" prompts. Write "respond with only JSON in this exact shape: { ... }" and parse the response yourself. Most local models do this stably.

If you write Antigravity plugins, building a thin local-LLM wrapper is worthwhile. One plugin handles "prompt → JSON output → parse → invoke function" end to end.

Actually Writing the Thin JSON Wrapper

I said "force JSON output and parse it yourself," but the naive implementation always breaks. Models want to add a preamble. They want to wrap things in code fences. An implementation that hands the raw response straight to json.loads fails something like one time in ten, in my experience. A 10% failure rate in an overnight batch means cleanup waiting for you in the morning.

The wrapper I put in front of routine Ollama work does exactly three things. Pass a schema to Ollama's format to constrain the structure. If it still comes back broken, show the model its own broken output and let it fix it once. If that fails too, give up and raise to the caller.

# local_json.py — receive structured data from a local model, safely
import json
import urllib.request
from typing import Any
 
BASE_URL = "http://localhost:11434"
MODEL = "gemma3:27b"
 
 
class LocalJSONError(RuntimeError):
    """Raised when even the repair retry fails to produce valid JSON"""
 
 
def _chat(messages: list[dict], schema: dict | None) -> str:
    payload: dict[str, Any] = {
        "model": MODEL,
        "messages": messages,
        "stream": False,
        "options": {"temperature": 0},
    }
    if schema is not None:
        # Ollama constrains output to this JSON Schema
        payload["format"] = schema
 
    req = urllib.request.Request(
        f"{BASE_URL}/api/chat",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=120) as res:
        return json.loads(res.read())["message"]["content"]
 
 
FENCE = "`" * 3   # assembled this way purely for this article's formatting
 
 
def _salvage(text: str) -> str:
    """Pull the payload out even when fences or a preamble are attached"""
    if FENCE in text:
        text = text.split(FENCE)[1]
        if text.startswith("json"):
            text = text[4:]
    start = min((i for i in (text.find("{"), text.find("[")) if i != -1), default=-1)
    if start == -1:
        return text.strip()
    end = max(text.rfind("}"), text.rfind("]"))
    return text[start:end + 1].strip() if end > start else text[start:].strip()
 
 
def ask_json(prompt: str, schema: dict, *, repair: bool = True) -> Any:
    messages = [{"role": "user", "content": prompt}]
    raw = _chat(messages, schema)
 
    try:
        return json.loads(_salvage(raw))
    except json.JSONDecodeError as first_error:
        if not repair:
            raise LocalJSONError(f"not parseable as JSON: {raw[:120]}") from first_error
 
        # Show the model its own broken output. Repeating the instruction alone won't fix it.
        messages += [
            {"role": "assistant", "content": raw},
            {"role": "user", "content":
                f"The output above could not be parsed as JSON ({first_error.msg}). "
                "Return valid JSON only, with no explanation or preamble."},
        ]
        retried = _chat(messages, schema)
        try:
            return json.loads(_salvage(retried))
        except json.JSONDecodeError as second_error:
            raise LocalJSONError(
                f"repair retry also failed: {retried[:120]}"
            ) from second_error
 
 
if __name__ == "__main__":
    schema = {
        "type": "object",
        "properties": {
            "category": {"type": "string",
                          "enum": ["bug", "feature", "question", "other"]},
            "confidence": {"type": "number"},
            "reason": {"type": "string"},
        },
        "required": ["category", "confidence"],
    }
    result = ask_json(
        "Classify the following review. Output JSON only.\n"
        "'Crashes immediately on launch. Reinstalling changes nothing.'",
        schema,
    )
    print(result)

Three things carry the weight here.

Passing a schema to format is the first line of defense — dramatically more reliable than asking for JSON in the prompt. Adding enum also kills casing drift, the kind where Bug and bug both show up in your dataset.

_salvage() is the second. Even with a schema, preambles and fences occasionally slip through. It naively slices from the first { to the last }, but that alone cut my failures noticeably.

For the repair retry, the essential part is including the broken output in the conversation. Re-sending the same instruction just reproduces the same breakage. Shown its own output, the model corrects it a decent share of the time.

I cap the retry at one. An input that isn't fixed on the second attempt usually isn't fixed on the third either. Rather than burning time, raise and let the caller route to the cloud. For light work like classification, I catch this exception and fall back to a cloud model directly. Local handles roughly nine cases in ten, the cloud picks up the remainder. That shape has needed the least babysitting of anything I've tried.

Local LLM Constraint 3: Speed

The last constraint is raw inference speed. Gemma 3 27B on Mac mini M2 Pro generates roughly 12–18 tokens/sec. Several times slower than cloud LLMs, which makes interactive use feel sluggish.

The answer is task allocation. Real-time tasks (code completion, dialog) → cloud. Batch tasks (overnight automation, log summarization, content generation) → local.

My site automation runs in the small hours, so a slower model doesn't matter. Local LLMs do their work in the quiet hours and keep API costs down. The division has settled into a stable rhythm.

Decide Upgrades on Your Own Numbers, Not Published Benchmarks

New local models land every few weeks, and each time the question is whether to switch. Published benchmark scores turned out to be nearly useless for that call. Those numbers come from high-end GPU rigs; they say very little about how a Q4-quantized build feels on my Mac mini.

So I keep a small benchmark and run it identically every time I swap models. It reports two things only: time to first token (TTFT), and tokens per second during generation. The first is the wait you feel in conversation; the second decides how long an overnight batch actually takes.

# bench_local.py — measure the OpenAI-compatible endpoint of Ollama / LM Studio
import json, statistics, time
import urllib.request
 
BASE_URL = "http://localhost:11434/v1"   # LM Studio: http://localhost:1234/v1
API_KEY = "ollama"                        # unused by Ollama, but blanks get rejected
MODEL = "gemma3:27b"
RUNS = 5
 
PROMPT = (
    "Explain what this function does in three sentences.\n\n"
    "def merge(a, b):\n    return {**a, **b}\n"
)
 
 
def one_run() -> tuple[float, float, int]:
    """Time one generation, returning (ttft_sec, generation_sec, output_tokens)"""
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT}],
        "stream": True,
        "options": {"temperature": 0, "seed": 42},  # pinned for comparability
    }).encode()
 
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
 
    start = time.perf_counter()
    ttft = None
    tokens = 0
 
    with urllib.request.urlopen(req) as res:
        for raw in res:
            line = raw.decode().strip()
            if not line.startswith("data: "):
                continue
            payload = line[6:]
            if payload == "[DONE]":
                break
            delta = json.loads(payload)["choices"][0]["delta"]
            if not delta.get("content"):
                continue
            if ttft is None:
                ttft = time.perf_counter() - start   # first token out
            tokens += 1
 
    total = time.perf_counter() - start
    return ttft, total - ttft, tokens
 
 
def pct(values: list[float], p: float) -> float:
    ordered = sorted(values)
    return ordered[min(int(len(ordered) * p), len(ordered) - 1)]
 
 
if __name__ == "__main__":
    one_run()  # discard: the first run includes model load time
 
    ttfts, rates = [], []
    for i in range(RUNS):
        ttft, gen_sec, tokens = one_run()
        ttfts.append(ttft)
        rates.append(tokens / gen_sec if gen_sec > 0 else 0)
        print(f"run {i + 1}: TTFT {ttft:.2f}s / {rates[-1]:.1f} tok/s / {tokens} tokens")
 
    print(f"\n{MODEL}")
    print(f"  TTFT   p50 {statistics.median(ttfts):.2f}s  p95 {pct(ttfts, 0.95):.2f}s")
    print(f"  tok/s  p50 {statistics.median(rates):.1f}   p95(slow side) {pct(rates, 0.05):.1f}")

Temperature and seed are pinned in options so that variation in output length doesn't jitter the tok/s figure. The first run is discarded because model load time contaminates TTFT — skip that and your first measurement reads tens of seconds and wrecks the average.

Note that tok/s is reported at the bottom 5%, not p95. Fast outliers are pleasant but irrelevant to the decision. How far the slow side sags is what determines batch duration.

On my Mac mini M2 Pro with 32GB of RAM, the numbers settle roughly like this:

The row that matters most is the second one. Same 27B parameters, but stepping up to Q8 crosses the effective memory ceiling, swapping begins, and the numbers move to a different planet. The "stay under 70% of effective RAM" rule of thumb earlier in this article is derived from exactly this measurement. I took that detour myself once — loosened quantization chasing quality, lost an order of magnitude of speed, and quietly stopped using the model.

Now, whenever a new model lands, I run this script first. If the numbers match or beat the incumbent and output quality hasn't regressed, I switch. That single check has been enough.

When Inference Is Too Slow, Suspect the Device Before the Model

I once loaded a new model, ran the benchmark from the previous section, and got roughly a tenth of the tokens per second I expected. Same quantization, similar parameter count, wildly different numbers. I spent a while blaming the model. The real cause was that inference had fallen back entirely to the CPU.

The awkward part is that nothing errors out. Responses still come back correctly — they just crawl. So it is easy to conclude "this model is too heavy for my machine" and move on. I threw away a perfectly good model that way once.

Check where Ollama is actually running

With a model loaded, ollama ps reports the execution target in the PROCESSOR column. One glance is enough.

# Warm the model up first, then check where it landed
ollama run gemma3:27b "warmup" > /dev/null
ollama ps

NAME          SIZE     PROCESSOR         UNTIL
gemma3:27b    18 GB    100% GPU          4 minutes from now

100% GPU is what you want. 100% CPU means the GPU is not being used at all, and a split like 42%/58% CPU/GPU means only some layers made it onto the GPU. Once a split happens, the speed is in a different category from a full offload. When benchmark numbers suddenly collapse, this column has usually changed.

When I see a split, the first thing I suspect is another model still resident and holding memory. ollama ps lists every loaded model, so if something unexpected is sitting there, ollama stop frees it. That alone often restores a full offload.

If the split persists, pin the layer count explicitly to isolate the cause.

# num_gpu sets how many layers go to the GPU (0 forces CPU-only)
curl -s http://localhost:11434/api/generate -d '{
  "model": "gemma3:27b",
  "prompt": "ping",
  "options": {"num_gpu": 99}
}' > /dev/null && ollama ps

If a generous value still leaves you with a split, memory is the constraint and no amount of layer tuning will fix it. Dropping a quantization level gives a much cleaner result than fighting the layer count. The 27B Q8 row in the earlier table was the same phenomenon, viewed from a different angle.

Checking GPU availability from your own scripts

Some setups run Python inference directly alongside Ollama. On a Linux box with an NVIDIA GPU, a False from torch.cuda.is_available() almost always traces back to one of three things.

The first row is the most common by a wide margin. An environment built from a plain pip install torch can end up with a CPU-only build, and no amount of driver work will change the answer. The version string settles it in seconds.

On Apple Silicon, torch.cuda is always False by design. The property to check is torch.backends.mps, and confusing the two sends you chasing a problem that does not exist. To settle the question in one shot, I run this before any benchmark.

# gpu_check.py — confirm the execution target before you measure anything
import platform
import torch
 
print(f"torch {torch.__version__} / {platform.machine()}")
 
if torch.cuda.is_available():
    print(f"CUDA: {torch.cuda.get_device_name(0)}")
elif torch.backends.mps.is_available():
    print("MPS: the Apple Silicon GPU is available")
else:
    # The version string distinguishes a CPU-only build from a visibility problem
    reason = ("a CPU-only build is installed" if "+cpu" in torch.__version__
              else "a GPU build is installed, but no device is visible")
    print(f"No accelerator — {reason}")

The ordering matters more than the checks themselves. Rather than benchmarking and then hunting for the reason something felt slow, confirm the execution target first and benchmark second. Since adopting that order, the time I spend evaluating models has dropped noticeably. Before you question a number, confirm where that number was produced.

Realistic Hardware Choices

If you're going to use local LLMs through Antigravity seriously, hardware matters. Three price tiers based on my experience:

Cheapest: leverage an existing M-series Mac. MacBook Pro M3 Pro or better with 32GB+ RAM runs Gemma 3 27B–class models usefully. No new investment to start.

Mid-range dedicated: Mac mini M2 Pro / M4 Pro with 32–64GB RAM. Around $2,000–$3,000 buys a 24/7 inference server. This is what I use.

High-end: Apple Silicon Mac Studio (96GB+) or a Linux workstation with NVIDIA RTX 4090 / 5090. For comfortable 70B-class operation. ~$7,000–$10,000 investment, but if you're spending $300+/month on cloud LLMs, ROI lands within a year.

Model the Local-vs-Cloud Breakeven in Monthly Terms

Listing hardware tiers doesn't answer the actual question: should you buy one? Before I turned a Mac mini into an inference server, I put three numbers onto the same monthly scale.

First, the local monthly cost. Purchase price divided by an amortization window, plus electricity. Even running 24/7, idle draw is modest, so use a realistic average rather than peak wattage.

Second, the cloud monthly cost: calls per day × average tokens per call × unit price. Most people estimate using input tokens only and land wide of the mark. Always add the output side — it's usually the more expensive half.

Third, the share you can actually offload. This is the crux. Moving everything local would make this simple, but in practice the hard judgments stay in the cloud. For me, roughly 70% of calls by count moved local.

# breakeven.py — put a local inference server and cloud API on the same monthly scale
from dataclasses import dataclass
 
 
@dataclass
class Local:
    hardware_usd: int        # purchase price
    amortize_months: int     # I use 36 months
    watt_avg: float          # average draw (W)
    usd_per_kwh: float = 0.17
 
    def monthly(self) -> float:
        depreciation = self.hardware_usd / self.amortize_months
        power = self.watt_avg * 24 * 30 / 1000 * self.usd_per_kwh
        return depreciation + power
 
 
@dataclass
class Cloud:
    calls_per_day: int
    in_tokens: int           # average input tokens per call
    out_tokens: int          # average output tokens per call
    usd_per_1k_in: float
    usd_per_1k_out: float
 
    def per_call(self) -> float:
        return (self.in_tokens / 1000 * self.usd_per_1k_in
                + self.out_tokens / 1000 * self.usd_per_1k_out)
 
    def monthly(self, offload_ratio: float = 0.0) -> float:
        """offload_ratio: share of calls diverted to local"""
        calls = self.calls_per_day * 30 * (1 - offload_ratio)
        return calls * self.per_call()
 
 
def report(local: Local, cloud: Cloud, offload: float) -> None:
    cloud_only = cloud.monthly(offload_ratio=0.0)
    hybrid = local.monthly() + cloud.monthly(offload_ratio=offload)
    diff = cloud_only - hybrid          # positive means hybrid is cheaper
 
    print(f"Cloud only    : ${cloud_only:>9,.2f}/mo")
    print(f"Local + cloud : ${hybrid:>9,.2f}/mo (hardware ${local.monthly():,.2f})")
    print(f"Delta         : ${diff:>+9,.2f}/mo")
 
    # Breakeven: where savings from offloaded calls equal the hardware's monthly cost
    breakeven_calls = local.monthly() / (30 * offload * cloud.per_call())
    print(f"Breakeven     : ~{breakeven_calls:,.0f} calls/day "
          f"(currently {cloud.calls_per_day})")
 
    if diff > 0:
        print(f"-> Hybrid wins. Hardware pays for itself in "
              f"{local.hardware_usd / diff:.1f} months")
    else:
        print("-> At this volume, cost alone does not justify the purchase")
 
 
if __name__ == "__main__":
    mac_mini = Local(hardware_usd=2_000, amortize_months=36, watt_avg=25)
    api = Cloud(calls_per_day=400, in_tokens=3_000, out_tokens=800,
                usd_per_1k_in=0.003, usd_per_1k_out=0.012)
    report(mac_mini, api, offload=0.7)

Replace the unit prices with the current published rates for whichever model you use. The values above exist to show the shape of the calculation, not to be quoted.

Running three volume bands through it (3,000 input / 800 output tokens per call, an effective $0.0186 per call) gives this:

The surprise, when I actually ran the numbers, was that breakeven sits at about 150 calls per day — far lower than I'd assumed. My gut said "you need thousands of calls before this pays." In fact, if you're pushing 3,000-token prompts 150 times a day, you're already at parity.

But that breakeven point tracks average prompt length directly. Rerun the same model with 300 input / 100 output tokens and the per-call cost falls to $0.0021, pushing breakeven up to roughly 1,300 calls per day. Decide on token volume, not call count. I estimated on call count alone the first time and was off by an order of magnitude.

One more caveat on the middle row. A 20.5-month payback does fit inside a 36-month amortization window. But given how fast local models turn over, I don't put much faith in a plan that takes nearly two years to break even — by then, models with entirely different memory requirements will likely be the norm. My practical line is whether payback lands under a year, as in the third row.

Which also means: if you're in the middle band, cost shouldn't be your deciding factor. I didn't buy the Mac mini over a monthly delta — I bought it for the single fact that it let me handle data I couldn't send to an external API. If you're deliberating purely on price, you can usually afford to wait.

Production Monitoring

If you're running local LLMs in production, basic monitoring is non-negotiable. The metrics I watch:

GPU / Neural Engine temperature and utilization. Long sustained loads can trigger thermal throttling and inference suddenly slows. Tools like asitop for periodic checks.

Memory usage and swap activity. Once swap kicks in, inference slows by orders of magnitude. Adjust model size to stay below the threshold.

Median and p95 response time. Looking only at median hides occasional latency spikes. p95 reveals the user-perceived bad experiences.

A simple Grafana dashboard or homemade visualization makes problem isolation much faster when something breaks.

When Switching to a Local Model Makes Every Response Turn English

The first thing that trips people up after wiring in a local LLM: task summaries that came back in your chosen language on the cloud model suddenly arrive in English the moment you point Antigravity at Ollama.

Your Antigravity settings are not being ignored. What happens is that smaller models tend to drop the language instruction from the system prompt. At the Gemma 3 4B scale, a single line saying "respond in Japanese" is not enough — the model is pulled around by where that instruction sits and by the language mix of the prompt itself.

Working through this on my own machine as an indie developer, two changes finally made it stable.

1. Put the output-language instruction at the end of each request, not only in the system prompt. The model reacts far more strongly to recent tokens
2. Include exactly one few-shot example in the target language. A single example is usually enough to lock the response language for the rest of the session

The opposite failure also exists: sometimes you do not want code comments translated. In that case, state both rules explicitly — prose in one language, code comments in another. Specify only one, and the model will drag everything toward it.

For where those language settings actually live, see "Setting Up Antigravity 2.0 with a Japanese UI", and if you want the whole model runtime pinned inside a container for reproducibility, "Antigravity DevContainer: A Complete Setup Guide" covers that ground.

Minimum Steps for Tomorrow

The smallest path to start tomorrow:

Install Ollama (brew install ollama), pull a small model with ollama pull gemma3:4b, launch with ollama serve, then add the custom provider in Antigravity using the settings above. Run a single text-generation task in Antigravity and confirm a response.

That's about an hour of work. Once it's working, climb up to bigger models and harder tasks.

The realistic framing: local LLMs are a complement to cloud LLMs, not a replacement. An Antigravity setup that handles both confidently widens the kind of work you can take on. I started accepting confidential projects only after building this combination. Hopefully this writeup helps anyone facing a similar wall.