Building Production-Grade Voice Agents with Gemini Live API in Antigravity — Bidirectional Audio, Screen Share, and Latency Patterns

Code Example 1: Resilient Live API Session Lifecycle

The first code example spins up a Live API session that recovers automatically from network drops. It's meant to run inside the Orchestrator agent. Instead of swallowing errors, it enforces explicit state machine transitions and exponential backoff.

// orchestrator/live-session.ts
// Purpose: Keep a Gemini Live API WebSocket session alive across
// transient disconnects and forced server-side closes.
 
import { GoogleGenAI, LiveSession, Modality } from "@google/genai";
 
type SessionState = "idle" | "connecting" | "live" | "retrying" | "closed";
 
export class LiveVoiceSession {
  private ai: GoogleGenAI;
  private session: LiveSession | null = null;
  private state: SessionState = "idle";
  private retries = 0;
  private readonly maxRetries = 3;
 
  constructor(apiKey: string, private onAudio: (pcm: ArrayBuffer) => void) {
    this.ai = new GoogleGenAI({ apiKey });
  }
 
  async start(): Promise<void> {
    if (this.state \!== "idle") {
      throw new Error(`cannot start in state: ${this.state}`);
    }
    this.state = "connecting";
 
    try {
      this.session = await this.ai.live.connect({
        model: "gemini-2.5-flash-live",
        config: {
          responseModalities: [Modality.AUDIO],
          speechConfig: {
            voiceConfig: { prebuiltVoiceConfig: { voiceName: "Aoede" } },
          },
          systemInstruction:
            "You are a calm customer support agent. Stop speaking immediately if the user starts talking.",
        },
        callbacks: {
          onmessage: (msg) => this.handleMessage(msg),
          onerror: (err) => this.handleError(err),
          onclose: (ev) => this.handleClose(ev),
        },
      });
      this.state = "live";
      this.retries = 0;
    } catch (err) {
      this.state = "closed";
      throw err; // caller logs and alerts
    }
  }
 
  private handleMessage(msg: unknown) {
    const data = extractAudioChunk(msg);
    if (data) this.onAudio(data);
  }
 
  private async handleError(err: Error) {
    console.error("[LiveSession] error", err.message);
    if (this.retries < this.maxRetries && this.state === "live") {
      this.state = "retrying";
      this.retries++;
      const backoff = Math.min(1000 * 2 ** this.retries, 8000);
      await new Promise((r) => setTimeout(r, backoff));
      this.state = "idle";
      await this.start();
    } else {
      this.state = "closed";
    }
  }
 
  private handleClose(_ev: CloseEvent) {
    if (this.state === "live") {
      // Server forced the close — attempt one more recovery.
      this.handleError(new Error("server closed"));
    }
  }
 
  async stop(): Promise<void> {
    this.state = "closed";
    await this.session?.close();
    this.session = null;
  }
}
 
function extractAudioChunk(msg: unknown): ArrayBuffer | null {
  const parts = (msg as any)?.serverContent?.modelTurn?.parts ?? [];
  for (const p of parts) {
    if (p?.inlineData?.mimeType?.startsWith("audio/")) {
      return base64ToArrayBuffer(p.inlineData.data);
    }
  }
  return null;
}
 
function base64ToArrayBuffer(b64: string): ArrayBuffer {
  const bin = atob(b64);
  const buf = new Uint8Array(bin.length);
  for (let i = 0; i < bin.length; i++) buf[i] = bin.charCodeAt(i);
  return buf.buffer;
}

The intentional design decisions here are a forced state machine, exponential backoff for reconnection, and explicit awareness of server-side closes. The official sample leaves state management to the developer. In production, without a typed state machine, onclose and onerror race each other and you end up with two concurrent sessions.

The expected behavior: a successful start() produces a setupComplete server event, audio chunks arrive in the callback roughly 600ms after the first user utterance (PCM16 at 20ms/chunk), and disconnects trigger up to three retries at 2s / 4s / 8s backoff.

Code Example 2: Audio Playback Queue with Barge-In

The second example is the Edge-layer playback queue. This is where beginners get stuck. Naively chaining AudioBufferSourceNode instances gives you no way to stop the model when the user starts talking.

// edge/audio-queue.ts
// Purpose: Play back model audio chunks seamlessly and stop playback
// immediately when the user begins speaking (barge-in).
 
export class AudioQueue {
  private ctx: AudioContext;
  private sources: AudioBufferSourceNode[] = [];
  private nextStartTime = 0;
 
  constructor(private sampleRate = 24000) {
    this.ctx = new AudioContext({ sampleRate });
  }
 
  enqueue(pcm16: ArrayBuffer): void {
    const float32 = pcm16ToFloat32(pcm16);
    const buffer = this.ctx.createBuffer(1, float32.length, this.sampleRate);
    buffer.copyToChannel(float32, 0);
 
    const src = this.ctx.createBufferSource();
    src.buffer = buffer;
    src.connect(this.ctx.destination);
 
    const startAt = Math.max(this.ctx.currentTime, this.nextStartTime);
    src.start(startAt);
    this.nextStartTime = startAt + buffer.duration;
    this.sources.push(src);
 
    src.onended = () => {
      this.sources = this.sources.filter((s) => s \!== src);
    };
  }
 
  interrupt(): void {
    // Barge-in: stop every active source immediately.
    for (const s of this.sources) {
      try {
        s.stop();
      } catch {
        // Source already finished — safe to ignore.
      }
    }
    this.sources = [];
    this.nextStartTime = this.ctx.currentTime;
  }
}
 
function pcm16ToFloat32(buf: ArrayBuffer): Float32Array {
  const view = new DataView(buf);
  const out = new Float32Array(buf.byteLength / 2);
  for (let i = 0; i < out.length; i++) {
    out[i] = view.getInt16(i * 2, true) / 0x8000;
  }
  return out;
}

Why track nextStartTime explicitly? Calling src.start(0) naively produces audible clicks at chunk boundaries. Scheduling each source against AudioContext.currentTime gives you gapless playback.

Why swallow exceptions silently? AudioBufferSourceNode.stop() throws on an already-finished source. Since barge-in is a latency race, the try/catch is cheaper than checking each source's state, and the UX impact is zero.

Pair this with a VAD that runs in the Edge layer (browser onaudioprocess + RMS threshold) and you can fire interrupt() without waiting on the Live API round trip. User speech to AI silence lands under 50ms of latency.

Code Example 3: Type-Safe Function Calling

The third example bridges Live API's function calling to internal APIs. This is the most safety-critical part of the stack. For a deeper dive into function-calling schema design on its own, see Mastering Gemini Function Calling with Antigravity (Python).

// orchestrator/tool-router.ts
// Purpose: Route Live API tool calls into internal APIs with
// type validation, timeouts, and per-session cost enforcement.
 
import type { LiveSession } from "@google/genai";
import { z } from "zod";
 
type ToolHandler<T> = (args: T) => Promise<Record<string, unknown>>;
 
interface Tool<T = unknown> {
  name: string;
  schema: z.ZodType<T>;
  handler: ToolHandler<T>;
  timeoutMs: number;
  costLimit: number; // expected cost per call (USD cents)
}
 
export class ToolRouter {
  private tools = new Map<string, Tool<any>>();
  private totalCost = 0;
  private readonly sessionCostCap = 500; // per-session cap (cents)
 
  register<T>(tool: Tool<T>): void {
    this.tools.set(tool.name, tool);
  }
 
  async dispatch(
    session: LiveSession,
    toolCalls: Array<{ id: string; name: string; args: unknown }>
  ): Promise<void> {
    const responses = await Promise.all(
      toolCalls.map((c) => this.runOne(c))
    );
    await session.sendToolResponse({ functionResponses: responses });
  }
 
  private async runOne(call: {
    id: string;
    name: string;
    args: unknown;
  }): Promise<{ id: string; name: string; response: Record<string, unknown> }> {
    const tool = this.tools.get(call.name);
    if (\!tool) {
      return {
        id: call.id,
        name: call.name,
        response: { error: `unknown tool: ${call.name}` },
      };
    }
 
    if (this.totalCost + tool.costLimit > this.sessionCostCap) {
      return {
        id: call.id,
        name: call.name,
        response: { error: "session_cost_cap_exceeded" },
      };
    }
 
    const parsed = tool.schema.safeParse(call.args);
    if (\!parsed.success) {
      return {
        id: call.id,
        name: call.name,
        response: { error: "invalid_args", detail: parsed.error.flatten() },
      };
    }
 
    try {
      const result = await withTimeout(
        tool.handler(parsed.data),
        tool.timeoutMs
      );
      this.totalCost += tool.costLimit;
      return { id: call.id, name: call.name, response: result };
    } catch (err) {
      return {
        id: call.id,
        name: call.name,
        response: {
          error: "tool_failed",
          message: err instanceof Error ? err.message : "unknown",
        },
      };
    }
  }
}
 
function withTimeout<T>(p: Promise<T>, ms: number): Promise<T> {
  return new Promise((resolve, reject) => {
    const t = setTimeout(() => reject(new Error("timeout")), ms);
    p.then((v) => {
      clearTimeout(t);
      resolve(v);
    }).catch((e) => {
      clearTimeout(t);
      reject(e);
    });
  });
}

Three intentional design choices here:

Zod validation: Live API occasionally returns stringified JSON in args. Using z.preprocess to accept string inputs eliminates the "flaky in prod" class of bugs almost entirely.

Per-session cost cap: Live API function calls are not free — many of them trigger paid external APIs. Without a cap, an adversarial user can drain thousands of yen in one session.

Parallel dispatch: Live API can emit multiple tool calls in a single message. Serial execution spikes latency; always run them concurrently.

Common Pitfalls

Here are five traps I and my clients have actually hit — none of which are explicit in the docs.

1. Sample rate mismatch makes audio sound sped up

Live API defaults to 16kHz input and 24kHz output. Browser getUserMedia often returns 44.1kHz or 48kHz. Sending the raw ArrayBuffer through makes the model hear "chipmunk speech" and transcription degrades.

Fix: Create your Edge-layer AudioContext with { sampleRate: 16000 } and resample through OfflineAudioContext before sending. If the browser refuses 16kHz, apply an explicit 3x downsampling chain with a BiquadFilter.

2. responseModalities: [AUDIO, TEXT] slows everything down

Requesting both modalities makes the model wait for text completion before emitting audio. First-audio latency balloons 1.5-2x.

Fix: Request audio only and enable outputAudioTranscription: { } if you need captions. Text then streams in parallel with audio.

3. Swapping systemInstruction by re-establishing sessions doubles your cost

systemInstruction is set once at setup and persists for the session's lifetime. Tearing down the session to change roles means paying the setup cost again every time.

Fix: Inject extra context via clientContent mid-session, or handle "expert routing" via function calling so you keep a single Live session alive.

4. Streaming screen frames at 30fps blows up your bill

realtimeInput video frames are billed as you send them. Full-screen 30fps easily hits hundreds of yen per minute.

Fix: Send only on change detection (pixel diff between frames) and cap at 0.5-2fps. On-demand high-frequency screen capture triggered by the user saying "look here" is the realistic production pattern.

5. Stale audio from the previous session plays after reconnecting

Closing the Live session does not automatically clear browser-side AudioBufferSourceNode instances queued for later playback. The tail of the last session plays into the first chunk of the new one.

Fix: Call audioQueue.interrupt() from stop() and explicitly close() and recreate the AudioContext. That residual noise disappears completely.

Running in Production with Antigravity Multi-Agents

Here is how these pieces compose inside Antigravity. I keep three agents resident in the Manager Surface:

Live Orchestrator agent: Owns the Live API session, mediates audio I/O, and dispatches tool calls. I put the state machine spec and reconnection policy in .agent/live-orchestrator.md.

Business Tools agent: Booking, search, RAG, and other internal APIs. Consumes typed function calls from the Orchestrator. .agent/business-tools.md documents each API's timeout and fallback.

Guardian agent: Monitors session duration, cost, and error rate. Sends terminate to the Orchestrator when thresholds are crossed. Runs as an Antigravity Background Agent with dashboards in the Manager Surface.

The reason I use three agents is simple: don't mix Live API's "long-lived, realtime, uncertain" traits with business logic's "short-lived, request/response, deterministic" traits. Mixing them warps the design toward whichever side has the louder failure mode. Antigravity's parallel agents feature is a very natural fit for this kind of separation.

Metrics That Actually Matter in Operations

These are the KPIs I put on the dashboard in production. Grafana + Prometheus or Cloudflare Workers Analytics Engine both work. If you want to tie agent-level traces together, pair this with Building an AI Observability Pipeline on Antigravity with OpenTelemetry for broader coverage.

• Session duration (P50 / P95): P95 climbing past 10 minutes means users are getting dragged into aimless small talk. Tighten Guardian's forced-termination policy.
• Time-to-first-audio: Above 1s is a degradation signal. Re-evaluate region and model tier (flash-live vs pro-live).
• Reconnection rate: More than 5% of sessions in an hour means WebSocket quality on the client is suspect.
• Tool call success rate: Under 95% typically means your function schema is ambiguous and Gemini is mis-invoking tools. Tighten Zod schemas and descriptions.
• Cost per session-minute: 1.5x over baseline usually means non-audio modalities (video, text) are leaking into the request budget.

Route all of these through Guardian and wire automatic auto-downgrade policies when thresholds are breached. Concrete actions: switch from pro-live to flash-live, cut video frame frequency, shorten max session duration.

Choosing Between flash-live and pro-live

Once Guardian is running and your baseline metrics stabilize, the highest-leverage optimization is picking the right Live model tier. I used to reflexively start on pro-live, but three recent projects changed my defaults.

When flash-live wins: Transactional support flows, structured data gathering, and anything where the model's role is to collect facts and dispatch tool calls. Time-to-first-audio is noticeably faster (200-400ms lower in my measurements), and function-calling fidelity has closed most of the gap against pro-live since the March 2026 update. For a booking or checkout agent, flash-live is now my default.

When pro-live wins: Open-ended reasoning, multi-turn troubleshooting where the model must remember earlier context well, and multilingual flows that switch languages mid-session. pro-live also handles screen-share reasoning noticeably better — if your agent needs to explain what is on screen, not just react to it, the extra cost is worth it.

The hybrid pattern: Start a session on flash-live and programmatically tear down + re-establish on pro-live only when the Orchestrator detects an escalation condition (user explicitly asks for escalation, frustration signals in transcript, tool failures exceeding a threshold). The cost profile becomes bimodal rather than uniformly expensive, and real users rarely notice the 1-2 second handoff gap because it coincides with human escalation moments.

One caveat: do not attempt dynamic tier switching mid-session via a single continuous connection. The API does not support live model hot-swapping, and trying to fake it by re-setting setup produces subtle state drift. Tear down and reconnect cleanly.

Real Scenario: Customer Support with Screen Share and RAG

One live production deployment I helped ship: an e-commerce support desk where users can ask voice questions while their "Order History" page is on screen.

• The user taps "Start voice support" in the help page. WebRTC connects to the Edge layer, and the Order History DOM is captured at 1fps and streamed.
• Live Orchestrator reads the screen context, queries internal APIs (orders DB, shipping API) via Business Tools.
• RAG uses prior FAQs pre-embedded in Vertex AI Vector Search, exposed as a search_knowledge function call.
• Guardian forces termination at 10 minutes and emails a summary to both the customer and the CS rep.

Outcome: inquiries that used to take 8 minutes in chat now resolve in an average of 2.5 minutes. In the first month, however, Live API costs ran close to 2x the estimate. After applying the "0.5fps screen share," "audio-only modality," and "session cap" fixes from above, costs landed in the expected range by month two.

Testing Strategy: Simulating Live Sessions in CI

A common question I get is how to write tests for Live API agents. You cannot dial real sessions from CI without quickly burning through your Gemini quota. The approach that has worked for me in three projects is a three-layer test pyramid, and I have come to rely on it for any realtime agent work.

Layer 1 — Unit tests with mock LiveSession: The LiveVoiceSession, AudioQueue, and ToolRouter classes are designed so their dependencies are injectable. I mock the @google/genai live.connect to return a fake LiveSession that I drive by hand. This layer catches state machine bugs (the onclose/onerror race I mentioned earlier) and regression-tests the retry logic. These tests run in under a second and are the primary gate on every PR.

Layer 2 — Replay tests against captured sessions: For each supported flow (booking, FAQ, escalation), I capture one real Live API session into a JSON transcript: the sequence of serverContent, toolCall, and audio chunks. The replay harness feeds those back into my Orchestrator with timing preserved. This layer catches schema drift in @google/genai and regressions in function argument handling. I re-record transcripts monthly to stay current with Gemini's behavior.

Layer 3 — Smoke tests against real Live API in staging: Once per deploy, a scripted agent runs a fixed two-minute dialog against the live service in a staging project. The only assertions are on tool call success rate, TTFT, and audio chunk integrity. Failures page the on-call. Because this is the only place real quota is spent, I scope it ruthlessly — two minutes, one model, one region.

This pyramid lets you ship Live API changes confidently. The replay layer is the real workhorse — it caught three production-bound regressions in the last quarter alone.

Security and Privacy Considerations You Cannot Skip

Voice agents amplify every privacy concern you had with chatbots. These are the ones that matter in production, roughly ordered by how often they bite:

PII redaction before logging: The audio transcripts you produce for debugging will contain names, phone numbers, and order numbers. Pipe them through a redaction pass (Cloud DLP works, or a local regex-based pass) before they land in logs or Sentry. Stream-based redaction on the transcript channel is what I use — it runs in the Orchestrator and touches all traffic.

User consent and recording disclosure: In many jurisdictions, you are required to disclose that the session is AI-mediated and, in some, explicitly that audio is being transmitted to a third party. Design the opening system turn to include the disclosure ("This conversation is handled by an AI assistant; audio is processed by Google Gemini...") and log the acknowledgment.

Function calling allowlist: Never let the Live session call arbitrary internal APIs. The ToolRouter above enforces a name-based allowlist, and each tool's handler should itself validate the caller's authority again — defense in depth. I have seen a prompt-injected agent attempt to invoke a refund_order tool the user had no right to call; the defense-in-depth check stopped it.

Rate limits per user, not just per session: An attacker who finds a way to spin up new sessions will happily do so. Track spend and duration at the user level and enforce caps there, in addition to the per-session cap in ToolRouter.

Model routing for regulated data: For healthcare or financial data, prefer a regional model endpoint where available, and document in your architecture which data classes are allowed to leave the region.

Deployment Checklist

Before pushing Live API to real users, I run through this checklist. It has caught embarrassing issues before every launch I have supported:

• [ ] Guardian agent is the first thing in the deploy, with alerts wired to PagerDuty or equivalent.
• [ ] Session duration cap is set to a value matching your support SLAs (I usually start at 10 minutes).
• [ ] Per-session costLimit across all tools sums to an acceptable worst case.
• [ ] Reconnection logic is tested by manually killing the WebSocket in staging.
• [ ] Audio sample rate is explicitly 16kHz on input, verified by capturing a staging session.
• [ ] responseModalities is audio-only; captions come from outputAudioTranscription.
• [ ] Screen-share fps is capped at 2fps by default and only spikes on explicit user intent.
• [ ] AudioContext is close()d and recreated on session stop; verified by Chrome's about:tracing.
• [ ] PII redaction is enabled on all transcript logging paths.
• [ ] Disclosure line is present in the opening system turn.
• [ ] Tool name allowlist matches the agents you actually defined (run diff against your .agent/ files).
• [ ] Model downgrade path (pro-live → flash-live) is wired and tested.
• [ ] Dashboard shows P95 session duration, TTFT, reconnect rate, tool success rate, and cost per minute.
• [ ] CI replay tests pass against a freshly recorded transcript.

The items are boring on purpose. Live API incidents in production are almost always about boring oversights — a forgotten cap, a missed allowlist entry, an untested reconnection path. Boring checklists beat heroic debugging.

When to Not Use Live API

I spend most of this article evangelizing Live API, so let me balance that. There are three scenarios where I actively recommend against it:

The interaction does not need sub-second latency: If your agent can reasonably reply in 2-3 seconds, classic generateContent with streaming text plus a separate TTS pass (ElevenLabs, Google Cloud TTS) is simpler, cheaper, and easier to test. Live API pays its premium in interactivity, not in throughput.

You need strong deterministic control over model output: Live API's barge-in and streaming behavior means you give up some of the fine-grained control you have with batch APIs. If your use case is "the model must emit exactly this structured response," a text-based function-calling flow is more appropriate.

Your sessions are mostly idle: Live API bills throughout the session, not just during exchanges. If a "session" in your product is a user leaving a browser tab open for hours with occasional questions, that's not a Live API session — that's a classic chat, and you should open a new request per turn.

Matching the tool to the job keeps costs predictable. I have seen teams default to Live API because it's the shiniest API available, then spend their runway on unused compute time. Pick it when the realtime dimension actually matters to the user.

Closing: The First Move You Should Make Today

When teams ask me where to start with Live API in production, my answer is always the same: build the Guardian agent first — the cost and time monitor. Building features before guardrails is how you ship expensive incidents. Monitoring first, Live session implementation second: that's the order that holds up under real traffic.

If you have 30 minutes today, spin up .agent/guardian.md in Antigravity and enumerate the metrics you want to capture at session start (start time, model, expected cost cap). Then approach the Live API SDK. It looks like a detour, but it's actually the shortest path from zero to production.

For the practical details, use the three code examples and five pitfalls in this article as your checklist. Gemini Live API opens up real creative possibilities, and it will also happily burn a hole in your budget if you treat it like a classic REST API. Spending that extra hour on architecture is what separates a demo from a service.