One night, a code-review agent I'd been running for six hours inside Antigravity started reading the same files three times in a row. Its prompt had accumulated nearly fifty useAuth.ts examined tool-call entries, and the agent itself was getting lost trying to read its own memory back.
This isn't unusual once an AgentKit 2.0 agent runs long enough. The history the LLM sees grows with every turn, and past a certain threshold the model is technically "looking at" old turns but no longer behaving as if it remembers them. This article shares the context compression sub-agent design I actually run in production for my own indie projects — the schema, the prompt, the integration hooks, and the evaluation harness.
Why "just truncate the history" breaks in production
The first instinct is always the same: when you're about to overflow the token budget, slice off the oldest N messages. I tried that. It failed in three predictable ways, every time.
First, the user's original instructions evaporate. Decisions like "write the new tests with React Testing Library" set in turn 1 are gone by hour three, and the agent quietly defaults back to vanilla Jest.
Second, tool-call/tool-result pairs get split. If a tool_use block survives but its matching tool_result is on the chopping block, AgentKit 2.0 returns an API error mid-run. Naive slicing doesn't preserve a syntactically valid history.
Third, the agent forgets what it has already tried, so it repeats the same failed approaches. Users experience this as "this agent suddenly got worse," which is the kind of regression that's painful to debug after the fact.
I hit all three of these in my own systems. That experience is exactly why I moved away from truncation and towards compression — and why I think it's worth the implementation effort.
It also helps to understand that this isn't a quirk of one particular model. The same failure modes show up regardless of whether you're driving the agent with Gemini 2.5 Pro, Claude, GPT-4o, or a self-hosted Gemma 4 instance. The mechanics are universal: long-context models attend less precisely to information buried under thousands of intervening tokens, so older instructions effectively fade. Compression isn't a workaround for one vendor's limitations — it's an architectural pattern any production agent eventually needs.
A second realization that took me too long: compression is not the same as memory. Memory systems retrieve relevant historical context on demand. Compression rewrites the live history in place. Both are valuable, and they layer well. But if you confuse them — for example, by relying on RAG-style retrieval to "fix" an over-long conversation — you'll find that the agent still has to chew through all the recent turns every step, and your token cost barely budges. Compression is the only mechanism that actually shortens what the model reads on every turn.
The architecture at a glance
The core idea is to introduce a separate sub-agent whose only job is to read the running agent's history and return a structured summary. AgentKit 2.0's sub-agent primitives let you register a small worker with its own system prompt and model.
The runtime loop looks like this. Every time the main agent steps, you compute the current message array's token count. If that count crosses a threshold (I use 60% of the model's max), the compression sub-agent is invoked. It reads the older messages, returns a structured summary, and the main agent's history is rewritten — the recent few turns stay, everything older is replaced by a single summary block.
A non-obvious but critical choice: use a smaller model for the compression sub-agent than the main one. In my own setup, the main agent runs on Gemini 2.5 Pro and the compressor runs on Gemma 4 Flash. If you use the bigger model for compression, the cost optimization story collapses immediately.
Step 1: Designing the compression trigger
There are three reasonable trigger styles: token-count threshold, turn-count threshold, and event-driven.
Choosing the right trigger style isn't purely an engineering decision — it tells you something about what your agent actually does. An agent that handles bursty workloads (research, large-codebase analysis) benefits from event-driven triggers because token consumption is non-uniform. An agent that handles steady incremental work (writing tests one by one, drafting documentation) is happier with simple turn-count triggers because the cost of measurement adds up. Most production agents are a mix of both, which is why I run a hybrid.
The token-count approach is the most direct — count the array's total tokens with tiktoken and fire when you near the ceiling. It's accurate but expensive to compute, so I run it every N turns rather than every turn.
The turn-count approach is coarser. "Compress every 30 turns no matter what." Easy to implement, but unreliable when token usage per turn varies wildly.
Event-driven means firing immediately after specific tool calls that you know return huge payloads (think search_codebase or read_many_files). When you understand your agent's bursty consumption pattern, this is the most efficient strategy.
In production I run a hybrid of token-threshold and event-driven. Here's the actual function:
// src/agent/compression-trigger.ts
import { encodingForModel } from "js-tiktoken";
const enc = encodingForModel("gpt-4o"); // close enough for estimation
export interface CompressionDecision {
shouldCompress: boolean;
reason: "token-threshold" | "tool-event" | "turn-limit" | "skip";
estimatedTokens: number;
}
const TOKEN_THRESHOLD_RATIO = 0.6;
const HEAVY_TOOLS = new Set(["search_codebase", "read_many_files"]);
export function evaluateCompression(
messages: Array<{ role: string; content: string; toolName?: string }>,
modelMaxTokens: number,
turnsSinceLastCompression: number
): CompressionDecision {
// 1. Token-based check
const totalTokens = messages.reduce(
(sum, m) => sum + enc.encode(m.content).length,
0
);
if (totalTokens > modelMaxTokens * TOKEN_THRESHOLD_RATIO) {
return { shouldCompress: true, reason: "token-threshold", estimatedTokens: totalTokens };
}
// 2. Heavy-tool event — fire even at 0.4x threshold
const last = messages[messages.length - 1];
if (last?.toolName && HEAVY_TOOLS.has(last.toolName)) {
if (totalTokens > modelMaxTokens * 0.4) {
return { shouldCompress: true, reason: "tool-event", estimatedTokens: totalTokens };
}
}
// 3. Safety net — force compression after 30 turns
if (turnsSinceLastCompression >= 30) {
return { shouldCompress: true, reason: "turn-limit", estimatedTokens: totalTokens };
}
return { shouldCompress: false, reason: "skip", estimatedTokens: totalTokens };
}This is called from the main loop every turn. The reason field exists so you can audit, after the fact, why compression fired (or didn't). I cannot stress enough how much you'll need that log later — without it, you have no way to tell whether you're firing too often or not enough.
Step 2: Locking down "what must not be lost" with a schema
The single most dangerous mistake in compression is letting the sub-agent return free-form prose. The model gets to decide what's "important," and that decision isn't reproducible. The first time I shipped this without a schema, the user's original constraint "use TDD for all new tests" was nowhere in the resulting summary, and the agent happily abandoned TDD.
The fix is to give the sub-agent a structured schema with required fields and force it to emit JSON.
// src/agent/compression-schema.ts
export interface CompressedContext {
// Constraints from the user — never lose these
user_constraints: string[];
// Decisions made so far, with rationale
decisions: Array<{ decision: string; rationale: string }>;
// Approaches that failed, so the agent doesn't retry them
failed_attempts: Array<{ approach: string; reason_failed: string }>;
// Summary of tool calls already made, to prevent duplicate calls
tool_calls_summary: Array<{ tool: string; target: string; outcome: "success" | "error" | "partial" }>;
// Free-text summary of the most recent 5 turns, for narrative continuity
recent_narrative: string;
// Where the task is right now
task_state: { completed: string[]; remaining: string[]; blockers: string[] };
}This schema is an explicit list of "things we cannot afford to lose during compression." More fields means better fidelity but higher inference cost. I've run with these six fields for six months in production; trimming any of them has reliably caused regressions.
You hand this to AgentKit 2.0's structured-output mechanism as a JSON Schema, which forces the sub-agent's output into this exact shape.
Each field in this schema earns its keep through a specific failure mode I've observed.
user_constraints exists because, as I mentioned, my first compression deployment lost a TDD constraint. Treating constraints as their own first-class field — separated from "decisions" or general narrative — makes them recoverable through schema validation. If the array is empty when you know constraints exist, you have a bug to investigate.
decisions and failed_attempts are deliberately separated. Without that split, the sub-agent will conflate them and you'll see an agent that "remembers" trying something but doesn't remember whether it worked. A failed approach written in the same field as a successful decision becomes indistinguishable from a working solution after compression.
tool_calls_summary exists because the most common cost waste in long-running agents is calling the same tool with the same arguments multiple times. The compressor explicitly summarizes this, and you can build a preflight check that warns the agent: "you've already searched the codebase for useAuth three times today."
recent_narrative and the preserved last 5 turns serve as a smooth handoff. If you only injected the structured object, the model would lose the flow of the conversation. The narrative is the glue that keeps the next turn from feeling like a context switch.
task_state is the one field where I've experimented the most. Some teams I've worked with prefer a free-form "current goal" string instead. After comparing both for a quarter, I came back to the structured completed/remaining/blockers form. It dramatically reduces the agent's tendency to re-do work that's already done — which by my measurement was the single biggest cost leak in long-running agents.
Step 3: Implementing the compression sub-agent
Here's the heart of the system. The sub-agent has one job: read the main agent's history and return that schema, filled in.
// src/agent/compression-agent.ts
import type { Message, AgentKit } from "@google/agentkit";
import type { CompressedContext } from "./compression-schema";
const COMPRESSION_SYSTEM_PROMPT = `
You are a specialized sub-agent that produces structured summaries of
agent execution history. Your input is a sequence of messages and
tool calls from the main agent. Your output is a CompressedContext
JSON object.
Follow these principles:
1. Preserve user constraints (language, libraries, naming rules) verbatim
2. Separate "decisions" from "failed attempts"
3. Summarize tool calls in three parts: what / where / outcome
4. Keep the most recent 5 turns as a readable narrative
5. Never invent — if it's not in the history, do not output it
`;
export async function compressContext(
agentkit: AgentKit,
messages: Message[]
): Promise<CompressedContext> {
// Reserve the most recent 5 turns; do not compress them
const compressTarget = messages.slice(0, -5);
const recent = messages.slice(-5);
if (compressTarget.length === 0) {
throw new Error("compressContext: nothing to compress");
}
const result = await agentkit.subAgents.run({
name: "context-compressor",
model: "gemma-4-flash",
systemPrompt: COMPRESSION_SYSTEM_PROMPT,
messages: [
{
role: "user",
content: `Summarize the following message sequence into the CompressedContext schema:\n\n${JSON.stringify(compressTarget)}`,
},
],
responseSchema: COMPRESSED_CONTEXT_JSON_SCHEMA,
temperature: 0.0, // compression must be deterministic
maxOutputTokens: 2048,
});
let parsed: CompressedContext;
try {
parsed = JSON.parse(result.text);
} catch (e) {
throw new Error(`Failed to parse compression output: ${result.text.slice(0, 200)}`);
}
// Minimum sanity check
if (!Array.isArray(parsed.user_constraints) || !parsed.task_state) {
throw new Error("Compression output is missing required fields");
}
// Append the preserved recent 5 turns to the narrative
const recentText = recent.map((m) => `[${m.role}] ${m.content.slice(0, 200)}`).join("\n");
parsed.recent_narrative = `${parsed.recent_narrative}\n\n[Last 5 turns preserved verbatim]\n${recentText}`;
return parsed;
}temperature: 0.0 is intentional. Compression has to be reproducible — if every run produces a different summary, you introduce drift into the agent's history, and your A/B harness can no longer measure anything meaningfully.
A truncated example of what the sub-agent returns:
{
"user_constraints": [
"TypeScript only",
"Use Vitest for tests",
"All auth logic lives under src/auth/"
],
"decisions": [
{
"decision": "Store the JWT refresh token in an httpOnly cookie",
"rationale": "Avoid XSS exposure of localStorage"
}
],
"failed_attempts": [
{
"approach": "useAuth hook placed directly in App.tsx",
"reason_failed": "Context Provider missing — runtime error"
}
],
"task_state": {
"completed": ["login form", "JWT issuance API"],
"remaining": ["refresh token rotation", "logout flow"],
"blockers": []
}
}When you read a few of these in production logs, two things become visible. First, the schema gives you a diff-able artifact — you can compare two consecutive compressions and immediately see what was added or lost between them. That's enormously valuable when you're debugging a regression. Free-form summaries don't diff well; structured ones do.
Second, the schema makes it easy to spot when the sub-agent is hallucinating. If the user_constraints array contains "Use Tailwind CSS" but the user never said any such thing in the original conversation, you have a clear signal that the compressor is fabricating content. With free-form summaries, you'd never catch that — it would just look like a slightly off paraphrase. The structured form turns subtle hallucinations into hard-to-miss bugs.
Step 4: Wiring it into the main agent
A returned summary on its own changes nothing. You actually need to rewrite the main agent's history. AgentKit 2.0 exposes an onBeforeStep hook that lets you do exactly that.
// src/agent/main-agent.ts
import { evaluateCompression } from "./compression-trigger";
import { compressContext } from "./compression-agent";
const mainAgent = agentkit.agents.create({
name: "main",
model: "gemini-2.5-pro",
systemPrompt: MAIN_PROMPT,
hooks: {
onBeforeStep: async (ctx) => {
const decision = evaluateCompression(
ctx.messages,
128_000, // gemini-2.5-pro context window
ctx.metadata.turnsSinceLastCompression ?? 0
);
ctx.logger.info({ decision }, "compression check");
if (!decision.shouldCompress) {
ctx.metadata.turnsSinceLastCompression =
(ctx.metadata.turnsSinceLastCompression ?? 0) + 1;
return;
}
const compressed = await compressContext(ctx.agentkit, ctx.messages);
const recentMessages = ctx.messages.slice(-5);
ctx.replaceMessages([
{
role: "system",
content: `[CONTEXT_SUMMARY]\n${JSON.stringify(compressed, null, 2)}`,
},
...recentMessages,
]);
ctx.metadata.turnsSinceLastCompression = 0;
ctx.metrics.recordCompression({
reason: decision.reason,
beforeTokens: decision.estimatedTokens,
});
},
},
});The trick is injecting the summary as a system message at the start of the rewritten history. The main agent reads it on the next step as "what I remember about this task."
There's a trap waiting here — tool-call integrity. If the slice that becomes recentMessages contains a tool_use block but not its matching tool_result, AgentKit will refuse to call the model. Before slicing, walk the array and extend the cut point so it never lands inside an unfinished tool pair.
There's another integration concern that bites teams later: where compression sits relative to your observability stack. If you're sending traces to a tool like Langfuse, Helicone, or your own OpenTelemetry pipeline, you want each compression event to appear as a discrete span. The metrics are different from regular agent steps — duration, input tokens, output tokens, the trigger reason — and conflating them with main-agent steps will skew every dashboard you build later. I always tag compression events with span.kind = "compression" so they can be filtered out (or in) at the dashboard level.
Another subtle point: the order in which you write the new history matters. AgentKit 2.0 caches prefix tokens for routes that support prompt caching. If you keep the system message in position 0 and only mutate later messages, you preserve the cache. If you reorder the system message or mix new content into position 0, you invalidate the cache and lose the cost savings of prompt caching on the very turn where compression runs. I always put the compressed summary as a new system message immediately after the original one, never replacing it.
Step 5: An evaluation harness for compression quality
Once compression is in place, instrument it. The three metrics I track are:
Task completion rate. Same starting prompt, with vs. without compression — what fraction of tasks reach success? If this drops, compression is hurting you and should be turned off.
Contradiction count. How often does the agent contradict a decision it already made? A compressed history that loses constraints will produce contradictions. Anything above 0.5 contradictions per task means the schema needs revision.
Cost reduction. How many fewer cumulative input tokens does the compressed run consume? In my deployments this lands between 40–55%. If it's below 30%, your threshold is too high and compression isn't running often enough.
A skeleton of the harness:
// src/agent/eval/compression-ab-test.ts
interface EvalResult {
taskId: string;
completed: boolean;
contradictions: number;
totalInputTokens: number;
totalOutputTokens: number;
durationMs: number;
}
export async function runABTest(
tasks: Array<{ id: string; prompt: string }>,
options: { withCompression: boolean }
): Promise<EvalResult[]> {
const results: EvalResult[] = [];
for (const task of tasks) {
const agent = createMainAgent({ enableCompression: options.withCompression });
const start = Date.now();
try {
const out = await agent.run({ prompt: task.prompt, maxSteps: 100 });
results.push({
taskId: task.id,
completed: out.status === "success",
contradictions: countContradictions(out.history),
totalInputTokens: out.usage.input,
totalOutputTokens: out.usage.output,
durationMs: Date.now() - start,
});
} catch (e) {
results.push({
taskId: task.id,
completed: false,
contradictions: 0,
totalInputTokens: 0,
totalOutputTokens: 0,
durationMs: Date.now() - start,
});
}
}
return results;
}
// Contradiction detection — judge LLM or rules; out of scope here
function countContradictions(history: Message[]): number {
// e.g. "use TypeScript" was the constraint — did we emit any .js?
return 0;
}Run this against a 50-task seed dataset and review the deltas weekly. That's the minimum bar I'd recommend before trusting compression in production.
A note on dataset construction: do not use a hand-curated dataset of "ideal" tasks for this evaluation. Use the messy tasks from your real production logs. The reason is that compression's failure modes show up specifically in long, branching, mistake-laden conversations. A clean dataset will tell you compression is harmless. Real production traffic will tell you whether compression actually preserves the things that matter.
I keep a rolling 50-task evaluation set sampled from the prior month's production runs, with personally identifying information stripped and stored in a private repo. Every release that touches the compression code reruns the full set and emits a delta report — completion rate change, average tokens per task change, contradiction rate change. If any of those numbers move in the wrong direction by more than 5%, the release is held until I understand why.
How compression interacts with prompt caching
If you're already using prompt caching with Gemini, Claude, or your own model gateway, the relationship between caching and compression deserves attention. Naive compression can entirely defeat caching, which would erase the cost savings you'd otherwise get from each. Done right, the two amplify each other.
The mechanics are straightforward once you see them. Prompt caching works by detecting that a prefix of the message array is identical to a previously seen prefix and reusing the model's internal state from that earlier computation. Every time you mutate the prefix — for example by inserting a fresh compression summary at position 0 — the cache is invalidated.
The fix is to keep the original system prompt in position 0 unchanged, and place the compression summary as a new system message at position 1. The first system message stays cacheable across runs and across users. The second changes whenever compression fires. The model treats both as system messages, but only the first contributes to the cache key.
Even better, if you're running enough traffic to justify it, you can hash the compression summary itself and use it as a sub-cache key. Multiple users hitting your agent with similar tasks will produce similar compression outputs, and you can cache the model state after the second system message as well. In my own deployment this added another 12% on top of the prefix-cache savings.
There's a related gotcha worth flagging: do not put a timestamp inside the compression output. Whatever you do, don't include the current time, run ID, or any user-specific identifier inside the structured summary. These fields invalidate the secondary cache on every run. Compression metadata belongs in observability tags, not in the prompt itself.
When NOT to use compression at all
I want to be clear about cases where compression is the wrong answer.
If your agent is single-turn (it takes one prompt, does its work, returns a result), compression has nothing to compress. Skip it. The complexity isn't free.
If your agent is multi-turn but each turn is independent (a chat assistant where each message stands alone), compression often hurts because cross-turn references are rare. The summary's overhead exceeds its benefit.
If your latency budget is tight (sub-200ms p99), even a 500ms compression event is unacceptable. You'd be better off using a model with a larger context window and accepting the cost increase as the price of meeting latency.
If you're operating at a scale where every millisecond of model time costs noticeable money but you don't have a real evaluation harness, compression is risky. Without measurement, you can't tell whether your changes are net positive — and "felt right" is a poor reason to ship infrastructure that rewrites the agent's own memory.
For any of these cases, it's better to invest in input-side optimization (smaller, better-targeted prompts) or scope-side optimization (designing tasks that finish in fewer turns) before reaching for compression.
Pitfalls I actually walked into
A few things I had to learn the hard way before this design was stable.
These are listed in roughly the order I encountered them. None of them are exotic — they all come from forgetting a small detail under deadline pressure. Reviewing this list before you ship will save you a few hours of confused debugging.
Leaving temperature at default. Most SDKs default to 0.7–1.0. With non-zero temperature, your compression output drifts run-to-run, and your A/B harness measures noise, not signal. Hard-pin to 0.0.
Slicing through a tool-call pair. messages.slice(-5) will happily cut a tool_use away from its tool_result. Always extend the cut point until the boundary lands outside any unresolved tool pair.
Injecting the summary as user role. This is tempting because it puts the summary "from outside the model's perspective," but the agent then thinks the user said it, and starts behaving as if the user agreed to every decision in the summary. Use system role, or at minimum prefix it with [CONTEXT_SUMMARY] as I did above.
Compressing too early. The user's initial instructions are higher quality than any summary. My threshold has a floor of 15 turns — below that, no compression runs no matter what. Premature compression has nearly all the cost and almost none of the benefit.
Forgetting to invalidate the trigger metadata after compression. I once shipped a bug where turnsSinceLastCompression was incremented but never reset, which meant compression eventually fired every single turn. Watch for this — the unit test is trivial but the symptom in production is "everything got mysteriously slower and more expensive."
Compressing in agents that produce streamed output to users. If your agent is mid-stream when compression decides to fire, you have a choice between awkwardly pausing the stream or queueing the compression for after the stream completes. I default to queueing. Pausing midstream is bad UX and almost never worth the marginal token savings.
Should you actually turn this on in production?
People hesitate on the rollout decision, and reasonably so — it's non-trivial machinery. My three-question checklist:
If your average task completes in under 30 turns, skip compression entirely. The 500ms–1.5s overhead per compression event isn't worth it. Long-running task types — code review, multi-step research, data analysis — routinely cross 50 turns and degrade visibly without compression.
If your A/B harness shows contradiction rates more than 1.2x the no-compression baseline, your schema or threshold needs tuning. Around 1.0x is the sign you're safe to enable by default.
Watch user-facing latency. Compression runs synchronously and will make some turns noticeably slower. Add a UI affordance ("Tidying up history…") so the slowdown reads as deliberate work rather than a stall.
A useful framing I've come back to: compression is best treated as a feature flag, not a permanent state. Roll it out behind a flag, run for two weeks with the A/B harness collecting data, and only flip it to default-on once the numbers are conclusive. If the metrics show ambiguity, leave it as opt-in for power users who run long tasks. This protects you from the worst case — silently degrading the experience for users whose tasks didn't need compression in the first place.
What to do today
The smallest concrete step you can take right now: dump your agent's full message array at turn 30 to JSON and run tiktoken over it. That single number tells you whether your agent actually needs compression yet.
If the number says yes, paste the evaluateCompression from this article into your codebase and tune only the threshold to your model's window. The schema and the evaluation harness can grow incrementally — don't try to ship all of it at once.
Compression is one of those engineering investments that feels disproportionate when you're building it and obvious in retrospect once it's running. The first time you watch a six-hour agent task finish without quality degradation, while showing 50% lower token usage on the dashboard, the implementation cost feels trivial. The hardest part isn't the code — it's deciding to do it before the third or fourth time you've debugged the same "agent forgot the constraint" bug. If this article spares you even one of those debugging sessions, it has paid for itself.
For adjacent reads, I've written about Antigravity context window management (input-side optimization) and Antigravity agent cost optimization (loop-side optimization). The compression pipeline in this article is the history-side optimization, and the three layered together compound nicely. If you want to zoom out further, the AI agent memory architecture guide covers how compression fits into a broader memory design.