When Parallel Agents Ran the Same Task Twice and Quietly Doubled the Bill — Field Notes on Measuring and Stopping Duplicates

The bill for our parallel agents came in about 1.9x higher than expected — because multiple workers were running the same task twice. These are field notes on measuring the duplication, stopping it with idempotency keys, and attributing cost per task.

multi-agent⁵⁰ idempotency¹¹ cost management⁵ production⁷¹ Antigravity³⁷⁶

✦ Premium Article

I opened the monthly bill and stopped scrolling. Volume was roughly flat, but the number was about 1.9x what I expected. Not a single alert had fired. Every request in the logs had completed successfully.

It took half a day to find. Of the three workers I was running in parallel, two were picking up the same task independently, each deciding it was theirs, and each running it to completion. The outputs matched, so everything looked correct. Underneath, generation charges were quietly stacking up twice.

These are field notes on catching that silent double-execution through measurement, stopping it with idempotency keys, and attributing cost per task. The setup assumes you delegate a batch of tasks across multiple agents — the way Antigravity's Managed Agents work. I built it as a layer that doesn't depend on a specific SDK, so it ports to the Gemini API or your own workers.

Why Duplication Happens Without Raising an Error

What makes double-execution nasty is that it throws no exception. Both workers succeed, both return correct results. The orchestrator simply discards or overwrites the second result, and the fact that the work ran twice leaves no trace.

The root cause is that an LLM won't explicitly decline "this isn't mine." Write the responsibility boundary loosely and several workers will each claim the same request as their own.

Symptom	What you see	What you don't
Double-run	Correct output, no error	The same task billed twice
Drop	Counts don't add up	Every worker judged it "out of scope" and left it
Loop	Higher latency	Workers bouncing the boundary back and forth

The first thing that helped was writing each worker's scope down to what it must NOT do, not just what it does.

# ❌ Ambiguous boundary (breeds duplicates and drops)
You are a data-processing agent. Create reports as needed.
 
# ✅ Name the out-of-scope work and hand it back
You are a worker dedicated to "JSON -> schema conversion".
In scope: convert the received JSON to the given schema and return it.
Out of scope: saving, sending, report generation, re-delegating to other workers.
If out-of-scope input arrives, return only {"status":"out_of_scope"} with no body
and hand it back to the orchestrator. Do not cover for it yourself.

But clarifying the prompt only makes duplication less likely. When a source of variability runs in parallel, you need a structural mechanism to actually stop it.

Making Duplication Visible With an Execution Ledger

Before stopping it, count it. If you don't record which task ran how many times, you can't judge whether a fix worked either.

The crux is how you define task identity. I built a fingerprint from the semantic body of the input — the normalized arguments. The trick is to strip out values that change on every run, like timestamps and execution IDs, before normalizing.

import hashlib
import json
import time
 
def task_fingerprint(task_type: str, payload: dict) -> str:
    """Key on semantic identity by dropping values that change per run."""
    volatile = {"request_id", "timestamp", "attempt", "trace_id"}
    stable = {k: v for k, v in payload.items() if k not in volatile}
    # Fix key order for a stable hash
    canonical = json.dumps(stable, sort_keys=True, ensure_ascii=False)
    digest = hashlib.sha256(f"{task_type}:{canonical}".encode()).hexdigest()
    return digest[:16]
 
 
class ExecutionLedger:
    """Records run counts per task. store is a thin KV wrapper (e.g. Redis)."""
 
    def __init__(self, store):
        self.store = store  # provides get/incr/expire
 
    def record(self, fp: str, worker_id: str) -> int:
        key = f"ledger:{fp}"
        count = self.store.incr(key)          # increment run count
        self.store.expire(key, 60 * 60 * 24)  # let it age out in 24h
        self.store.sadd(f"{key}:workers", worker_id)
        return count  # >= 2 means this task was run more than once

After wiring this into production for about a week, the duplication rate came into focus. Roughly 18% of all tasks ran two or more times, and nearly all of it concentrated in two specific task types. I had assumed it was an "every now and then" problem, so that 18% was a genuine surprise.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦An execution ledger that detects double-runs via task fingerprints

✦A SET NX claim-before-run idempotency pattern that stops double billing

✦Per-agent, per-task cost attribution that catches billing anomalies early

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Stopping the Double-Run: Claim-Before-Run With Idempotency Keys

The ledger is an after-the-fact record. To stop the double billing itself, you have to reserve the work before running it.

The pattern is simple. Before starting, a worker writes a reservation keyed on the fingerprint in a "claim it only if no one has" fashion. Only the worker that acquires the claim executes; the others wait for the existing result or step aside. The KV's atomic "write if absent" operation (SET NX) is the foundation.

import time
 
class IdempotencyGuard:
    def __init__(self, store, ttl_seconds: int = 900):
        self.store = store
        self.ttl = ttl_seconds
 
    def claim(self, fp: str, worker_id: str) -> bool:
        """True if we won the claim. False if someone is already running/ran it."""
        key = f"claim:{fp}"
        # NX: write only if absent (atomic). This is the heart of the race.
        acquired = self.store.set(key, worker_id, nx=True, ex=self.ttl)
        return bool(acquired)
 
    def save_result(self, fp: str, result: dict) -> None:
        self.store.set(f"result:{fp}", json.dumps(result), ex=self.ttl)
 
    def wait_result(self, fp: str, timeout: float = 30.0) -> dict | None:
        """The loser waits for the leading run's result."""
        deadline = time.time() + timeout
        while time.time() < deadline:
            raw = self.store.get(f"result:{fp}")
            if raw is not None:
                return json.loads(raw)
            time.sleep(0.5)
        return None  # on timeout, hand the decision back to the orchestrator
 
 
def run_task(worker_id, task_type, payload, guard, execute_fn):
    fp = task_fingerprint(task_type, payload)
    if guard.claim(fp, worker_id):
        result = execute_fn(payload)   # the actual LLM run happens only here
        guard.save_result(fp, result)
        return {"source": "executed", "result": result}
    # We lost the claim -> another worker is running the same task
    shared = guard.wait_result(fp)
    if shared is not None:
        return {"source": "reused", "result": shared}
    return {"source": "fallback", "result": execute_fn(payload)}

The important part is not silently letting a worker that lost the claim re-run execute_fn. My first version unconditionally added a "run it myself just in case" fallback, and duplication barely dropped as a result. Only once I made waiting for the leading result the primary path — and narrowed the fallback to genuine timeouts — did the duplication rate fall from about 18% to 0.4%.

TTL needs care too. Too short and the claim expires before the leading run finishes, putting you back into double-execution; too long and a failed task blocks its own retry for ages. I started at 3x the task's p95 runtime and tuned from there.

Attributing Cost to Tasks and Agents

Even after stopping duplicates, I keep recording cost per execution so I can see ahead of time where the next blowup will come from. Understanding the shape of cumulative billing lets you catch anomalies while they're still small.

The main reason multi-agent cost spikes is accumulating conversation history. As steps progress, each input carries the entire prior history.

def cumulative_input_tokens(steps: int, avg_per_step: int = 1000) -> int:
    """Because each step includes prior history, input grows as a series."""
    # 1000*(1+2+...+steps) = 1000 * steps*(steps+1)/2
    return avg_per_step * steps * (steps + 1) // 2
 
for s in (1, 5, 10, 20):
    print(f"{s:2d} steps: cumulative input {cumulative_input_tokens(s):>7,} tokens")
# 1 steps: cumulative input   1,000 tokens
# 5 steps: cumulative input  15,000 tokens
# 10 steps: cumulative input  55,000 tokens
# 20 steps: cumulative input 210,000 tokens

55,000 at 10 steps, 210,000 at 20. Double the steps and input roughly quadruples. Knowing this shape makes it obvious why a step-count cap and folding history into interim summaries actually help.

Record each run's cost alongside the fingerprint, worker, and task type, so you can later aggregate which task type is driving the bill.

def log_cost(store, fp, worker_id, task_type, usage: dict):
    record = {
        "fp": fp,
        "worker": worker_id,
        "task_type": task_type,
        "input_tokens": usage["input_tokens"],
        "output_tokens": usage["output_tokens"],
        "ts": time.time(),
    }
    store.rpush(f"cost:{task_type}", json.dumps(record))
 
def task_type_summary(store, task_type) -> dict:
    rows = [json.loads(r) for r in store.lrange(f"cost:{task_type}", 0, -1)]
    runs = len(rows)
    in_tok = sum(r["input_tokens"] for r in rows)
    out_tok = sum(r["output_tokens"] for r in rows)
    return {
        "task_type": task_type,
        "runs": runs,
        "avg_input": in_tok // runs if runs else 0,
        "total_tokens": in_tok + out_tok,
    }

If you bill usage through Stripe Meter Events, this per-execution record doubles as the reconciliation source against your billing meter. With this attribution, you can tell at a glance whether a billing anomaly is "more runs" or "bigger per run." Our 1.9x was the former — a volume (duplication) problem. Had I not been recording cost per execution, I'd have suspected a price change or history bloat and dug for days in the wrong place.

Tightening Gradually: Thresholds and Alerts

A one-time fix won't hold. Prompt changes and concurrency tweaks bring duplicates back. Keep the ledger permanent and stay able to notice when a threshold is crossed.

Metric	Normal range	Alert condition
Duplication rate (share with record >= 2)	under 1%	over 3% for 15 min
claim-fail -> fallback ratio	under 1%	over 5%
Avg input tokens per task type	baseline +/-20%	over +50%

Making everything strict at once buries you in false alarms. I watched only the duplication rate first, then added the rest once things settled. Monitoring sticks better when you start from the one signal that matters, not all of them at once.

Wrap-Up: Count First, Then Stop

Silent double-execution hides behind correct output and inflates only the bill. Because no error fires, measurement is the only handle you get.

As a next step, if you're an indie developer running parallel agents today, drop in just the fingerprint-based execution ledger and measure your own duplication rate. Once you have the number, you can decide whether it's worth stopping with an idempotency key or still within tolerance. I brushed it off as "every now and then" until I measured it. Only after counting did it become a problem worth stopping.

I hope these notes help in your own production work. Thank you for reading.

Thank You for Reading

Antigravity Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.