Flow Control for Autonomous Agents: Backpressure and Queues That Keep Production Alive

The Moment Naive Parallelism Breaks Production

First, look at the "breaking code" I actually wrote. It just fans agent-generated tasks out at once with Promise.all.

// Before: shove every arriving job downstream at once
async function runAgentTasks(tasks: AgentTask[]): Promise<void> {
  // whether tasks.length is 4 or 400, all of them start at once
  await Promise.all(
    tasks.map(async (task) => {
      const result = await agent.run(task);   // generation is fast
      await pushToGitHub(result);              // downstream clogs here
    }),
  );
}

This runs fine while there are a few tasks. But when four sites' worth of tasks arrive almost simultaneously, each carrying several articles, pushToGitHub fires dozens of times at once. GitHub starts returning 429 partway through, and failed tasks are swallowed here (Promise.all rejects if any one rejects, but the pushes that already succeeded can't be undone). The result was the worst state of all: "half of it landed, and I can't tell which ones failed."

There are three problems. First, there is no cap on concurrency. Second, there is no mechanism to match arrival rate to service rate. Third, overflowed work has nowhere to go. We'll fill these in one at a time.

Decouple Arrival From Processing With a Queue

The first move is to insert a queue between arrival and processing. Agents only enqueue work; only a worker pulling from the queue sends it downstream. Now the wave of arrivals no longer slams directly into the processing side.

// A bounded queue. When full, it makes the enqueue side wait.
class BoundedQueue<T> {
  private items: T[] = [];
  private waiters: Array<() => void> = [];
 
  constructor(private readonly maxLength: number) {}
 
  // When full, wait until a slot frees up (this is where backpressure begins)
  async enqueue(item: T): Promise<void> {
    while (this.items.length >= this.maxLength) {
      await new Promise<void>((resolve) => this.waiters.push(resolve));
    }
    this.items.push(item);
  }
 
  dequeue(): T | undefined {
    const item = this.items.shift();
    const waiter = this.waiters.shift();
    if (waiter) waiter(); // a slot opened, so wake one waiting enqueue
    return item;
  }
 
  get length(): number {
    return this.items.length;
  }
}

The key is maxLength. Without a cap, when arrival rate keeps exceeding service rate, the queue grows until it eats all your memory. With a cap, the moment it fills, enqueue is made to wait — and that very act of waiting becomes the backpressure I'll explain below.

Cap Concurrency With a Semaphore

The worker that pulls work off the queue and sends it downstream must not run in unlimited number. Set a concurrency cap matched to downstream's service rate. A semaphore handles this.

// A semaphore that limits the number of jobs passing through to N
class Semaphore {
  private available: number;
  private queue: Array<() => void> = [];
 
  constructor(permits: number) {
    this.available = permits;
  }
 
  async acquire(): Promise<void> {
    if (this.available > 0) {
      this.available -= 1;
      return;
    }
    await new Promise<void>((resolve) => this.queue.push(resolve));
  }
 
  release(): void {
    const next = this.queue.shift();
    if (next) {
      next(); // let one waiter through (permit is handed off)
    } else {
      this.available += 1;
    }
  }
}

The permit count is the concurrency cap against downstream. To allow at most three concurrent pushes to GitHub, set permits to three. Don't pick this number by feel — pick it from measurement, as below. Too high and 429s return; too low and you can't use downstream's capacity.

Match Downstream Rate Limits With a Token Bucket

Capping concurrency still isn't enough. Many downstreams rate-limit not by "how many at once" but by "how many per unit time." Against an API capped at ten calls per second, even with a concurrency of one, hammering it in quick succession returns 429. This is where a token bucket helps.

// Consume tokens refilled at a fixed rate; tolerate some burst
class TokenBucket {
  private tokens: number;
  private lastRefill = Date.now();
 
  constructor(
    private readonly capacity: number,     // max bucket size (burst allowance)
    private readonly refillPerSec: number, // tokens refilled per second
  ) {
    this.tokens = capacity;
  }
 
  private refill(): void {
    const now = Date.now();
    const elapsedSec = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsedSec * this.refillPerSec);
    this.lastRefill = now;
  }
 
  // Wait until a token is available, then consume one
  async take(): Promise<void> {
    this.refill();
    while (this.tokens < 1) {
      const waitMs = ((1 - this.tokens) / this.refillPerSec) * 1000;
      await new Promise((r) => setTimeout(r, Math.ceil(waitMs)));
      this.refill();
    }
    this.tokens -= 1;
  }
}

The trick is setting refillPerSec slightly below downstream's rate limit. For an API allowing ten calls per second, putting refillPerSec at eight or nine leaves headroom for measurement error and retries, and you almost never hit 429. capacity is the burst allowance. If your pattern is "usually idle but occasionally arrives in clumps," set the capacity a bit larger.

Backpressure: Stop Accepting When It Clogs

Combine these parts and the worker's flow becomes "pull from queue, take a semaphore permit, take a token, send downstream." When downstream lags, tokens don't accumulate, permits don't free up, and the worker can't pull from the queue. The queue then fills, and enqueue is made to wait.

This "enqueue is made to wait" is backpressure — the mechanism that propagates downstream congestion upstream and throttles intake. The fact that downstream is clogged travels through the queue to the agents, and the agents can't inject new work, so they wait. As a result, arrival rate automatically converges toward service rate.

// Pipeline assembled from the parts, with flow control
class FlowControlledPipeline<T> {
  private readonly queue: BoundedQueue<T>;
  private readonly semaphore: Semaphore;
  private readonly bucket: TokenBucket;
  private running = true;
  private inFlight = 0;
 
  constructor(opts: {
    maxQueueLength: number;
    maxConcurrency: number;
    refillPerSec: number;
    burstCapacity: number;
    handle: (item: T) => Promise<void>;
    onOverflow: (item: T, reason: string) => Promise<void>;
  }) {
    this.queue = new BoundedQueue<T>(opts.maxQueueLength);
    this.semaphore = new Semaphore(opts.maxConcurrency);
    this.bucket = new TokenBucket(opts.burstCapacity, opts.refillPerSec);
    this.opts = opts;
    void this.loop();
  }
 
  private opts: {
    handle: (item: T) => Promise<void>;
    onOverflow: (item: T, reason: string) => Promise<void>;
  } & Record<string, unknown>;
 
  // Agents just call this. If full, they wait here (= backpressure)
  async submit(item: T): Promise<void> {
    await this.queue.enqueue(item);
  }
 
  private async loop(): Promise<void> {
    while (this.running) {
      const item = this.queue.dequeue();
      if (item === undefined) {
        await new Promise((r) => setTimeout(r, 20));
        continue;
      }
      await this.semaphore.acquire();
      await this.bucket.take();
      this.inFlight += 1;
      void this.process(item);
    }
  }
 
  private async process(item: T): Promise<void> {
    try {
      await this.opts.handle(item);
    } catch (err) {
      await this.opts.onOverflow(item, String(err));
    } finally {
      this.inFlight -= 1;
      this.semaphore.release();
    }
  }
}

With an interface that only exposes submit, the agent side never has to be aware backpressure exists. When it clogs, they naturally wait; when it frees, they naturally proceed. That "no need to be aware" state is what makes operations sustainable.

Overflowed Work Goes to a Dead Letter

Even with backpressure throttling intake, if downstream is down for a long time, waiting alone gets you nowhere. Work that fails after a fixed number of retries should not be swallowed — send it to a dead-letter store, a place to park jobs that failed processing. The point is to not discard it but to leave it where a human can pick it up later.

async function handleWithDeadLetter(item: AgentTask): Promise<void> {
  const maxAttempts = 3;
  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    try {
      await pushToGitHub(item.result);
      return;
    } catch (err) {
      if (attempt === maxAttempts) {
        // park it with a reason in a store (KV, R2, logs)
        await deadLetterStore.put(item.id, {
          item,
          reason: String(err),
          failedAt: new Date().toISOString(),
        });
        return;
      }
      // exponential backoff; give downstream time to recover
      await new Promise((r) => setTimeout(r, 2 ** attempt * 500));
    }
  }
}

When Promise.all burned me, the worst part was "I can't tell which ones failed." Once I added a dead letter, I settled into a routine: check the parked items each morning, and if downstream has recovered, re-inject them. You can't avoid failures themselves, but you can avoid failures becoming invisible.

Set Thresholds From Real Numbers

Finally, how to choose these parameters. Pick by feel and you'll either over-defend and go slow, or under-defend and hit 429. I measure and decide in this order.

First, measure service rate. Against downstream, raise concurrency from one in small steps and find the value just before error rate starts climbing. In my setup, pushes to GitHub were stable around three concurrent and eight per second. Cloudflare deploys are better serial anyway, so I cap them at one concurrent.

Second, set queue length. The queue is a buffer for absorbing temporary overshoot. Too long, and when it clogs, stale work lingers forever and loses freshness. I use "service rate × tolerable delay" as a guide. Clearing eight per second and tolerating up to two minutes of delay gives a cap of about 960. For article generation at a few dozen per day, far smaller is plenty.

Third, draw the line between backoff and dead letter. My default is "retry three times, then park." More retries than that delays recovery when downstream is genuinely down, and stacks up wasted cost.

The numbers differ per site and per app, but the method is the same. Measure first, set thresholds just before errors appear, absorb overshoot in the queue, and divert what the queue can't absorb to a dead letter. Keep that order, and you can raise parallelism without breaking production.

Flow control is an unglamorous mechanism, but if you're running multiple agents in production, it's worth adding before you add intelligence. If you want to try it next, pick one spot where you currently fan work out with Promise.all, and start by inserting a BoundedQueue and a semaphore. Even that one spot should noticeably cut how often 429s appear.

I hope this helps anyone running multiple agents in production the way I am. Thank you for reading.