Leasing Runtime Environments to Parallel Agents: Port Ranges, DB Schemas, and Dev Servers, One Per Agent

Implementation 1: the lease allocator

First, an allocator that takes an agent index and assembles one lease: it computes the port range, creates the dedicated schema, and writes the env file.

// lease.mjs — lease one runtime environment for a single agent
import { writeFileSync } from 'node:fs';
import pg from 'pg';
 
// Ports consumed per agent. Reserve dev(3000s)/api(4000s)/db-proxy(5000s)
// as one band. Ten leaves room for peripheral tooling.
const PORTS_PER_AGENT = 10;
const BASE = { dev: 3000, api: 4000, dbProxy: 5000 };
 
// Derive the port band deterministically from the agent index (0,1,2,...).
// Fixing the index means the same ports come back on every restart.
function derivePorts(agentIndex) {
  const offset = agentIndex * PORTS_PER_AGENT;
  return {
    dev: BASE.dev + offset,       // e.g. agent0=3000, agent1=3010
    api: BASE.api + offset,
    dbProxy: BASE.dbProxy + offset,
  };
}
 
// Clone a dedicated schema from a "pre-migrated template."
// Do not run migrate every time. This was the biggest bottleneck to parallelism.
async function provisionSchema(client, schema, template = 'tpl_migrated') {
  // DROP SCHEMA can't take a placeholder for an identifier, so narrow the
  // character set before concatenating.
  for (const name of [schema, template]) {
    if (!/^[a-z0-9_]+$/.test(name)) {
      throw new Error(`unsafe schema name: ${name}`);
    }
  }
  // Clean up any leftover, then clone the template structure.
  await client.query(`DROP SCHEMA IF EXISTS ${schema} CASCADE`);
  // clone_schema takes values, not identifiers, so placeholders work here.
  await client.query('SELECT clone_schema($1, $2)', [template, schema]);
  return schema;
}
 
export async function acquireLease(agentIndex, agentId) {
  const ports = derivePorts(agentIndex);
  const schema = `agent_${agentIndex}`;
 
  const client = new pg.Client({ connectionString: process.env.ADMIN_DATABASE_URL });
  await client.connect();
  try {
    await provisionSchema(client, schema);
  } finally {
    await client.end();
  }
 
  // Inject the schema via search_path. App code keeps its table names as-is.
  const dbUrl =
    `${process.env.APP_DATABASE_URL}?options=-c%20search_path%3D${schema}`;
 
  const env = [
    `PORT=${ports.dev}`,
    `API_PORT=${ports.api}`,
    `DB_PROXY_PORT=${ports.dbProxy}`,
    `DATABASE_URL=${dbUrl}`,
    `AGENT_ID=${agentId}`,
    `LEASE_INDEX=${agentIndex}`,
  ].join('\n');
 
  writeFileSync(`.env.agent.${agentIndex}`, env + '\n');
  return { agentIndex, agentId, schema, ports, envFile: `.env.agent.${agentIndex}` };
}

Using search_path is the crux. The application can keep writing table names like users and orders, and they resolve inside whichever schema you injected at connect time. You separate the data container without changing a single line of the code you hand the agent.

Implementation 2: clone a pre-migrated schema from a template

clone_schema is the trick that avoids running migrations every time. Keep one tpl_migrated schema that you have already migrated, and hand each agent a copy of its structure.

-- Clone the structure of a pre-migrated template into another schema.
-- search_path is pinned to pg_catalog so that defaults and constraint
-- definitions always render schema-qualified (the later rewrites rely on it).
CREATE OR REPLACE FUNCTION clone_schema(src text, dst text)
RETURNS void AS $$
DECLARE
  obj record;
BEGIN
  EXECUTE format('CREATE SCHEMA %I', dst);
 
  -- 1) Create the sequences first; serial defaults get repointed at them later.
  FOR obj IN
    SELECT sequencename FROM pg_sequences WHERE schemaname = src
  LOOP
    EXECUTE format('CREATE SEQUENCE %I.%I', dst, obj.sequencename);
  END LOOP;
 
  -- 2) Copy table structure. Primary keys, unique and CHECK constraints,
  --    indexes and defaults arrive here.
  FOR obj IN
    SELECT tablename FROM pg_tables WHERE schemaname = src
  LOOP
    EXECUTE format(
      'CREATE TABLE %I.%I (LIKE %I.%I INCLUDING ALL)',
      dst, obj.tablename, src, obj.tablename
    );
  END LOOP;
 
  -- 3) A serial default is copied verbatim and still points at the src
  --    sequence. Repoint nextval at the dst side.
  FOR obj IN
    SELECT c.relname AS tbl, a.attname AS col,
           pg_get_expr(d.adbin, d.adrelid) AS def
    FROM pg_class c
    JOIN pg_namespace n ON n.oid = c.relnamespace
    JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum > 0
    JOIN pg_attrdef  d ON d.adrelid = c.oid AND d.adnum = a.attnum
    WHERE n.nspname = dst
      AND c.relkind = 'r'
      AND pg_get_expr(d.adbin, d.adrelid) LIKE 'nextval(%'
  LOOP
    EXECUTE format(
      'ALTER TABLE %I.%I ALTER COLUMN %I SET DEFAULT %s',
      dst, obj.tbl, obj.col,
      replace(obj.def, quote_ident(src) || '.', quote_ident(dst) || '.')
    );
  END LOOP;
 
  -- 4) LIKE does not copy foreign keys. Recreate them from the src definitions.
  FOR obj IN
    SELECT c.relname AS tbl, con.conname AS name,
           pg_get_constraintdef(con.oid) AS def
    FROM pg_constraint con
    JOIN pg_class c     ON c.oid = con.conrelid
    JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname = src AND con.contype = 'f'
  LOOP
    EXECUTE format(
      'ALTER TABLE %I.%I ADD CONSTRAINT %I %s',
      dst, obj.tbl, obj.name,
      replace(obj.def, quote_ident(src) || '.', quote_ident(dst) || '.')
    );
  END LOOP;
END;
$$ LANGUAGE plpgsql SET search_path = pg_catalog;

What the template clone quietly drops

With INCLUDING ALL, primary keys, unique constraints, CHECK constraints, defaults and indexes do carry over. That much matched my expectation.

What tripped me up came a few days later. An agent cheerfully handed back an implementation whose test inserted a row with no parent record — and passed. Against the template, that insert should have failed. When I looked, the cloned schema had not a single foreign key in it.

The PostgreSQL documentation is explicit about what LIKE carries. INCLUDING CONSTRAINTS copies CHECK constraints; foreign key constraints are not copied at all. Writing INCLUDING ALL changes nothing there. Meanwhile primary keys, unique constraints and indexes do arrive by way of INCLUDING INDEXES, so as long as you only skim the table list, the clone looks perfectly healthy. That is the part I'd call dangerous.

The same shape of gap sits in serial columns. A GENERATED ... AS IDENTITY column is safe: INCLUDING IDENTITY creates a new sequence in the destination. But serial is only sugar for "integer column plus a nextval default," and that default travels as an expression under INCLUDING DEFAULTS. The cloned column therefore keeps sharing the template's sequence, and you get IDs where agent B continues the numbering agent A started — a jump nobody can explain from the application code.

A clone is verified by constraint count, not by row count. That is the line I drew the night I read that pull request. Right after acquiring a lease I run the query below, and if the counts don't match the template, startup stops there.

-- Check that the template and the clone agree on constraint counts
SELECT n.nspname AS schema,
       count(*) FILTER (WHERE con.contype = 'f') AS foreign_keys,
       count(*) FILTER (WHERE con.contype = 'c') AS check_constraints
FROM pg_constraint con
JOIN pg_class c     ON c.oid = con.conrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname IN ('tpl_migrated', 'agent_0')
GROUP BY n.nspname
ORDER BY n.nspname;

Views, triggers and functions don't travel through LIKE either. If your template holds any of those, I'd add a short block that recreates them from pg_views or pg_trigger, the same way the foreign keys are rebuilt. I may be wrong about how common this is, but I suspect more setups are standing on this hole than realize it.

Clone speed now scales with the number of constraints. Back when the function only copied tables, my setup landed at about 80ms on average, against around 900ms for creating a fresh database with createdb and re-running migrations, even on a small schema. Rebuilding foreign keys adds to that first figure, so it's worth taking one \timing measurement in your own environment when you adopt this. Even so, the more agents you add, the more the template-clone-versus-re-migrate gap lands directly on startup time.

Counter-intuitive point 1: schema isolation is faster than database isolation

At first I assumed "one database per agent" would be cleanest — fully independent namespaces, easy to separate privileges. But measured, it became the startup bottleneck.

Agents rebuild their environment constantly: fail and retry, switch tasks mid-way. If each rebuild costs 900ms, startup eats the gains of going parallel. Isolation does not need to be as strong as "separate database"; "separate schema" is enough — that is what actually working through it taught me. Adding just an agent_id column to a shared DB is the fastest of all, but a mistaken DELETE from one agent hits everyone, so I ruled it out. As a balance between isolation strength and startup speed, schema-per-agent sits in the middle.

Counter-intuitive point 2: a random free port makes operation harder, not easier

"Just grab any unused port dynamically" is the natural instinct. I did that first. But with parallel agents it makes cleanup harder.

Agents restart dev servers repeatedly. When a previous process lingers as a zombie, the next start escapes to a different random port, and the cleanup target changes every time. Who is using which port becomes non-deterministic, and the health-check target won't settle either.

With a deterministic port, cleanup is one line.

# Reliably clean up whatever is left on agent 2's band (3020).
# Because the port is deterministic, the cleanup target is always the same.
lsof -ti:3020 | xargs -r kill -9

Same agent index, same 3020 for the dev server, every time. Monitoring and cleanup both get a fixed target. "Deterministic and traceable" is easier to automate than "random to avoid collisions." This was the opposite of what I expected going in.

Implementation 3: reclaiming the lease — don't leak crashed resources

A quietly important part of parallel operation is reclaiming the lease. When an agent crashes, its schema and ports are left dangling. Left alone, schemas pile up and ports fill.

Keep one mechanism that sweeps expired leases, anchored on the lease ID.

// reap.mjs — reclaim an expired lease (run periodically or at startup)
import pg from 'pg';
import { execSync } from 'node:child_process';
 
const PORTS_PER_AGENT = 10;
const DEV_BASE = 3000;
 
export async function reapLease(agentIndex) {
  // 1) Drop the schema
  const client = new pg.Client({ connectionString: process.env.ADMIN_DATABASE_URL });
  await client.connect();
  try {
    await client.query(`DROP SCHEMA IF EXISTS agent_${agentIndex} CASCADE`);
  } finally {
    await client.end();
  }
 
  // 2) Clean the band (deterministic port means the target is fixed)
  const devPort = DEV_BASE + agentIndex * PORTS_PER_AGENT;
  try {
    execSync(`lsof -ti:${devPort} | xargs -r kill -9`, { stdio: 'ignore' });
  } catch {
    // Nothing to clean is fine; keep reclaiming even if this step fails
  }
}

Treat a lease as something that must always be returned once borrowed, and resources stop growing monotonically even during long automated runs. Design this in early, and you can run agents for three days straight without the environment silting up.

Implementation 4: put the working directory in the same lease

Ports and the data container are now separated. What's left is files. As long as several agents share one checkout, one agent's pnpm install swaps out the node_modules another is using, and one agent's build artifacts leak into another's test run.

I hand out a git worktree as part of the lease. Each branch gets its own directory, and because the actual .git store is shared, disk usage stays modest. Unlike cloning the repository once per agent, there's no duplicated fetch either.

// added to lease.mjs — hand out a worktree as part of the lease
import { execFileSync } from 'node:child_process';
import { existsSync } from 'node:fs';
 
// Provision a worktree dedicated to one agent.
// The branch name is derived from the index too, so reclamation has one obvious target.
function provisionWorktree(agentIndex, baseRef = 'origin/main') {
  const dir = `../wt-agent-${agentIndex}`;
  const branch = `agent/${agentIndex}`;
 
  if (existsSync(dir)) return dir; // keep re-acquisition idempotent
 
  // -B recreates the branch, so leftovers from a previous lease snap back to baseRef
  execFileSync('git', ['worktree', 'add', '-B', branch, dir, baseRef], {
    stdio: 'inherit',
  });
  return dir;
}

I use -B rather than -b because I don't want to inherit whatever the previous lease left behind. Agents happily leave abandoned commits around. If the next lease reuses the same index, results are much easier to read when the starting point is always origin/main.

Reclamation needs a matching pair of calls.

// added to reap.mjs — remove the worktree along with its registration
execFileSync('git', ['worktree', 'remove', '--force', `../wt-agent-${agentIndex}`], {
  stdio: 'ignore',
});
execFileSync('git', ['worktree', 'prune'], { stdio: 'ignore' });

If you skip worktree remove and just rm -rf the directory, git keeps a ghost registration. The next worktree add to that path is refused, and one morning that took down my entire scheduled start-up. Treat prune as part of reclamation, not an optional extra.

After the lease is handed over — one step before you send the task

If you send the task immediately after creating the lease, the agent starts verifying behavior while the dev server is still coming up. I've watched an agent read a connection refusal as "my implementation is broken" and start editing code that was already correct.

So I put an acceptance check between lease acquisition and task submission.

#!/usr/bin/env bash
# ready.sh <agentIndex> — wait for the dev server before handing over the task
set -euo pipefail
IDX="$1"; PORT=$((3000 + IDX * 10)); DEADLINE=$((SECONDS + 60))
 
until curl -sf "http://127.0.0.1:${PORT}/api/health" >/dev/null; do
  if (( SECONDS > DEADLINE )); then
    echo "agent ${IDX}: dev server not ready on ${PORT}" >&2
    exit 1
  fi
  sleep 1
done
echo "agent ${IDX}: ready on ${PORT}"

The 60-second cutoff exists because failing fast beats waiting. If the server hasn't come up by then, it's almost always a lease misconfiguration — not the kind of problem more time solves.

I also write the lease values verbatim at the top of the instructions I give the agent.

Start this task by loading .env.agent.2.
Use only http://127.0.0.1:3020 for the dev server; do not fall back to another port.
The database is already pinned to search_path=agent_2.
Your working directory is ../wt-agent-2.

The "do not fall back" line matters more than it looks. An agent that decides the assigned port is taken will helpfully wander off to 3001 or 3002 — and the moment it does, resources escape the lease and fall outside reclamation.

Measuring serial, naive parallel, and leased parallel on the same work

Back to the 1.4x slowdown from the opening. I ran the same three tasks (add a UI component, add an API route, run a schema migration) three ways and took wall-clock times. M2 Pro with 32GB, a Next.js 15 app, PostgreSQL 16 running locally, median of three runs each.

Naive parallel loses to serial not because of the failures themselves but because of the time agents spend investigating after a failure. An agent that receives EADDRINUSE reads config files, digs through logs, and sometimes rewrites code. A human recognizes "the port is just taken" in three seconds; the agent burns minutes. The higher the parallelism, the more that investigation time dominates.

The 8 minutes for leased parallel is essentially pinned to the heaviest of the three tasks, the schema migration. Which means further gains have to come from how the work is split, not from more parallelism. Seeing exactly where the returns flatten out, in numbers, was the part I valued most.

Operational details the docs don't mention

A few small things I only learned by doing that make a real difference.

You hit the inotify watch limit first. Start N dev servers and each one watches node_modules, consuming inotify instances. On Linux the default limit is low, and I reached ENOSPC (the watch limit, not disk) at three to four agents in parallel. Narrow each agent's watch scope to its own worktree, or raise fs.inotify.max_user_watches.

Run migrations once, outside lease acquisition. Update the tpl_migrated template exactly once, before starting any agent. Letting an agent call migrate inside its lease defeats the point of cloning and races migrations against each other. Migrations before the lease, cloning inside it — keep the roles separate.

Make the lease ID a log correlation key. In parallel, you lose track of which agent's which attempt a log line belongs to. Stamp AGENT_ID and LEASE_INDEX on every log line and you can trace failures afterward. The debuggability of parallel operation is largely decided right here.

Where to start

You don't need to build all of this at once. In order of how much the collision hurts, I'd suggest:

Deterministic port allocation first. Just making EADDRINUSE disappear changes how parallelism feels. Then schema isolation with template cloning, which lowers the risk of data loss and makes retries cheap. Worktree separation next, which makes file-related mysteries disappear and failures far easier to attribute. Reclamation and the readiness check last — introduce them when you step into long-running automation, and that's soon enough.

Parallel agents are decided less by cleverness and more by "environments that don't overlap" — that's the sense I reached the long way around. Next time you run two or more agents at once, try pinning the ports first. That's the first fork in the road.

I hope this helps with your own design decisions. Thank you for reading.