Rehearsing an Agent's Actions Before They Touch Production — Designing a Zero-Side-Effect Dry-Run Layer

Switching the boundary by mode

Next, implement the boundary that receives Effects. It carries a mode and switches behavior: execute immediately in production, record only in dry-run.

// boundary.ts — the side-effect boundary itself
import { Effect } from "./effect";
 
export type RunMode = "dry-run" | "commit";
 
export class SideEffectBoundary {
  private planned: Effect[] = [];
 
  constructor(private mode: RunMode) {}
 
  // Tools never execute side effects directly — they always call this.
  async apply(effect: Effect): Promise<void> {
    if (this.mode === "dry-run") {
      // Do not execute. Just record it as a plan.
      this.planned.push(effect);
      return;
    }
    // Production mode: record before executing (leave a trace first).
    this.planned.push(effect);
    await effect.commit();
  }
 
  // Deletions are high-risk, so make a surge in their count detectable.
  get deleteCount(): number {
    return this.planned.filter((e) => e.kind === "delete").length;
  }
 
  get plan(): readonly Effect[] {
    return this.planned;
  }
}

On the tool side, you rewrite code that used to hit an SDK directly so it goes through boundary.apply() instead.

// tools/remote-config.ts — a tool declaring a side effect
import { SideEffectBoundary } from "../boundary";
 
export async function setCampaignFlag(
  boundary: SideEffectBoundary,
  key: string,
  value: boolean,
  client: RemoteConfigClient,
): Promise<void> {
  const before = await client.get(key); // reads aren't side effects — always run them
  await boundary.apply({
    kind: "kv-write",
    summary: `Change Remote Config ${key} from ${before} to ${value}`,
    target: `remote-config/${key}`,
    before,
    after: value,
    commit: async () => {
      await client.set(key, value);
    },
  });
}

With this shape, the tool no longer needs to know whether it's a dry-run. It just tells the boundary what it wants to do; whether to execute or record is the boundary's decision. The separation of concerns lands cleanly.

Showing the plan to a human as a diff

A recorded plan is useless if you can't review it before applying. I format mine to read like a code-review diff. Distinguishing risk visually — red for deletions, yellow for writes — lets me catch the seed of an accident even when skimming an overnight job's log in the morning.

// render.ts — format a plan into human-readable diff text
import { Effect } from "./effect";
 
export function renderPlan(plan: readonly Effect[]): string {
  if (plan.length === 0) return "(no changes)";
 
  const lines: string[] = [];
  const deletes = plan.filter((e) => e.kind === "delete").length;
  lines.push(`Planned changes: ${plan.length} (deletions: ${deletes})`);
  lines.push("");
 
  for (const e of plan) {
    const mark = e.kind === "delete" ? "[DELETE]" : "[CHANGE]";
    lines.push(`${mark} ${e.target}`);
    lines.push(`    ${e.summary}`);
    if (e.kind !== "delete" && e.before !== undefined) {
      lines.push(`    before: ${JSON.stringify(e.before)}`);
      lines.push(`    after:  ${JSON.stringify(e.after)}`);
    }
  }
  return lines.join("\n");
}

In practice I send this diff to Slack or a local log, and if the deletion count exceeds a threshold I set in advance, I require human approval before proceeding to real application. The Remote Config near-miss I opened with is exactly the case this caught. My intent was a single write, but the plan contained four deletions. When the numbers don't add up, you know something is off.

A two-phase design that applies the very same plan

Once a human approves the plan built during the dry-run, you apply it to production. The crucial point here is that the plan shown during the dry-run and the actions executed in production must be the same list of Effects. Rebuilding them separately invites the worst kind of drift: something different from what you reviewed gets executed.

// run.ts — two-phase run: dry-run first, then commit the same plan after approval
import { SideEffectBoundary } from "./boundary";
import { renderPlan } from "./render";
 
export async function runWithRehearsal(
  agentTask: (b: SideEffectBoundary) => Promise<void>,
  approve: (planText: string, deletes: number) => Promise<boolean>,
): Promise<{ applied: boolean; planText: string }> {
  // Phase 1: dry-run. Build the plan with zero side effects.
  const dry = new SideEffectBoundary("dry-run");
  await agentTask(dry);
  const planText = renderPlan(dry.plan);
 
  // Let a human (or a rule) decide whether to approve the plan.
  const ok = await approve(planText, dry.deleteCount);
  if (!ok) {
    return { applied: false, planText };
  }
 
  // Phase 2: execute the approved plan as-is.
  // Note: do NOT re-run agentTask — commit the Effects captured in phase 1.
  const commit = new SideEffectBoundary("commit");
  for (const effect of dry.plan) {
    await commit.apply(effect);
  }
  return { applied: true, planText };
}

There's a fork in the road here. In phase 2, do you commit the same Effects, or do you re-run agentTask in commit mode? I chose the former. The latter looks simpler, but an agent's judgment wobbles, and a second run can produce a different plan than the first. If the plan you carefully reviewed and the actions actually applied diverge, the dry-run was pointless. Apply the exact Effect list you reviewed. That is the heart of the two-phase design.

Read consistency: not letting the dry-run lie

This is the part I stumbled over in production, and it's rarely spelled out in the docs. Because a dry-run causes no side effects, it builds its plan on the assumption that the state of the world it read won't change until the moment of application. In reality, time passes between the dry-run and the real apply. During the minutes or hours you wait for approval, external state can shift.

Here's what bit me. At dry-run time, a Remote Config key existed, so the plan was "update it." But while it sat awaiting approval, another job deleted that key. When commit ran after approval, it tried to update a key that no longer existed and errored out. The plan wasn't a lie — the world had moved.

The fix was to give each Effect a precondition, checked immediately before commit. I confirm that the before value read during the dry-run still holds just before applying, and abort if it differs. It's optimistic locking, carried into an agent's external actions.

// add a precondition to Effect
export interface Effect {
  kind: EffectKind;
  summary: string;
  target: string;
  before?: unknown;
  after?: unknown;
  // called just before commit. Returning false means the world changed — abort.
  precondition?: () => Promise<boolean>;
  commit: () => Promise<void>;
}

// verify the precondition at commit time in boundary.ts
async apply(effect: Effect): Promise<void> {
  if (this.mode === "dry-run") {
    this.planned.push(effect);
    return;
  }
  if (effect.precondition) {
    const stillValid = await effect.precondition();
    if (!stillValid) {
      throw new StalePlanError(
        `Plan is stale: the precondition for ${effect.target} changed`,
      );
    }
  }
  this.planned.push(effect);
  await effect.commit();
}

Since adding this small step, "the world differs from what I reviewed" accidents have gone to zero. A dry-run does not promise the future; it only captures the world at the moment the plan was made. Asking, right before applying, whether that premise still holds — only with that does the dry-run become a safety device you can trust.

The numbers after running it

After folding this dry-run layer into the autonomous operation of six sites, several numbers changed clearly.

Before, agent jobs carrying external side effects produced unintended production changes about two or three times a month. Most were minor and quickly reversible, but some — Remote Config, store metadata — were hard to undo. In the two months since adding the dry-run layer and a deletion threshold, the number of bad actions that reached production is zero. Plans stopped before reaching production — caught by diff review and by the precondition combined — came to eleven.

I also measured the time review takes. Back when I read full-text logs instead of formatted diffs, I spent four to five minutes per job confirming. After formatting deletions in red and putting counts in the header, normal cases take about a ten-second skim, and I only stop when counts cross the threshold. The lighter a safety device is, the more likely you keep using it for real. Heavy safety devices quietly decay into "approve anyway" and lose their meaning.

On cost: because a dry-run executes nothing, it consumes almost no external API write quota. But building the dry-run plan still incurs the agent's inference cost, so running dry then real means two passes and higher inference spend. I make only high-risk actions (deletions, production setting changes) two-phase, and skip the dry-run for safe, read-heavy jobs.

Where to put a dry-run, and where not to

Finally, the criteria I use by situation. Making every action a dry-run is unrealistic on both cost and operational load.

I always dry-run hard-to-reverse external side effects: rewriting production Remote Config, updating store metadata, changing DNS or Cloudflare settings, and deletions of any kind. Once executed, these are costly to recover — sometimes impossible. The old "cut only once" rule maps exactly onto this territory.

I skip the dry-run for easily reversible actions and read-heavy jobs. A Git commit can be reviewed and rolled back; draft article generation never reaches production directly. Wrapping those in a dry-run just doubles the work for almost no accident prevented.

If I had to name a single axis, it's: "Can this be undone?" Reversible actions are well served by canaries and rollback. Reserve the pre-application rehearsal of a dry-run for the actions that can't be undone. Shadow mode is for observation, canaries for controlling post-application blast radius, dry-runs for blocking errors before application — the three don't compete; they guard different moments.

Handing external actions to an agent is a question of trust and, at the same time, a question of design. Having a human judge trustworthiness every single time doesn't scale; handing it over entirely is frightening. What fills the gap, I've come to feel, is a mechanism that checks one more time before applying. Measure as often as you like; cut only once. That old rule is what I'm translating into code here. I hope it offers a useful design hint to anyone else facing external side effects.