Rolling Back a Half-Finished Agent: Compensating Transactions for Partial Failure

A minimal Saga orchestrator

Here's the skeleton: a step definition and a runner. Each step carries do and compensate, and only the steps whose do succeeded get compensated, in reverse order.

type StepResult<T> = { ok: true; value: T } | { ok: false; error: Error };
 
interface SagaStep<Ctx> {
  name: string;
  // Forward. Store the result in context for later steps and rollback decisions.
  do: (ctx: Ctx) => Promise<unknown>;
  // Undo. Called in reverse order, only if its `do` succeeded.
  compensate: (ctx: Ctx) => Promise<void>;
  // Crossing this step means no going back (the point of no return, below).
  pivot?: boolean;
}
 
interface SagaReport {
  status: "completed" | "compensated" | "stuck";
  completedSteps: string[];
  failedStep?: string;
  compensationFailures: string[];
}
 
async function runSaga<Ctx>(steps: SagaStep<Ctx>[], ctx: Ctx): Promise<SagaReport> {
  const done: SagaStep<Ctx>[] = [];
  let crossedPivot = false;
 
  for (const step of steps) {
    try {
      await step.do(ctx);
      done.push(step);
      if (step.pivot) crossedPivot = true;
    } catch (error) {
      // The moment partial failure becomes certain.
      if (crossedPivot) {
        // We're past the point of no return -> switch to forward recovery (below).
        return {
          status: "stuck",
          completedSteps: done.map((s) => s.name),
          failedStep: step.name,
          compensationFailures: [],
        };
      }
      return await compensate(done, ctx, step.name);
    }
  }
  return { status: "completed", completedSteps: done.map((s) => s.name), compensationFailures: [] };
}

Compensation undoes the successful steps in reverse order. The key point: if even one compensation fails, don't swallow it — record it and move on. Stopping midway creates an even nastier intermediate state: "halfway through the rollback."

async function compensate<Ctx>(
  done: SagaStep<Ctx>[],
  ctx: Ctx,
  failedStep: string,
): Promise<SagaReport> {
  const compensationFailures: string[] = [];
  // Reverse = undo from the last successful step backward.
  for (const step of [...done].reverse()) {
    try {
      await step.compensate(ctx);
    } catch (e) {
      // Don't stop on a failed compensation. Record it and escalate to a human.
      compensationFailures.push(step.name);
    }
  }
  return {
    status: "compensated",
    completedSteps: done.map((s) => s.name),
    failedStep,
    compensationFailures,
  };
}

A non-empty compensationFailures is the region automatic recovery couldn't reach. I don't leave that to the agent — that part, and only that part, goes to a human (me) review queue. Not automating everything turned out to be the condition for being able to keep automating at all.

Writing the release as a Saga

Let's port the opening incident onto this skeleton. Order matters: put the hard-to-undo operations later and the easy-to-undo ones first. That way, when something fails near the end, the cost of folding back what came before is small.

const releaseSaga: SagaStep<ReleaseCtx>[] = [
  {
    name: "admob-mediation",
    do: async (ctx) => {
      ctx.admobSnapshot = await admob.getMediationConfig(ctx.appId); // save current state for rollback
      await admob.updateMediationConfig(ctx.appId, ctx.newMediation);
    },
    compensate: async (ctx) => {
      if (ctx.admobSnapshot) await admob.updateMediationConfig(ctx.appId, ctx.admobSnapshot);
    },
  },
  {
    name: "appstore-metadata",
    do: async (ctx) => {
      ctx.metaSnapshot = await appstore.getMetadata(ctx.appId);
      await appstore.updateMetadata(ctx.appId, ctx.newMetadata);
    },
    compensate: async (ctx) => {
      if (ctx.metaSnapshot) await appstore.updateMetadata(ctx.appId, ctx.metaSnapshot);
    },
  },
  {
    name: "remote-config-flag",
    // Lighting the flag makes the new feature visible to users = effectively irreversible.
    pivot: true,
    do: async (ctx) => {
      await remoteConfig.setFlag(ctx.appId, "new_feature", true);
    },
    compensate: async (ctx) => {
      await remoteConfig.setFlag(ctx.appId, "new_feature", false); // idempotent
    },
  },
];

The crucial detail is that each do snapshots the current state into context before rewriting it. Compensation doesn't query the external world for "what the past should be" — it just restores the value it saved a moment ago. An external API might return a different value at compensation time, and trusting that would contaminate the rollback.

The point of no return (pivot) and forward recovery

Not every operation is reversible. Sending a user notification, finalizing a charge, exposing a new feature by flipping a flag — none of these can be un-happened. In a Saga, this line is the pivot (point of no return).

My operating rule is simple: gather pivots near the end of the Saga, and if anything fails after crossing a pivot, switch from compensation (backward) to forward recovery. That's why the code above checks crossedPivot and returns status: "stuck". Stuck means "we can't go back, so the only way out is forward to completion."

Forward recovery retries just the remaining steps idempotently and drives them to done. That's exactly why every step after the pivot is designed to be idempotent, snapshot-free, and independently retryable. Put a pivot early and nearly every step after it becomes a forward-recovery target, and the design burden spikes. The "easy-to-undo first" ordering pays off more than it looks.

Screen out un-compensatable steps up front

The thing that helped most in production wasn't a code trick — it was an ordering trick. Operations that are both "irreversible once done" and "prone to failing" get pulled out ahead of the Saga body as a separate, lightweight validation step.

For example, App Store metadata updates can be rejected by character limits or banned-word checks. Hit that on the third step of the body and you're stuck rolling back AdMob and Remote Config. So before the body runs, I run "metadata dry-run validation only," and if it's rejected there, the Saga body never starts.

In my own numbers: before this up-front check, about 60% of total Saga failures were metadata-related, and most of those failed only after the body was partway through. After adding the up-front validation, post-launch partial failures dropped to a handful per month, by feel. Not causing a partial failure is far cheaper than folding one back — cheaper than any amount of polishing your compensation logic.

Detect partial failure fast

Even with correct compensation, slow detection means the intermediate state lingers in production. Early on, I'd wait for the agent's completion report before reading the logs, so it took about 90 seconds on average to notice a partial failure. Late at night, users step on a mixed old/new screen during those 90 seconds.

Now I stream the success/failure of each do straight to structured logs and raise an alert the instant compensate runs. I cut detection-to-compensation-start to about 8 seconds on average. More than the raw seconds, what matters is catching the fact that "compensation ran" at an external observation point, not from the agent's self-report. If the agent crashes mid-run the self-report never arrives, but the per-step structured logs remain.

// Wrap do/compensate in an instrumentation layer to unify the observation point.
function observed<Ctx>(step: SagaStep<Ctx>): SagaStep<Ctx> {
  return {
    ...step,
    do: async (ctx) => {
      const t = Date.now();
      try {
        const r = await step.do(ctx);
        log.info({ saga: "release", step: step.name, phase: "do", ms: Date.now() - t });
        return r;
      } catch (e) {
        log.error({ saga: "release", step: step.name, phase: "do", failed: true });
        throw e;
      }
    },
    compensate: async (ctx) => {
      log.warn({ saga: "release", step: step.name, phase: "compensate", started: true });
      await step.compensate(ctx);
      log.warn({ saga: "release", step: step.name, phase: "compensate", done: true });
    },
  };
}

How much to automate, and where to stop

Finally, this is less about technique than about stance. At first I tried to run the whole thing — compensation included — fully automatic. But cases where compensationFailures is non-empty — where even the rollback failed — always go to my review queue to close out by hand. It happens only a few times a year, but when it does, pushing through with automation-on-automation only widens the wound.

My grandfathers were temple carpenters, and they always checked the surface by hand after the pieces were joined. They never skipped that final pass with the hand after the machine had cut. When I hand a multi-step side effect to an agent, I keep that last check in my own hands too — and it's precisely because that line exists that I can comfortably hand off the rest.

If you're adding this design next, start by picking one multi-step task you already run and ask, for each step, "can I write a single function that undoes this?" When you find an operation you can't, that's your pivot. Begin by pushing pivots toward the end, and the whole compensation layer gets much lighter. I hope this helps in your own work.