Never Miss a Managed Agent Completion: Pairing a Serverless Receiver with Polling Reconciliation

Build an idempotent receiver on Cloudflare Workers

First, the push receiver. Three design goals: verify the signature, reject duplicates, and never drop even an unknown job.

// worker: receiver for the Managed Agent completion webhook
// KV bindings: JOBS (job records) / SEEN (delivery-dedup records)
interface Env {
  JOBS: KVNamespace;
  SEEN: KVNamespace;
  WEBHOOK_SECRET: string;
}
 
export default {
  async fetch(req: Request, env: Env): Promise<Response> {
    if (req.method !== "POST") return new Response("nope", { status: 405 });
 
    const raw = await req.text();
    // 1) Verify signature on the RAW body (recompute HMAC, constant-time compare)
    const ok = await verifySignature(raw, req.headers.get("x-agent-signature"), env.WEBHOOK_SECRET);
    if (!ok) return new Response("bad signature", { status: 401 });
 
    const ev = JSON.parse(raw) as {
      deliveryId: string;   // unique per delivery (used for dedup)
      jobId: string;        // unique job key
      state: "running" | "succeeded" | "failed";
      seq: number;          // monotonic version number stamped by the sender
      finishedAt?: string;
    };
 
    // 2) Reject duplicate delivery (never process the same deliveryId twice)
    if (await env.SEEN.get(ev.deliveryId)) {
      return new Response("duplicate", { status: 200 }); // 200 stops the resend
    }
 
    // 3) Apply the transition monotonically (create a record even for unknown jobs)
    await applyTransition(env, ev);
 
    // 3 days of retention outlasts the sender's retry window
    await env.SEEN.put(ev.deliveryId, "1", { expirationTtl: 60 * 60 * 24 * 3 });
    return new Response("ok", { status: 200 });
  },
};

What matters here is returning 200, not a 4xx, when you detect a duplicate. To the sender, 200 signals "already accepted," which stops the retries. Return an error and the more dutiful the counterpart, the longer it keeps resending and loading you. In production I once returned 409 for duplicates, and the sender's retries spun endlessly; ever since, I pin this to 200.

And do not discard an unknown jobId. The next applyTransition creates the record if it is missing, then applies the transition. That is the answer to the race from the previous section, where receipt outruns the start record.

Keep transitions monotonic to protect the single truth

Here is the body of applyTransition. This is the merge point of the two channels; both push and pull pass through this same function.

type JobState = "unknown" | "running" | "succeeded" | "failed";
const RANK: Record<JobState, number> = { unknown: 0, running: 1, succeeded: 2, failed: 2 };
 
interface JobRecord {
  jobId: string;
  state: JobState;
  seq: number;          // max seq already adopted
  finishedAt?: string;
  claimedAt?: string;   // marker that the side effect ran (see below)
  createdAt: string;
}
 
async function applyTransition(env: Env, ev: { jobId: string; state: JobState; seq: number; finishedAt?: string }) {
  const key = `job:${ev.jobId}`;
  const cur = (await env.JOBS.get(key, "json")) as JobRecord | null;
 
  // Start from unknown if there is no record (prevents dropping unknown jobs)
  const base: JobRecord = cur ?? { jobId: ev.jobId, state: "unknown", seq: -1, createdAt: new Date().toISOString() };
 
  // Monotonicity: adopt only a newer seq AND a non-regressing state
  const forward = ev.seq > base.seq && RANK[ev.state] >= RANK[base.state];
  if (!forward) return; // ignore stale or rollback notifications
 
  const next: JobRecord = {
    ...base,
    state: ev.state,
    seq: ev.seq,
    finishedAt: ev.finishedAt ?? base.finishedAt,
  };
  await env.JOBS.put(key, JSON.stringify(next));
 
  // Run the side effect exactly once, only on first arrival at a terminal state
  if ((next.state === "succeeded" || next.state === "failed") && !base.claimedAt) {
    await runSideEffectOnce(env, next);
  }
}

RANK ranks the states, and combined with seq (the sender's monotonic version number) it guarantees "no regression." Even if ordering flips and a stale running arrives after succeeded, forward becomes false and it is ignored. A finished job rolling back to running cannot happen by construction.

Side effects — publishing a file, kicking off a downstream job, sending a notification — run only on first arrival at a terminal state. Even if push and pull both carry succeeded at once, whichever reaches first sets claimedAt, and the other sees it and holds back. Once you duplicate the path, this exactly-once guarantee is mandatory.

Make polling reconciliation the backstop

The reason the state still converges when push goes silent is that pull exists. Run it periodically with Cron Triggers, and query the cloud only for jobs that have not reached a terminal state.

// scheduled(): reconcile unfinished jobs against the cloud at a fixed interval
export async function reconcile(env: Env) {
  const list = await env.JOBS.list({ prefix: "job:" });
  const now = Date.now();
 
  for (const k of list.keys) {
    const rec = (await env.JOBS.get(k.name, "json")) as JobRecord;
    if (rec.state === "succeeded" || rec.state === "failed") continue; // terminal: no reconcile
 
    // Fetch the authoritative state from the cloud (Managed Agents get API)
    const remote = await fetchJobStatus(env, rec.jobId);
    if (!remote) continue;
 
    // Feed it through the SAME applyTransition (keep the merge point single)
    await applyTransition(env, {
      jobId: rec.jobId,
      state: remote.state,
      seq: remote.seq,           // assume the get API returns the same seq
      finishedAt: remote.finishedAt,
    });
 
    // Catch "started but stuck in unknown/running for too long" as an anomaly
    const ageMin = (now - Date.parse(rec.createdAt)) / 60000;
    if (rec.state !== "succeeded" && rec.state !== "failed" && ageMin > 30) {
      await alertStuckJob(env, rec.jobId, Math.round(ageMin));
    }
  }
}

Feeding polling through the same applyTransition as the webhook is the crux of this design. Keeping the merge point single prevents the accident where the "webhook-sourced record" and the "polling-sourced record" hold separate truths. Both channels can only write results that have passed the monotonicity rule.

Calibrate the interval by working back from the job's expected duration. For jobs that finish in minutes, a 1-to-2-minute interval works; for ones that take tens of minutes, 5 minutes is enough. A KV list gets heavier as keys grow, so evict terminal records to a separate prefix the next day, or let them expire via TTL, to keep the reconciliation set from bloating. I run unattended jobs for six sites, and after switching to folding terminal records away at 24 hours, the cost of one reconcile pass settled to nearly constant.

Operating it as an indie developer: thresholds and monitoring

Duplication is meaningless as "just buying insurance"; you feel safe only once you can see that the insurance is working. Here are the metrics I actually watch.

"Reconcile rescues" in particular is an excellent leading indicator. When this normally-zero value starts to climb, it is the sign that the webhook has begun to fail quietly, and you can act before users or you notice real harm. A push-only setup cannot even observe this symptom. The real value of duplication, I believe, is not the speed guarantee but exactly this: making one channel's silence visible.

Set thresholds by your own operational feel. In my case, I hold an internal SLA that unattended job completion means "my record reaches a terminal state within 5 minutes of the start." Just reviewing weekly how many times I broke that bar already reveals where along the path things are weakening.

Where implementations tend to trip

A few pitfalls I actually stepped on in production.

Do not blindly trust the sender's seq. Some platforms stamp an independent version number per state, so a contiguous seq across running and succeeded is not guaranteed. In that case it was safer to prioritize the state RANK and demote seq to a tiebreaker within the same state.

Do not overtrust KV's eventual consistency. A read in another region right after SEEN.put can return a stale value, and for a very short window a duplicate slips through. That is precisely why duplicate detection is only a first line of defense; guarantee the ultimate exactly-once with the claimedAt check inside applyTransition. Splitting duplicate defense and side-effect exactly-once into separate layers is a lesson I learned the hard way.

Do not defer signature verification until after parsing the body. Verifying after JSON.parse lets a malformed body throw and gives an attacker room to probe the receiver's behavior. Verify on the raw body and never break the order of parsing only after it passes. For the fundamentals of this receiver, the companion piece a practical pattern for building webhook receivers that never drop events is also worth a look.

Protect side-effect exactly-once with idempotency keys. The claimedAt marker alone still leaves a gap if you crash between the KV write and the side-effect execution. If the side-effect side (payment, publish APIs) accepts an idempotency key, use the jobId as that key and defend it doubly with the receiver's marker. Designing a whole unattended batch to survive re-runs is covered in keeping Managed Agents' unattended batches unbroken across re-runs: idempotency and checkpoints.

How you safely launch a job in the first place is the flip side of receiver design. The launch-side discipline is collected in designing background tasks to run safely on the Managed Agents API.

If you start this design today

First, add just the reconcile polling to one unattended job you already have. Even without a webhook yet, the pull path alone establishes the "never miss a completion" foundation. Then layer push on as an optimization, and once you confirm that "reconcile rescues" stays pinned at 0, your duplication is working correctly.

Mechanisms that run asynchronously on the cloud, like Managed Agents, will surely become ordinary in indie development from here. I am still learning as I operate it, but I feel the idea of designing "the fast path" and "the path that never drops" separately applies just as well to other asynchronous work. Thank you for reading.