Designing an HITL Approval Pipeline That Survives Production — Routing Probabilistic Actions Safely With Antigravity Agent

The core data model for an approval queue

To make an approval pipeline durable, the data model has to be boring and strict. I recommend collapsing the queue into a single PostgreSQL table and enforcing transitions in code.

CREATE TABLE agent_actions (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  idempotency_key TEXT NOT NULL UNIQUE,
  agent_run_id    UUID NOT NULL,
  level           TEXT NOT NULL CHECK (level IN ('L0','L1','L2','L3')),
  confidence      NUMERIC(4,3) NOT NULL,
  payload         JSONB NOT NULL,
  payload_hash    TEXT NOT NULL,
  diff_summary    TEXT NOT NULL,
  state           TEXT NOT NULL DEFAULT 'pending'
                  CHECK (state IN ('pending','approved','rejected','executed','rolled_back','expired')),
  reviewer_id     UUID,
  reviewed_at     TIMESTAMPTZ,
  reason          TEXT,
  expires_at      TIMESTAMPTZ NOT NULL,
  executed_at     TIMESTAMPTZ,
  rollback_token  TEXT,
  created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
 
CREATE INDEX idx_actions_state_level ON agent_actions(state, level, expires_at);
CREATE INDEX idx_actions_agent_run ON agent_actions(agent_run_id);

Three details matter most. First, the unique idempotency_key blocks duplicate approvals when an agent re-submits the same decision on a retry. The scariest production incident I have seen was five copies of the same action piling up after a network blip. Second, expires_at is mandatory from day one: anything not reviewed within 24 hours auto-transitions to expired. Approving stale information is usually worse than letting it lapse. Third, every action carries a rollback_token issued at execution. Without that guarantee, you should not be running automation in production at all.

Transitions belong behind a function that rejects any disallowed move at the entry point. That alone wipes out a category of race conditions in the reviewer UI.

type State = 'pending'|'approved'|'rejected'|'executed'|'rolled_back'|'expired';
 
const TRANSITIONS: Record<State, State[]> = {
  pending:     ['approved', 'rejected', 'expired'],
  approved:    ['executed', 'rolled_back'],
  rejected:    [],
  executed:    ['rolled_back'],
  rolled_back: [],
  expired:     [],
};
 
export async function transition(
  actionId: string,
  to: State,
  reviewer: string,
  reason?: string,
) {
  return db.transaction(async (tx) => {
    const row = await tx.query<{state: State}>(
      `SELECT state FROM agent_actions WHERE id = $1 FOR UPDATE`,
      [actionId],
    );
    if (!row.rows.length) throw new Error('action not found');
    const cur = row.rows[0].state;
    if (!TRANSITIONS[cur].includes(to)) {
      throw new Error(`illegal transition: ${cur} -> ${to}`);
    }
    await tx.query(
      `UPDATE agent_actions
         SET state = $2, reviewer_id = $3, reviewed_at = NOW(), reason = $4
       WHERE id = $1`,
      [actionId, to, reviewer, reason ?? null],
    );
  });
}

The FOR UPDATE lock exists to keep two reviewers from approving the same item simultaneously. I skipped it in my first deployment and saw three duplicate approvals within hours of going live. If you have more than one reviewer, the row lock is not optional.

Confidence scoring and routing

L1 routing only earns its keep when the agent's confidence is calibrated. In Antigravity Agent I make every tool call self-evaluate and save that into confidence. A naive logit-average tends to flatline into a narrow band and stops being useful for routing. What I recommend is a linear combination of three signals:

1. Whether the tool's preconditions were satisfied (binary).
2. The 30-day rollback rate for similar actions (normalized).
3. The agent's own self-evaluation, run through a separate calibration pass.

A small router that uses domain-specific thresholds looks like this:

type ActionLevel = 'L0' | 'L1' | 'L2' | 'L3';
type Route = { route: 'auto' | 'review'; reviewers: number };
 
export function routeAction(action: {
  confidence: number;
  level: ActionLevel;
  domain: string;
}): Route {
  if (action.level === 'L0') return { route: 'auto', reviewers: 0 };
 
  const threshold = thresholdFor(action.domain);
  if (action.level === 'L1' && action.confidence >= threshold) {
    return { route: 'auto', reviewers: 0 };
  }
  if (action.level === 'L3') return { route: 'review', reviewers: 2 };
  return { route: 'review', reviewers: 1 };
}

The 0.85 baseline is not fixed. I recalibrate monthly: if a domain's rollback rate exceeds 1.0% over the trailing 30 days, I raise the threshold to 0.90. If it sits below 0.2% for 60 days, I drop it to 0.80 and let more traffic flow to auto. AdMob-adjacent decisions stay pinned at 0.95 regardless, because the policy surface is volatile and a single mistake is expensive — that is a line drawn against my own revenue mix as an indie developer, and it would move if the revenue mix moved.

Trim the reviewer UI down to a three-second decision

This is the part I underestimated for the longest time. My first reviewer UI dumped the agent's full reasoning trace as JSON, which pushed each review to 90+ seconds. With 80 items per day, I devolved into a stamp machine within a week.

The fix is to put only the minimum decision surface on screen and tuck everything else behind a disclosure.

<ApprovalCard action={a}>
  <Header level={a.level} confidence={a.confidence} expiresIn={a.expiresAt} />
  <DiffSummary text={a.diffSummary} />      {/* single-sentence summary */}
  <ImpactBadges payload={a.payload} />       {/* blast radius, est. DAU impact */}
  <PrimaryActions onApprove={onApprove} onReject={onReject} />
  <details>
    <summary>Details (raw payload, trace, related runs)</summary>
    <RawPayload payload={a.payload} />
    <TraceView runId={a.agentRunId} />
  </details>
</ApprovalCard>

The trick is forcing the agent to generate DiffSummary as one short sentence — the kind a non-engineer could decide on. My prompt literally tells the agent "write so that a middle-school student could decide." That single change pulled my median review time from about 90 seconds down to roughly 12 seconds. Approval quality and review time do not trade off; narrowing the inputs makes human errors less frequent, not more.

For ImpactBadges I show two or three numbers per card — estimated affected DAU, whether the change touches public store listings, and the count of related errors in the last seven days. Concrete numbers anchor the call and reduce the psychological cost of clicking Approve.

Audit logs and recording why an approval happened

I want to be careful with this section. When an external platform — AdMob, App Store, Google Play — flags an account, the support team eventually wants a timeline of who approved what. Trying to reconstruct that after the fact is uniformly a losing game.

I append every state transition to a separate append-only table, agent_action_events. Having an immutable event log lets you answer "who is responsible for this decision?" with a single ID lookup three months later.

CREATE TABLE agent_action_events (
  id          BIGSERIAL PRIMARY KEY,
  action_id   UUID NOT NULL REFERENCES agent_actions(id),
  event_type  TEXT NOT NULL,
  actor       TEXT NOT NULL,
  payload     JSONB NOT NULL,
  occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
 
CREATE INDEX idx_events_action ON agent_action_events(action_id, occurred_at);

I also require a free-text reason on every approval. If you offer canned options, reviewers will reflexively pick one and the field becomes a blank. Demanding even thirty characters of free text dramatically improves your odds of reconstructing intent later. The conclusion is the same whether you run solo or as a team.

One audit hygiene rule: do not put PII into approval payloads. If a customer email lands in the queue, the retention requirements travel with every backup of the log. I store emails as sha256(email + pepper) before they enter the queue, and design the reviewer UI so nothing needs to decrypt them.

Failure modes I have hit and how to dodge them

A short field journal of patterns I have walked into so others do not have to.

First, regenerating the payload after approval is dangerous. If the payload at approval time and at execution time diverge, your reviewer approved A and B shipped. Pin the approved payload's payload_hash and roll back on mismatch at execution.

Second, L3 dual approval breaks down when both reviewers share the same context — same company, same incentives, same calendar. Running solo, I assemble two independent reviewers from different contexts: a family member familiar with the business and an external advisor. I would not recommend approving a policy response alone in production.

Third, do not extend expires_at. The first time you push a 24-hour expiry to 48, it will become 72, then 96. If something missed its window, let it expire and have the agent regenerate from fresh context. New context tends to be a better foundation than stretched approvals.

Fourth, hotkeys on the approval UI invite accidental a mashing. I now require a 1.5-second hover before Approve fires. Reject can stay instant — falsely rejecting is recoverable, falsely approving is not.

Fifth, do not defer the rollback token. I spent the first two weeks of this pipeline without rollback infrastructure and ended up halting an entire L2 category to recover from a single bad ship. Any action without a confirmed rollback procedure belongs in L3, not L1 or L2.

Graduating actions toward automation

HITL is not a permanent station. It is a runway toward automation. To move actions L2 → L1 or L1 → L0, I gate on four criteria:

1. At least 100 historical decisions in the last 60 days.
2. Rollback rate below 0.5%, stable.
3. Domain does not touch policies or contracts.
4. A documented rollback procedure exists and runs in under 30 seconds.

When all four hold, I consider a graduation. I review graduations once per quarter, by hand, not by agent. The reason is small but important: an agent grading its own promotion tends to under-report rollback rates, biased toward keeping itself empowered.

The reverse holds too. When rollback rates worsen, the domain gets demoted. I demoted L1 down to L2 twice during AdMob policy refreshes in 2025. That is not a failure; it is the system working as designed.

After six months of running this pipeline, the most useful framing has not been "approve everything" but "shrink what needs approval at all." Back in 1997, when I taught myself programming as a teenager, an online mentor described art as "a natural language open to anyone." That phrase keeps surfacing when I narrow the reviewer UI to a single screen and cut the decision logic into four tiers — they are different problems, but the disposition is the same.

A practical first step: build the agent_actions table for your own domain and let it carry roughly 30 L2-equivalent actions through manual approval for a month. Once the rollback distribution is visible, you have an evidence-based threshold to set for L1, and the pipeline starts paying for itself.